---
title: "Discussion on \"Teaching a Small Language Model to Triage Incidents: A Beginner’s Unsloth and QLoRA Tutorial\" | Hashnode"
description: "Discussion on \"Teaching a Small Language Model to Triage Incidents: A Beginner’s Unsloth and QLoRA Tutorial\". A beginner-friendly Colab + Unsloth experiment in synthetic data, QLoRA, held-out evaluation, and strict schema validation."
url: "https://hashnode.com/posts/teaching-a-small-language-model-to-triage-incidents-a-beginner-s-unsloth-and-qlora-tutorial/6ae3e635d6f91c173574e3a9"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [Teaching a Small Language Model to Triage Incidents: A Beginner’s Unsloth and QLoRA Tutorial](https://nightthoughts.me/teaching-a-small-language-model-to-triage-incidents-a-beginner-s-unsloth-and-qlora-tutorial?utm_source=hashnode&utm_medium=feed)

A beginner-friendly Colab + Unsloth experiment in synthetic data, QLoRA, held-out evaluation, and strict schema validation.

Oct 30 · 31 min read · nightthoughts.me  
Updated Nov 7  
Read the article

-   [#ai](https://hashnode.com/tag/ai)
-   [#llm](https://hashnode.com/tag/llm)
-   [#machine-learning](https://hashnode.com/tag/machine-learning)
-   [#qlora](https://hashnode.com/tag/qlora)
-   [#finetuning](https://hashnode.com/tag/finetuning)
-   [#unsloth](https://hashnode.com/tag/unsloth)
-   [#google-colab](https://hashnode.com/tag/google-colab)

## Written by

[Haithem Slimi](https://hashnode.com/@haithem-slimi)

@haithem-slimi · Hong Kong, China

  
[View profile](https://hashnode.com/@haithem-slimi)

Passing down what we know. Discovering what's next.

Father of two, tech lover. Building systems by day, raising curious minds by night. I write hands-on guides and reproducible experiments on running, fine-tuning and evaluating AI locally on Apple Silicon, on AI agents, and on how AI changes software and finance. The code behind every experiment is on GitHub: github.com/hslimi/night-thoughts

## Responses 5

Markdown supported · ⌘↵ to post

[Julian Neagu](https://hashnode.com/@julianneagulab)

[Nov 1](https://hashnode.com/posts/teaching-a-small-language-model-to-triage-incidents-a-beginner-s-unsloth-and-qlora-tutorial/6ae3e635d6f91c173574e3a9/comment/6ae702bdec361e7786ce9a63)

The rising validation loss is the part I'd dig into next. Training loss looks nice, but it doesn't mean much if the model gets worse on unseen cases. I'd try fewer steps and compare the same test set.

[Kartik N V J K](https://hashnode.com/@kartik-nvj)

[Oct 30](https://hashnode.com/posts/teaching-a-small-language-model-to-triage-incidents-a-beginner-s-unsloth-and-qlora-tutorial/6ae3e635d6f91c173574e3a9/comment/6ae49d987a8fca3c7281696d)

Summary accuracy at 0.00% means exact match could never rise above 0.00%, so that row mostly measures the summary. I'd compute exact match over the five structured fields and score the summary on its own. The unknown\_worker\_saturation case would then count as a full match.

[Haithem Slimi](https://hashnode.com/@haithem-slimi)

[Oct 30](https://hashnode.com/posts/teaching-a-small-language-model-to-triage-incidents-a-beginner-s-unsloth-and-qlora-tutorial/6ae3e635d6f91c173574e3a9/comment/6ae49d987a8fca3c7281696d#reply-6ae4fe1d8a83f36217700821)

Yes, that’s correct. With summary accuracy at 0%, six-field exact match is necessarily 0%, so it doesn’t reveal how often the other five fields were all correct. A separate five-field structured exact-match score would be more informative, while the summary could be evaluated independently for semantic adequacy rather than exact wording. The unknown\_worker\_saturation example would indeed count as a full structured match under that metric. I’d keep both metrics rather than replace the original: six-field exact match measures the complete output contract, and five-field exact match isolates structured classification performance.

[Neurobro](https://hashnode.com/@neurobro)

[Oct 30](https://hashnode.com/posts/teaching-a-small-language-model-to-triage-incidents-a-beginner-s-unsloth-and-qlora-tutorial/6ae3e635d6f91c173574e3a9/comment/6ae45e90ad61bfcfec09bd47)

The split where 2 scenarios went 0/8 on category and the other 3 went 8/8 is the most interesting part to me. Feels less like a weak model and more like it latched onto surface words - a signing-key "rotation" reads a lot like a deploy. Would be curious if human-written paraphrases of those two flip the numbers. Also respect for counting fenced JSON as a hard fail instead of quietly repairing it.

[Haithem Slimi](https://hashnode.com/@haithem-slimi)

[Oct 30](https://hashnode.com/posts/teaching-a-small-language-model-to-triage-incidents-a-beginner-s-unsloth-and-qlora-tutorial/6ae3e635d6f91c173574e3a9/comment/6ae45e90ad61bfcfec09bd47#reply-6ae4fd84ceb94a7055ae95fa)

That’s a plausible hypothesis, although the aggregate results alone don’t establish it. I’d need to inspect the raw predictions and compare the misleading cues across examples.

Human-written paraphrases would be a good follow-up: keep the same labels and held-out scenarios, remove some of the obvious lexical cues, and evaluate the same adapter without retraining. That would help distinguish scenario understanding from keyword matching.

And yes, fenced JSON was intentionally a hard failure because the output was defined as a strict application interface rather than text to be repaired silently.
