What you need

  • A 24GB NVIDIA GPU (RTX 3090/4090) — or rent one by the hour for ~2€ total (see RunPod vs Vast.ai)
  • Linux or Windows + WSL2, Python 3.10+, ~40GB free disk
  • A dataset: 500–5,000 examples of the behavior you want (we build it in step 1)

Step 1 — Build the dataset (the part that decides everything)

One JSON object per line (train.jsonl), in chat format:

{"messages": [ {"role": "system", "content": "You are the support assistant of ACME S.p.A...."}, {"role": "user", "content": "Customer writes: invoice not received..."}, {"role": "assistant", "content": "THE EXACT ANSWER, in the exact style/format you want"} ]}

The rules that matter more than any hyperparameter:

  • Consistency beats volume. 800 examples with identical formatting/tone beat 20,000 noisy ones. Every inconsistency in the data becomes a behavior the model learns.
  • Show the target behavior, don't describe it. The assistant turns must BE what you want to get.
  • Hold out 30–50 examples in a separate eval.jsonl — never trained on, used in step 6.

Step 2 — Install Unsloth (5 min)

$ python -m venv .venv && source .venv/bin/activate
$ pip install unsloth

Unsloth wraps the PEFT/TRL stack with heavy optimizations — roughly 2× faster and lighter on VRAM than vanilla setups, which is what makes a 14B comfortable on 24GB.

Step 3 — The training script (train.py)

from unsloth import FastLanguageModel from trl import SFTTrainer, SFTConfig from datasets import load_dataset model, tokenizer = FastLanguageModel.from_pretrained( "unsloth/phi-4", load_in_4bit=True, max_seq_length=2048) model = FastLanguageModel.get_peft_model( model, r=16, lora_alpha=16, lora_dropout=0, target_modules=["q_proj","k_proj","v_proj","o_proj", "gate_proj","up_proj","down_proj"]) dataset = load_dataset("json", data_files="train.jsonl", split="train") dataset = dataset.map(lambda ex: {"text": tokenizer.apply_chat_template( ex["messages"], tokenize=False)}) trainer = SFTTrainer(model=model, tokenizer=tokenizer, train_dataset=dataset, dataset_text_field="text", args=SFTConfig(per_device_train_batch_size=2, gradient_accumulation_steps=8, num_train_epochs=2, learning_rate=2e-4, lr_scheduler_type="cosine", logging_steps=10, output_dir="out", bf16=True)) trainer.train() model.save_pretrained_gguf("out_gguf", tokenizer, quantization_method="q4_k_m")

The knobs, decoded: r=16 is the adapter capacity (8–32 is the sane range; higher learns more and forgets more), 2 epochs is the default for behavior tuning, and effective batch = 2×8 = 16 via gradient accumulation. This configuration uses roughly 16–20GB of VRAM.

Step 4 — Train and read the loss (30–90 min)

$ python train.py
  • Healthy run: loss starts ~1.5–2.5 and declines smoothly toward ~0.5–1.0.
  • Loss flat from the start → data format problem (usually the chat template mapping).
  • Loss near zero → memorization; the model will parrot training examples. Fewer epochs.

Step 5 — Load into Ollama (5 min)

The script already exported a merged Q4_K_M GGUF into out_gguf/. Create a Modelfile:

FROM ./out_gguf/unsloth.Q4_K_M.gguf
PARAMETER num_ctx 8192
$ ollama create acme-assistant -f Modelfile
$ ollama run acme-assistant

Your model now works everywhere Ollama does — including the Open WebUI setup from our first tutorial and the vLLM server (use the merged FP16 export + AWQ for that path).

Step 6 — Evaluate honestly (the step everyone skips)

  • Run your held-out eval.jsonl questions against BOTH the base model and your fine-tune. Count: target behavior achieved? format correct?
  • Then ask both models 10 GENERAL questions unrelated to your domain — if the tuned model got notably worse at them, that's catastrophic forgetting: retrain with fewer epochs or lower r.
  • Keep the eval set forever: rerun it on every future retrain and base-model upgrade.

Troubleshooting

  • CUDA out of memory — lower max_seq_length to 1024, then batch size to 1 (raise gradient_accumulation_steps to keep effective batch ~16).
  • Model answers in the wrong format after tuning — inconsistent training data, or you served it with a different system prompt than the one trained on. Align them.
  • It parrots training examples verbatim — overfitting: 1 epoch, or more diverse data.
  • No 24GB GPU at hand — the whole tutorial runs identically on a rented cloud GPU: a 1–2 hour session costs a couple of euros.

Where to go next