How to Fine-Tune an Open-Weights LLM: A Hands-On Guide Using Inkling

A fine-tune vs RAG vs prompt decision guide, with the ROI math most guides skip

tldr: Fine-tuning an LLM means continuing to train an existing model on your own data so it specializes in your task. With an open-weights model like Thinking Machines' Inkling (975B parameters, 41B active), you can do this with LoRA (on hosted fine-tuning infrastructure for the full model, or a single multi-GPU node for its smaller sibling) instead of retraining from scratch. But fine-tune only after prompt engineering and RAG have failed you; it's the most expensive and least flexible of the three, and most teams reach for it too early.

What is LLM fine-tuning?

Fine-tuning is additional training that adjusts a pretrained model's weights using examples of the behavior you want. The model already knows language and reasoning from pretraining; fine-tuning teaches it your task, format, or domain: the specialized diction of your field, a house response style, a structured output your prompts can't reliably produce.

There are two broad approaches. Full fine-tuning updates every weight; it's the most powerful and the most expensive, and for a 975B model it's out of reach for almost everyone. LoRA and other PEFT (parameter-efficient fine-tuning) methods freeze the base model and train small adapter matrices instead, a fraction of a percent of the parameters. You get most of the benefit at a small fraction of the compute and memory, which is why most practical fine-tunes in 2026 are LoRA.

LoRA fine-tuning vs full fine-tuning: frozen base weights with small swappable adapters versus updating all 975B parameters into a 2TB checkpoint

Should you fine-tune at all?

Most "how to fine-tune" guides skip this question entirely: fine-tuning is usually the wrong first move. Work down this ladder and stop at the first rung that solves your problem.

Decision tree for prompt engineering vs RAG vs fine-tuning an LLM: fix with prompts first, use RAG for missing knowledge, fine-tune only for stubborn behavior gaps

	Prompt engineering	RAG	Fine-tuning
Fixes	Instruction-following, format nudges	Missing / changing knowledge	Entrenched behavior, style, and format gaps
Cost to change	Edit a string	Update the index	Retrain
Time to first result	Minutes	Hours	Days
Handles fresh facts	No	Yes	No (frozen at training)
Recurring cost	Per-token only	Retrieval infra + tokens	Hosting the tuned model
When it's the right call	Almost always try first	Knowledge gaps, citations, freshness	Behavior/format that prompting can't hold, at volume

One distinction settles most of these debates: RAG changes what the model knows; fine-tuning changes how the model behaves. If your problem is "it doesn't know our internal docs," fine-tuning is the wrong tool; you'll bake in stale facts and still miss retrieval. If your problem is "it will not stop writing three paragraphs when we need one line, no matter how we prompt it," that's a fine-tune. Most teams that fine-tune for knowledge should have built RAG.

Why does a fine-tune-first open-weights model matter?

Thinking Machines released Inkling on July 15, 2026 as an open-weights model designed to be fine-tuned, and that design intent is the story. The specs that matter:

975B total parameters, 41B active: a Mixture-of-Experts model, so only a subset of parameters fire per token. You get large-model quality at mid-model inference cost.
Open weights (Apache 2.0): download it, run it on your own infrastructure, fine-tune it however you want. No per-token API tax on the base model, no terms limiting what you tune it for.
Up to 1M-token context and native multimodal input (text, image, audio).
Controllable thinking effort: you dial reasoning up or down for the cost/latency trade-off you need.
Inkling-Small (276B / 12B active) for teams that want the recipe at a lighter weight.

Open weights change the fine-tuning calculus specifically. With a closed API model you fine-tune inside the vendor's platform on the vendor's terms and pay to host the result there. With Inkling you own the artifact, and the ROI math later in this article treats it that way.

Prerequisites and environment setup

What you need before the first training step:

Hardware. Be clear-eyed about scale: even with LoRA, all 975B weights must be loaded, which is roughly 2TB in bf16, beyond any single node. The practical paths are Thinking Machines' hosted Tinker platform for full Inkling, or self-hosting Inkling-Small (276B total, 12B active) with quantized LoRA on a multi-GPU node.
Libraries. A current PyTorch, a PEFT/LoRA library, and the Inkling model weights or Tinker access.
A dataset. This is where fine-tunes succeed or fail. You want instruction/response pairs (or your task's equivalent) that demonstrate the exact behavior you're after, formatted consistently. As a rule of thumb, hundreds of clean, consistent examples beat tens of thousands of noisy ones.

Step by step: fine-tuning Inkling

The shape of the workflow. Treat the code as illustrative and confirm the exact APIs against current docs.

LLM fine-tuning workflow: dataset prep, LoRA config, training run, evaluation, deploy, with early-stopping and revisit-data feedback loops

Prepare the dataset. Convert your examples to the expected chat/JSONL format, split train/validation, then do the step people skip: manually read 50 random rows. Bad examples in are bad behavior out, and no amount of training fixes a dirty dataset.
Configure LoRA. Set rank, alpha, target modules, learning rate, and epochs. Start conservative (low rank, few epochs); over-training a LoRA is the most common way to wreck a model's general ability.
Run training. Launch the job, watch training and validation loss together. Training loss falling while validation loss rises is overfitting; stop early.
Evaluate. Score the tuned model on a held-out set your prompts and base model also ran, so you're measuring the delta the fine-tune bought. "It feels better" is not an eval; a fixed test set with a metric is.
Deploy. Merge or serve the adapter, put it behind your inference stack, and keep the base model available so you can A/B the tuned version against it in production.

# ILLUSTRATIVE: confirm API surface against current docs
from peft import LoraConfig, get_peft_model

lora_config = LoraConfig(
    r=16,                     # start low; raise only if under-fitting
    lora_alpha=32,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],  # confirm for Inkling's arch
    lora_dropout=0.05,
    task_type="CAUSAL_LM",
)
model = get_peft_model(base_model, lora_config)
# train loop: standard HF Trainer / SFT loop, watch val loss, stop early

When does fine-tuning pay off?

Here's the break-even framing most guides omit. Fine-tuning has a one-time cost (the training run) and a recurring cost (hosting the tuned model, or a per-token premium on a hosted platform). RAG and prompt engineering ride on your existing API spend. So fine-tuning pays off only when the per-request benefit (fewer tokens, higher success rate, or a capability you can't get otherwise) clears its recurring hosting cost across your request volume.

The rough decision:

Low volume, changing requirements → don't fine-tune. The training cost never amortizes and every requirement change means retraining. Prompt or RAG.
High volume, stable narrow task, prompting plateaus → fine-tune. If a tuned model cuts tokens-per-request or lifts success rate at millions of requests a month, the hosting cost is noise against the savings.
Need current facts → RAG, at any volume.

Plug your own numbers into the Gemini true-cost formula from our pricing breakdown to compare a tuned open-weights model's fully-loaded per-token cost against a hosted API; that's the comparison that decides it, and it usually hinges on your GPU utilization.

What are the most common fine-tuning mistakes?

Overfitting. Too many epochs or too high a rank; the model memorizes your training set and loses general ability. Watch validation loss and stop early.
A weak eval. Judging by vibes or by the same examples you trained on. Both lie. Hold out a real test set.
Data leakage. Test examples that overlap your training data inflate every score. De-duplicate across the split.
Fine-tuning for knowledge. The single most common category error: baking facts into weights where they go stale and can't be updated, when RAG would have kept them fresh and citable.
Skipping the baseline. If you never measured prompt-only and RAG performance, you can't prove the fine-tune bought anything.

FAQ

What's the difference between RAG and fine-tuning?

RAG retrieves relevant documents at query time and feeds them to the model, changing what it knows without touching its weights, ideal for facts that change or need citing. Fine-tuning retrains the weights on examples, changing how it behaves, ideal for stubborn format, style, or domain behavior. They solve different problems and are often used together.

How much does it cost to fine-tune an LLM?

With LoRA on an open-weights model, the training run is typically GPU-hours in the tens-to-hundreds of dollars for a focused task, not the millions that full pretraining costs. The bigger number is recurring: hosting the tuned model.

Can you fine-tune an open-weights model on one GPU?

For small models, yes. For frontier-scale open weights like Inkling, no: even LoRA needs all 975B parameters in memory, which means hosted platforms or a multi-GPU node for the smaller Inkling-Small variant. What LoRA saves you is the training compute and the cluster that full fine-tuning would demand.

What is LoRA fine-tuning?

LoRA (Low-Rank Adaptation) freezes the pretrained model and trains small "adapter" matrices injected into its layers, updating well under 1% of the parameters. You get most of full fine-tuning's benefit at a fraction of the compute, memory, and cost, and you can swap adapters without touching the base model.