Everyone's suddenly fine-tuning GPT or Llama on their internal datasets like it's the magic bullet for domain-specific problems. I've watched three companies burn months and six figures on this. The math doesn't work at smaller scales.

You're better off with retrieval-augmented generation plus careful prompting. A $5k fine-tuning run on 10k examples gets you maybe 2-3% accuracy gains over a well-crafted system prompt with relevant context. The infrastructure overhead alone kills the ROI. You need proper eval frameworks, data cleanup pipelines, version control for training runs. Most teams don't have this.

Save fine-tuning for when you've already exhausted prompt engineering and you've got >100k high-quality labeled examples. Until then you're just optimizing the wrong thing.