Comparing Fine-Tuning Strategies: Which Parts of a GPT Model Should We Update?

TL;DR I compared four ways of fine-tuning a small GPT model for SMS spam classification: Train only the classification head Train the final transformer block Train the final half of the transformer