Knowledge distillation

Knowledge distillation involves using a large teacher model to train a smaller student model. The student model not only learns from the correct labels but also from the teacher's output distribution.