The within-batch negatives deserve attention when annotated class names and noisy alt text share a batch. Two different images with the same class label are not semantically unrelated, yet a contrastive objective can treat their pairings as negatives. Captioning supervision is asking a different question about those same examples.

I would keep that tension in mind when interpreting the loss-weight ablation: more captioning weight may change which distinctions the representation preserves, rather than simply adding finer detail everywhere. Comparing same-class retrieval errors across the loss settings would be a useful complement to the aggregate zero-shot classification result.