This paper has been accepted to the UniReps Workshop in NeurIPS 2023.
Contrastive language picture pretraining has turn out to be the usual method for coaching imaginative and prescient language fashions. Regardless of the utility of CLIP visible options as world representations for pictures, they’ve limitations relating to duties involving object localization, pixel-level understanding of the picture, or 3D notion. Multi-task coaching is a well-liked resolution to deal with this disadvantage, however amassing a large-scale annotated multi-task dataset incurs vital prices. Moreover, coaching on separate activity particular datasets can also be difficult from optimization and coaching perspective attributable to aligning gradients and information coming from completely different enter distributions and duties. To beat these shortcomings, we examine pseudo-labeling with task-specific specialists to enhance CLIP options for more difficult down-stream duties. In our method, we leverage a number of present open-source pretrained fashions and pseudo-label an uncurated web-scale image-caption dataset with the specialists. We then practice CLIP with contrastive loss and activity particular losses with pseudo labels by the lightweight heads that we connect to the imaginative and prescient spine.