Abstract

Aligning image and text encoders from scratch using contrastive learning requires large amounts of paired image-text data. We alleviate this need by aligning individually pre-trained language and vision representation models using a much smaller amount of paired data with a curriculum learning algorithm to learn fine-grained vision-language alignments. TOnICS (Training with Ontology-Informed Contrastive Sampling) initially samples minibatches whose image-text pairs contain a wide variety of objects to learn object-level vision-language alignment, and progressively samples minibatches where all image-text pairs contain the same object to learn finer-grained contextual alignment. Aligning pre-trained BERT and VinVL-OD models to each other using TOnICS outperforms CLIP on downstream zero-shot image retrieval using< 1% as much training data.

Metadata

publication

Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern …, 2023

year

2023

publication date

2023

authors

Tejas Srinivasan, Xiang Ren, Jesse Thomason

link

https://openaccess.thecvf.com/content/CVPR2023W/O-DRUM/html/Srinivasan_Curriculum_Learning_for_Data-Efficient_Vision-Language_Alignment_CVPRW_2023_paper.html

resource_link

https://openaccess.thecvf.com/content/CVPR2023W/O-DRUM/papers/Srinivasan_Curriculum_Learning_for_Data-Efficient_Vision-Language_Alignment_CVPRW_2023_paper.pdf

conference

Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

pages

5619-5624