On the Difference of BERT-style and CLIP-style Text Encoders

Zhihong Chen, Guiming Hardy Chen, Shizhe Diao, Xiang Wan, Benyou Wang
6/6/2023
cs.CL

No official code found. Best implementation: zhjohnchan/bert-clip-synesthesia (14 stars, last commit 2023-06-07).

Abstract

Masked language modeling (MLM) has been one of the most popular pretraining recipes in natural language processing, e.g., BERT, one of the representative models. Recently, contrastive language-image pretraining (CLIP) has also attracted attention, especially its vision models that achieve excellent performance on a broad range of vision tasks. However, few studies are dedicated to studying the text encoders learned by CLIP. In this paper, we analyze the difference between BERT-style and CLIP-style text encoders from three experiments: (i) general text understanding, (ii) vision-centric text understanding, and (iii) text-to-image generation. Experimental analyses show that although CLIP-style text encoders underperform BERT-style ones for general text understanding tasks, they are equipped with a unique ability, i.e., synesthesia, for the cross-modal association, which is more similar to the senses of humans.

View on arXivView PDF

Code Implementations(1)

zhjohnchan/bert-clip-synesthesiaCommunity100%

[Findings of ACL-2023] This is the official implementation of On the Difference of BERT-style and CLIP-style Text Encoders.

140Shell, PythonDec 19, 20223 years ago

Similar Papers

Is Multimodal Vision Supervision Beneficial to Language?

87% match2023

CLIP4STR: A Simple Baseline for Scene Text Recognition with Pre-trained Vision-Language Model

86% match1 repo1532023

Is BERT Blind? Exploring the Effect of Vision-and-Language Pretraining on Visual Language Understanding

86% match1 repo212023

Enhancing Fine-Grained Vision-Language Pretraining with Negative Augmented Samples

85% match2024

Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs

85% match1 repo3642024

Text-Guided Semantic Image Encoder

85% match2025

Cite this paper

@article{chen2023difference,
  title  = {On the Difference of BERT-style and CLIP-style Text Encoders},
  author = {Zhihong Chen and Guiming Hardy Chen and Shizhe Diao and Xiang Wan and Benyou Wang},
  year   = {2023},
  eprint = {2306.03678},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2306.03678v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2306.03678.svg)](https://codeofpaper.com/paper/2306.03678)

Discussion