CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval

Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei +2 more
4/18/2021
cs.CV

Official code: ArrowLuo/CLIP4Clip (1,034 stars, MIT, last commit 2024-04-12).

Abstract

Video-text retrieval plays an essential role in multi-modal research and has been widely used in many real-world web applications. The CLIP (Contrastive Language-Image Pre-training), an image-language pre-training model, has demonstrated the power of visual concepts learning from web collected image-text datasets. In this paper, we propose a CLIP4Clip model to transfer the knowledge of the CLIP model to video-language retrieval in an end-to-end manner. Several questions are investigated via empirical studies: 1) Whether image feature is enough for video-text retrieval? 2) How a post-pretraining on a large-scale video-text dataset based on the CLIP affect the performance? 3) What is the practical mechanism to model temporal dependency between video frames? And 4) The Hyper-parameters sensitivity of the model on video-text retrieval task. Extensive experimental results present that the CLIP4Clip model transferred from the CLIP can achieve SOTA results on various video-text retrieval datasets, including MSR-VTT, MSVC, LSMDC, ActivityNet, and DiDeMo. We release our code at https://github.com/ArrowLuo/CLIP4Clip.

View on arXivView PDF

Code Implementations(1)

ArrowLuo/CLIP4ClipOfficial100%
MIT

An official implementation for "CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval"

1,034138PythonApr 13, 20212 years agoMIT
activitynetclipdidemolsmdcmsrvtt+10 more

Similar Papers

MV-Adapter: Multimodal Video Transfer Learning for Video Text Retrieval

90% match1 repo142024

Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval

90% match1 repo3802021

MV-Adapter: Multimodal Video Transfer Learning for Video Text Retrieval

89% match1 repo142023

M2-RAAP: A Multi-Modal Recipe for Advancing Adaptation-based Pre-training towards Effective and Efficient Zero-shot Video-text Retrieval

89% match1 repo1802024

RTQ: Rethinking Video-language Understanding Based on Image-text Model

89% match1 repo152023

Image-to-Video Transfer Learning based on Image-Language Foundation Models: A Comprehensive Survey

88% match2025

Cite this paper

@article{luo2021clipclip,
  title  = {CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval},
  author = {Huaishao Luo and Lei Ji and Ming Zhong and Yang Chen and Wen Lei and Nan Duan and Tianrui Li},
  year   = {2021},
  eprint = {2104.08860},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/2104.08860}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2104.08860.svg)](https://codeofpaper.com/paper/2104.08860)

Discussion