Zero-Shot Video Captioning with Evolving Pseudo-Tokens

Yoad Tewel, Yoav Shalev, Roy Nadler, Idan Schwartz, Lior Wolf
7/22/2022
cs.CV

Official code: YoadTew/zero-shot-video-to-text (75 stars, last commit 2022-10-22).

Abstract

We introduce a zero-shot video captioning method that employs two frozen networks: the GPT-2 language model and the CLIP image-text matching model. The matching score is used to steer the language model toward generating a sentence that has a high average matching score to a subset of the video frames. Unlike zero-shot image captioning methods, our work considers the entire sentence at once. This is achieved by optimizing, during the generation process, part of the prompt from scratch, by modifying the representation of all other tokens in the prompt, and by repeating the process iteratively, gradually improving the specificity and comprehensiveness of the generated sentence. Our experiments show that the generated captions are coherent and display a broad range of real-world knowledge. Our code is available at: https://github.com/YoadTew/zero-shot-video-to-text

View on arXivView PDF

Code Implementations(1)

YoadTew/zero-shot-video-to-textOfficial100%
758PythonJul 15, 20224 years ago

Similar Papers

RETTA: Retrieval-Enhanced Test-Time Adaptation for Zero-Shot Video Captioning

93% match2024

Zero-Shot Dense Video Captioning by Jointly Optimizing Text and Moment

89% match2023

MultiCapCLIP: Auto-Encoding Prompts for Zero-Shot Multilingual Visual Captioning

88% match1 repo362023

MeaCap: Memory-Augmented Zero-shot Image Captioning

87% match1 repo562024

ZeroNLG: Aligning and Autoencoding Domains for Zero-Shot Multimodal and Multilingual Natural Language Generation

87% match1 repo222023

DRCap: Decoding CLAP Latents with Retrieval-Augmented Generation for Zero-shot Audio Captioning

85% match1 repo1.1k2024

Cite this paper

@article{tewel2022zeroshot,
  title  = {Zero-Shot Video Captioning with Evolving Pseudo-Tokens},
  author = {Yoad Tewel and Yoav Shalev and Roy Nadler and Idan Schwartz and Lior Wolf},
  year   = {2022},
  eprint = {2207.11100},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2207.11100v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2207.11100.svg)](https://codeofpaper.com/paper/2207.11100)

Discussion