MS-DETR: Natural Language Video Localization with Sampling Moment-Moment Interaction

Jing Wang, Aixin Sun, Hao Zhang, Xiaoli Li
5/30/2023
cs.CV

No official code found. Best implementation: jingwangsg/MS-DETR (17 stars, last commit 2023-06-03).

Abstract

Given a query, the task of Natural Language Video Localization (NLVL) is to localize a temporal moment in an untrimmed video that semantically matches the query. In this paper, we adopt a proposal-based solution that generates proposals (i.e., candidate moments) and then select the best matching proposal. On top of modeling the cross-modal interaction between candidate moments and the query, our proposed Moment Sampling DETR (MS-DETR) enables efficient moment-moment relation modeling. The core idea is to sample a subset of moments guided by the learnable templates with an adopted DETR (DEtection TRansformer) framework. To achieve this, we design a multi-scale visual-linguistic encoder, and an anchor-guided moment decoder paired with a set of learnable templates. Experimental results on three public datasets demonstrate the superior performance of MS-DETR.

View on arXivView PDF

Code Implementations(1)

jingwangsg/MS-DETRCommunity100%

An official implementation for MS-DETR in ACL'23

170PythonMay 25, 20233 years ago

Similar Papers

DemaFormer: Damped Exponential Moving Average Transformer with Energy-Based Modeling for Temporal Language Grounding

90% match2023

Learning 2D Temporal Adjacent Networks for Moment Localization with Natural Language

90% match1 repo1.1k2019

The Devil is in the Spurious Correlations: Boosting Moment Retrieval with Dynamic Learning

89% match1 repo92025

Prompting Large Language Models to Reformulate Queries for Moment Localization

88% match2023

SCANet: Scene Complexity Aware Network for Weakly-Supervised Video Moment Retrieval

88% match1 repo572023

Background-aware Moment Detection for Video Moment Retrieval

87% match1 repo162023

Cite this paper

@article{wang2023msdetr,
  title  = {MS-DETR: Natural Language Video Localization with Sampling Moment-Moment Interaction},
  author = {Jing Wang and Aixin Sun and Hao Zhang and Xiaoli Li},
  year   = {2023},
  eprint = {2305.18969},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2305.18969v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2305.18969.svg)](https://codeofpaper.com/paper/2305.18969)

Discussion