Video captioning with recurrent networks based on frame- and video-level features and visual content classification

Rakshith Shetty, Jorma Laaksonen
12/9/2015
cs.CV

Official code: aalto-cbir/neuraltalkTheano (8 stars, MIT, last commit 2016-04-25).

Abstract

In this paper, we describe the system for generating textual descriptions of short video clips using recurrent neural networks (RNN), which we used while participating in the Large Scale Movie Description Challenge 2015 in ICCV 2015. Our work builds on static image captioning systems with RNN based language models and extends this framework to videos utilizing both static image features and video-specific features. In addition, we study the usefulness of visual content classifiers as a source of additional information for caption generation. With experimental results we show that utilizing keyframe based features, dense trajectory video features and content classifier outputs together gives better performance than any one of them individually.

View on arXivView PDF

Code Implementations(1)

aalto-cbir/neuraltalkTheanoOfficial100%
MIT
83CSS, HTMLDec 8, 201510 years agoMIT

Similar Papers

A Semantics-Assisted Video Captioning Model Trained with Scheduled Sampling

89% match1 repo552019

Fine-Grained Captioning of Long Videos through Scene Graph Consolidation

82% match2025

Bridging Vision and Language: Modeling Causality and Temporality in Video Narratives

82% match2024

EVC-MF: End-to-end Video Captioning Network with Multi-scale Features

82% match2024

Video ReCap: Recursive Captioning of Hour-Long Videos

82% match1 repo2082024

Video ReCap: Recursive Captioning of Hour-Long Videos

81% match1 repo2082024

Cite this paper

@article{shetty2015video,
  title  = {Video captioning with recurrent networks based on frame- and video-level features and visual content classification},
  author = {Rakshith Shetty and Jorma Laaksonen},
  year   = {2015},
  eprint = {1512.02949},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/1512.02949}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/1512.02949.svg)](https://codeofpaper.com/paper/1512.02949)

Discussion