LifelongMemory: Leveraging LLMs for Answering Queries in Long-form Egocentric Videos

Ying Wang, Yanlai Yang, Mengye Ren
12/7/2023
cs.CVcs.LG

Official code: agentic-learning-ai-lab/lifelong-memory (33 stars, MIT, last commit 2025-10-27). 1 more implementation below.

Abstract

In this paper we introduce LifelongMemory, a new framework for accessing long-form egocentric videographic memory through natural language question answering and retrieval. LifelongMemory generates concise video activity descriptions of the camera wearer and leverages the zero-shot capabilities of pretrained large language models to perform reasoning over long-form video context. Furthermore, LifelongMemory uses a confidence and explanation module to produce confident, high-quality, and interpretable answers. Our approach achieves state-of-the-art performance on the EgoSchema benchmark for question answering and is highly competitive on the natural language query (NLQ) challenge of Ego4D. Code is available at https://github.com/agentic-learning-ai-lab/lifelong-memory.

View on arXivView PDF

Code Implementations(2)

agentic-learning-ai-lab/lifelong-memoryOfficial100%
MIT

Code for LifelongMemory: Leveraging LLMs for Answering Queries in Long-form Egocentric Videos

330PythonDec 13, 202311 months agoMIT
llmmultimodalnatural-language-queriesvideo-understanding
supmo668/VideoAgentCommunity62%

Based on the idea of VideoAgent: Long-form Video Understanding with Large Language Model as Agent (https://github.com/wxh1996/VideoAgent/tree/master) and LifelongMemory: Leveraging LLMs for Answering Queries in Long-form Egocentric Videos (https://agenticlearning.ai/lifelong-memory/)

20Shell, PythonDec 27, 20241 years ago

Similar Papers

Graph it first! Enabling Reasoning on Long-form Egocentric Videos through Scene Graphs

88% match2026

Long Video Understanding with Learnable Retrieval in Video-Language Models

88% match2023

Too Many Frames, Not All Useful: Efficient Strategies for Long-Form Video QA

87% match1 repo442024

FocusGraph: Graph-Structured Frame Selection for Embodied Long Video Question Answering

87% match2026

AVATAAR: Agentic Video Answering via Temporal Adaptive Alignment and Reasoning

87% match2025

EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding

87% match1 repo1222023

Cite this paper

@article{wang2023lifelongmemory,
  title  = {LifelongMemory: Leveraging LLMs for Answering Queries in Long-form Egocentric Videos},
  author = {Ying Wang and Yanlai Yang and Mengye Ren},
  year   = {2023},
  eprint = {2312.05269},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2312.05269v3}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2312.05269.svg)](https://codeofpaper.com/paper/2312.05269)

Discussion