Towards Benchmarking and Improving the Temporal Reasoning Capability of Large Language Models

Qingyu Tan, Hwee Tou Ng, Lidong Bing
6/15/2023
cs.CLcs.AI

Official code: DAMO-NLP-SG/TempReason (34 stars, last commit 2024-01-11).

Abstract

Reasoning about time is of fundamental importance. Many facts are time-dependent. For example, athletes change teams from time to time, and different government officials are elected periodically. Previous time-dependent question answering (QA) datasets tend to be biased in either their coverage of time spans or question types. In this paper, we introduce a comprehensive probing dataset \tempreason to evaluate the temporal reasoning capability of large language models. Our dataset includes questions of three temporal reasoning levels. In addition, we also propose a novel learning framework to improve the temporal reasoning capability of large language models, based on temporal span extraction and time-sensitive reinforcement learning. We conducted experiments in closed book QA, open book QA, and reasoning QA settings and demonstrated the effectiveness of our approach. Our code and data are released on https://github.com/DAMO-NLP-SG/TempReason.

View on arXivView PDF

Code Implementations(1)

DAMO-NLP-SG/TempReasonOfficial100%
342Shell, PythonMay 19, 20232 years ago

Similar Papers

Unlocking Temporal Question Answering for Large Language Models with Tailor-Made Reasoning Logic

90% match1 repo92023

MenatQA: A New Dataset for Testing the Temporal Comprehension and Reasoning Abilities of Large Language Models

90% match1 repo142023

TimeBench: A Comprehensive Evaluation of Temporal Reasoning Abilities in Large Language Models

88% match1 repo372023

Learning to Reason Over Time: Timeline Self-Reflection for Improved Temporal Reasoning in Language Models

88% match1 repo142025

Towards Robust Temporal Reasoning of Large Language Models via a Multi-Hop QA Dataset and Pseudo-Instruction Tuning

88% match1 repo22023

Enhancing Temporal Sensitivity and Reasoning for Time-Sensitive Question Answering

88% match2024

Cite this paper

@article{tan2023towards,
  title  = {Towards Benchmarking and Improving the Temporal Reasoning Capability of Large Language Models},
  author = {Qingyu Tan and Hwee Tou Ng and Lidong Bing},
  year   = {2023},
  eprint = {2306.08952},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2306.08952v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2306.08952.svg)](https://codeofpaper.com/paper/2306.08952)

Discussion