Distance between Relevant Information Pieces Causes Bias in Long-Context LLMs

Runchu Tian, Yanghao Li, Yuepeng Fu, Siyang Deng, Qinyu Luo +8 more
10/18/2024
cs.CLcs.AI

No official code found. Best implementation: Rachum-thu/LongPiBench (14 stars, MIT, last commit 2024-12-16).

Abstract

Positional bias in large language models (LLMs) hinders their ability to effectively process long inputs. A prominent example is the "lost in the middle" phenomenon, where LLMs struggle to utilize relevant information situated in the middle of the input. While prior research primarily focuses on single pieces of relevant information, real-world applications often involve multiple relevant information pieces. To bridge this gap, we present LongPiBench, a benchmark designed to assess positional bias involving multiple pieces of relevant information. Thorough experiments are conducted with five commercial and six open-source models. These experiments reveal that while most current models are robust against the "lost in the middle" issue, there exist significant biases related to the spacing of relevant information pieces. These findings highlight the importance of evaluating and reducing positional biases to advance LLM's capabilities.

View on arXivView PDF

Code Implementations(1)

Rachum-thu/LongPiBenchCommunity100%
MIT

The repository for papaer "Distance between Relevant Information Pieces Causes Bias in Long-Context LLMs"

141Shell, PythonOct 15, 20241 years agoMIT

Similar Papers

Positional Biases Shift as Inputs Approach Context Window Limits

91% match2025

Position-Aware Parameter Efficient Fine-Tuning Approach for Reducing Positional Bias in LLMs

91% match2024

Mitigate Position Bias in Large Language Models via Scaling a Single Dimension

89% match1 repo122024

Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization

89% match2024

Insights into LLM Long-Context Failures: When Transformers Know but Don't Tell

86% match1 repo192024

Why Does the Effective Context Length of LLMs Fall Short?

85% match1 repo822026

Cite this paper

@article{tian2024distance,
  title  = {Distance between Relevant Information Pieces Causes Bias in Long-Context LLMs},
  author = {Runchu Tian and Yanghao Li and Yuepeng Fu and Siyang Deng and Qinyu Luo and Cheng Qian and Shuo Wang and Xin Cong and Zhong Zhang and Yesai Wu and Yankai Lin and Huadong Wang and Xiaojiang Liu},
  year   = {2024},
  eprint = {2410.14641},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2410.14641v3}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2410.14641.svg)](https://codeofpaper.com/paper/2410.14641)

Discussion