Context-DPO: Aligning Language Models for Context-Faithfulness

Baolong Bi, Shaohan Huang, Yiwei Wang, Tianchi Yang, Zihan Zhang +9 more
12/18/2024
cs.CLcs.AIcs.IR

Official code: byronBBL/Context-DPO (23 stars, last commit 2025-02-17).

Abstract

Reliable responses from large language models (LLMs) require adherence to user instructions and retrieved information. While alignment techniques help LLMs align with human intentions and values, improving context-faithfulness through alignment remains underexplored. To address this, we propose $\textbf{Context-DPO}$, the first alignment method specifically designed to enhance LLMs' context-faithfulness. We introduce $\textbf{ConFiQA}$, a benchmark that simulates Retrieval-Augmented Generation (RAG) scenarios with knowledge conflicts to evaluate context-faithfulness. By leveraging faithful and stubborn responses to questions with provided context from ConFiQA, our Context-DPO aligns LLMs through direct preference optimization. Extensive experiments demonstrate that our Context-DPO significantly improves context-faithfulness, achieving 35% to 280% improvements on popular open-source models. Further analysis demonstrates that Context-DPO preserves LLMs' generative capabilities while providing interpretable insights into context utilization. Our code and data are released at https://github.com/byronBBL/Context-DPO

View on arXivView PDF

Code Implementations(1)

byronBBL/Context-DPOOfficial100%

Official repository of paper "Context-DPO: Aligning Language Models for Context-Faithfulness"

230PythonDec 18, 20241 years ago

Similar Papers

SSFO: Self-Supervised Faithfulness Optimization for Retrieval-Augmented Generation

84% match1 repo222025

Insights into Alignment: Evaluating DPO and its Variants Across Multiple Tasks

84% match2024

Aligning Large Language Models with Counterfactual DPO

83% match2024

RADIANT: Retrieval AugmenteD entIty-context AligNmenT -- Introducing RAG-ability and Entity-Context Divergence

83% match2025

Hit-RAG: Learning to Reason with Long Contexts via Preference Alignment

82% match2026

When Does Context Help? Error Dynamics of Contextual Information in Large Language Models

82% match2026

Cite this paper

@article{bi2024contextdpo,
  title  = {Context-DPO: Aligning Language Models for Context-Faithfulness},
  author = {Baolong Bi and Shaohan Huang and Yiwei Wang and Tianchi Yang and Zihan Zhang and Haizhen Huang and Lingrui Mei and Junfeng Fang and Zehao Li and Furu Wei and Weiwei Deng and Feng Sun and Qi Zhang and Shenghua Liu},
  year   = {2024},
  eprint = {2412.15280},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2412.15280v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2412.15280.svg)](https://codeofpaper.com/paper/2412.15280)

Discussion