Align on the Fly: Adapting Chatbot Behavior to Established Norms

Chunpu Xu, Steffi Chern, Ethan Chern, Ge Zhang, Zekun Wang +4 more
12/26/2023
cs.CL

Official code: GAIR-NLP/OPO (50 stars, last commit 2024-03-02).

Abstract

In this paper, we aim to align large language models with the ever-changing, complex, and diverse human values (e.g., social norms) across time and locations. This presents a challenge to existing alignment techniques, such as supervised fine-tuning, which internalize values within model parameters. To overcome this, we propose an On-the-fly Preference Optimization (OPO) method, which is a real-time alignment that works in a streaming way. It employs an external memory to store established rules for alignment, which can constrain LLMs' behaviors without further training, allowing for convenient updates and customization of human values. We also introduce a scalable evaluation to assess the proposed method more effectively. Experimental results on both human-annotated and auto-generated questions from legal and moral domains indicate the effectiveness of the proposed OPO method. Our code and data are released at https://github.com/GAIR-NLP/OPO.

View on arXivView PDF

Code Implementations(1)

GAIR-NLP/OPOOfficial100%
509PythonDec 24, 20232 years ago

Similar Papers

Aligning CodeLLMs with Direct Preference Optimization

88% match2024

OmniAlign-V: Towards Enhanced Alignment of MLLMs with Human Preference

88% match1 repo1442025

Aligning Large Language Models with Human: A Survey

88% match1 repo7422023

Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges

87% match2025

She had Cobalt Blue Eyes: Prompt Testing to Create Aligned and Sustainable Language Models

87% match2023

A Survey on Personalized Alignment -- The Missing Piece for Large Language Models in Real-World Applications

86% match2025

Cite this paper

@article{xu2023align,
  title  = {Align on the Fly: Adapting Chatbot Behavior to Established Norms},
  author = {Chunpu Xu and Steffi Chern and Ethan Chern and Ge Zhang and Zekun Wang and Ruibo Liu and Jing Li and Jie Fu and Pengfei Liu},
  year   = {2023},
  eprint = {2312.15907},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2312.15907v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2312.15907.svg)](https://codeofpaper.com/paper/2312.15907)

Discussion