Configurable Safety Tuning of Language Models with Synthetic Preference Data

Victor Gallego
3/30/2024
cs.CLcs.AI

Official code: vicgalle/configurable-safety-tuning (17 stars, MIT, last commit 2024-07-27).

Abstract

State-of-the-art language model fine-tuning techniques, such as Direct Preference Optimization (DPO), restrict user control by hard-coding predefined behaviors into the model. To address this, we propose a novel method, Configurable Safety Tuning (CST), that augments DPO using synthetic preference data to facilitate flexible safety configuration of LLMs at inference time. CST overcomes the constraints of vanilla DPO by introducing a system prompt specifying safety configurations, enabling LLM deployers to disable/enable safety preferences based on their need, just changing the system prompt. Our experimental evaluations indicate that CST successfully manages different safety configurations and retains the original functionality of LLMs, showing it is a robust method for configurable deployment. Data and models available at https://github.com/vicgalle/configurable-safety-tuning

View on arXivView PDF

Code Implementations(1)

vicgalle/configurable-safety-tuningOfficial100%
MIT

Data and models for the paper "Configurable Safety Tuning of Language Models with Synthetic Preference Data"

173PythonMar 29, 20242 years agoMIT
alignmentdpollmpreference-learningsafety

Similar Papers

S3LoRA: Safe Spectral Sharpness-Guided Pruning in Adaptation of Agent Planner

87% match2025

What Makes and Breaks Safety Fine-tuning? A Mechanistic Study

86% match1 repo122024

Safe Delta: Consistently Preserving Safety when Fine-Tuning LLMs on Diverse Datasets

85% match1 repo592026

Locking Down the Finetuned LLMs Safety

85% match1 repo112024

Controllable Safety Alignment: Inference-Time Adaptation to Diverse Safety Requirements

85% match2026

Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Models

84% match1 repo292026

Cite this paper

@article{gallego2024configurable,
  title  = {Configurable Safety Tuning of Language Models with Synthetic Preference Data},
  author = {Victor Gallego},
  year   = {2024},
  eprint = {2404.00495},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2404.00495v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2404.00495.svg)](https://codeofpaper.com/paper/2404.00495)

Discussion