On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification

Yongliang Wu, Yizhou Zhou, Zhou Ziheng, Yingzhe Peng, Xinyu Ye +5 more
8/7/2025
cs.LG

Official code: yongliang-wu/DFT (1,037 stars, last commit 2026-01-04). 2 more implementations below.

Abstract

We present a simple yet theoretically motivated improvement to Supervised Fine-Tuning (SFT) for the Large Language Model (LLM), addressing its limited generalization compared to reinforcement learning (RL). Through mathematical analysis, we reveal that standard SFT gradients implicitly encode a problematic reward structure that may severely restrict the generalization capabilities of model. To rectify this, we propose Dynamic Fine-Tuning (DFT), stabilizing gradient updates for each token by dynamically rescaling the objective function with the probability of this token. Remarkably, this single-line code change significantly outperforms standard SFT across multiple challenging benchmarks and base models, demonstrating greatly improved generalization. Additionally, our approach shows competitive results in offline RL settings, offering an effective yet simpler alternative. This work bridges theoretical insight and practical solutions, substantially advancing SFT performance. The code will be available at https://github.com/yongliang-wu/DFT.

View on arXivView PDF

Code Implementations(3)

yongliang-wu/DFTOfficial100%

[ICLR 2026] On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification.

1,03726ANTLR, ShellAug 1, 20259 months ago
shreyansh26/dynamic-fine-tuningCommunity63%

A re-implementation of the Dynamic Fine-Tuning paper (On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification)

20Shell, PythonSep 14, 20251 years ago
chain-of-thoughtcotfine-tuningllmmath+2 more
Lauorie/DFTCommunity62%

Reproduced the DFT method without using Verl. https://arxiv.org/abs/2508.05629

242Jinja, ShellAug 11, 20251 years ago

Similar Papers

Anchored Supervised Fine-Tuning

91% match1 repo502025

Simplify RLHF as Reward-Weighted SFT: A Variational Method

88% match2025

One-Token Rollout: Guiding Supervised Fine-Tuning of LLMs with Policy Gradient

88% match2025

SRFT: A Single-Stage Method with Supervised and Reinforcement Fine-Tuning for Reasoning

88% match2025

Supervised Fine Tuning on Curated Data is Reinforcement Learning (and can be improved)

88% match2025

Nested-ReFT: Efficient Reinforcement Learning for Large Language Model Fine-Tuning via Off-Policy Rollouts

88% match2025

Cite this paper

@article{wu2025generalization,
  title  = {On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification},
  author = {Yongliang Wu and Yizhou Zhou and Zhou Ziheng and Yingzhe Peng and Xinyu Ye and Xinting Hu and Wenbo Zhu and Lu Qi and Ming-Hsuan Yang and Xu Yang},
  year   = {2025},
  eprint = {2508.05629},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2508.05629v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2508.05629.svg)](https://codeofpaper.com/paper/2508.05629)

Discussion