Good Pretraining, Bad SFT: Checkpoint Quality Across the Training Stack

Sohir Maskey, Philipp Scholl, Jonas Knupp, Pit Neitemeier, Sascha Wirges
9/8/2026
cs.AIcs.CL

No confident code implementation found for this paper. 1 possible match can be shown below.

Abstract

Language-model checkpoints are commonly selected by pretraining loss or benchmark scores, assuming that the highest-scoring checkpoint will remain the best starting point for subsequent training. We show that this assumption can fail in a full 30B mixture-of-experts training pipeline. The checkpoints that perform better after the full downstream training stack also have higher solution density, i.e., retain downstream performance under local weight perturbations.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. 1 weaker match is hidden by default — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

The Development of Sycophancy in Language Models: Tracing Vulnerabilities from Pretraining to Post-Training

83% match2026

RL Excursions during Pre-Training: Re-examining Policy Optimization for LLM training

82% match2026

Trajectory Soup: Pushing the Compute-Scaling Frontier of LLM Mid-training via Diverse Trajectories

82% match2026

Instability in Downstream Task Performance During LLM Pretraining

82% match2025

Large Pre-Training Datasets Don't Always Guarantee Robustness after Fine-Tuning

82% match1 repo12024

RoBERTa: A Robustly Optimized BERT Pretraining Approach

82% match1 repo32.2k2019

Cite this paper

@article{maskey2026good,
  title  = {Good Pretraining, Bad SFT: Checkpoint Quality Across the Training Stack},
  author = {Sohir Maskey and Philipp Scholl and Jonas Knupp and Pit Neitemeier and Sascha Wirges},
  year   = {2026},
  eprint = {2609.08966},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2609.08966v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2609.08966.svg)](https://codeofpaper.com/paper/2609.08966)

Discussion