IMU-1: Sample-Efficient Pre-training of Small Language Models

George Grigorev
1/25/2026
cs.LGcs.AI

No confident code implementation found for this paper. 2 possible matches can be shown below.

Abstract

We present IMU-1, a 430M-parameter language model trained on 72B tokens that approaches the benchmark performance of models trained on 56x more data. We describe a validated training recipe combining recent architectural interventions (QK-norm attention, per-head gating, value residuals, LayerNorm scaling) with optimization advances (NorMuon with cautious weight decay, muP parametrization) and a three-stage training schedule with post-hoc checkpoint EMA. We provide ablations for each component and release code, weights and data to enable reproduction: https://huggingface.co/thepowerfuldeez/imu1_base

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. 2 weaker matches are hidden by default — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

TinyHelen's First Curriculum: Training and Evaluating Tiny Language Models in a Simpler Language Environment

85% match1 repo152024

Stable Language Model Pre-training by Reducing Embedding Variability

85% match2024

Scaling Smart: Accelerating Large Language Model Pre-training with Small Model Initialization

85% match1 repo552024

Domain-Adaptive Continued Pre-Training of Small Language Models

85% match2025

1.5-Pints Technical Report: Pretraining in Days, Not Months -- Your Language Model Thrives on Quality Data

84% match1 repo3422024

Scalable Parameter and Memory Efficient Pretraining for LLM: Recent Algorithmic Advances and Benchmarking

84% match1 repo112025

Cite this paper

@article{grigorev2026imu,
  title  = {IMU-1: Sample-Efficient Pre-training of Small Language Models},
  author = {George Grigorev},
  year   = {2026},
  eprint = {2602.02522},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2602.02522v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2602.02522.svg)](https://codeofpaper.com/paper/2602.02522)

Discussion