Rethinking Optimization and Architecture for Tiny Language Models

Yehui Tang, Fangcheng Liu, Yunsheng Ni, Yuchuan Tian, Zheyuan Bai +5 more
2/12/2026

No official code found. Best implementation: YuchuanTian/RethinkTinyLM (127 stars, last commit 2025-01-14).

DOISemantic Scholar

Code Implementations(1)

YuchuanTian/RethinkTinyLMCommunity63%

[ICML'24] The official implementation of “Rethinking Optimization and Architecture for Tiny Language Models”

1276PythonFeb 5, 20241 years ago

Cite this paper

@article{tang2026rethinking,
  title  = {Rethinking Optimization and Architecture for Tiny Language Models},
  author = {Yehui Tang and Fangcheng Liu and Yunsheng Ni and Yuchuan Tian and Zheyuan Bai and Yi-Qi Hu and Sichao Liu and Shangling Jui and Kai Han and Yunhe Wang},
  year   = {2026},
  doi    = {10.48550/arXiv.2402.02791},
  url    = {https://doi.org/10.48550/arXiv.2402.02791},
  journal = {ICML 2024 2024}
}

Discussion