STLM Engineering Report: Dropout

Dylan Hillier, Leon Guertler, Bobby Cheng, Cheston Tan
9/9/2024
cs.CL

No official code found. Best implementation: LeonGuertler/SuperTinyLanguageModels (31 stars, MIT, last commit 2025-09-03).

Abstract

In this work we explore the relevance of dropout for modern language models, particularly in the context of models on the scale of <100M parameters. We explore it's relevance firstly in the regime of improving the sample efficiency of models given small, high quality datasets, and secondly in the regime of improving the quality of its fit on larger datasets where models may underfit. We find that concordant with conventional wisdom, dropout remains effective in the overfitting scenario, and that furthermore it may have some relevance for improving the fit of models even in the case of excess data, as suggested by previous research. In the process we find that the existing explanation for the mechanism behind this performance gain is not applicable in the case of language modelling.

View on arXivView PDF

Code Implementations(1)

LeonGuertler/SuperTinyLanguageModelsCommunity100%
MIT
316PythonMar 14, 20241 years agoMIT

Similar Papers

Drop Dropout on Single-Epoch Language Model Pretraining

85% match2025

Investigating the Synergistic Effects of Dropout and Residual Connections on Language Model Training

84% match2024

LoRA Meets Dropout under a Unified Framework

82% match2024

Dropout Robustness and Cognitive Profiling of Transformer Models via Stochastic Inference

81% match1 repo2026

How Bad is Training on Synthetic Data? A Statistical Analysis of Language Model Collapse

79% match2024

Do Neural Topic Models Really Need Dropout? Analysis of the Effect of Dropout in Topic Modeling

79% match1 repo42023

Cite this paper

@article{hillier2024stlm,
  title  = {STLM Engineering Report: Dropout},
  author = {Dylan Hillier and Leon Guertler and Bobby Cheng and Cheston Tan},
  year   = {2024},
  eprint = {2409.05423},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2409.05423v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2409.05423.svg)](https://codeofpaper.com/paper/2409.05423)

Discussion