Universal Language Model Fine-tuning for Text Classification

Jeremy Howard, Sebastian Ruder
1/18/2018
cs.CLcs.LGstat.ML

Official code: fastai/fastai (28,227 stars, Apache-2.0, PyTorch, last commit 2026-09-21).

Abstract

Inductive transfer learning has greatly impacted computer vision, but existing approaches in NLP still require task-specific modifications and training from scratch. We propose Universal Language Model Fine-tuning (ULMFiT), an effective transfer learning method that can be applied to any task in NLP, and introduce techniques that are key for fine-tuning a language model. Our method significantly outperforms the state-of-the-art on six text classification tasks, reducing the error by 18-24% on the majority of datasets. Furthermore, with only 100 labeled examples, it matches the performance of training from scratch on 100x more data. We open-source our pretrained models and code.

View on arXivView PDF

Code Implementations(1)

fastai/fastaiOfficial100%
PyTorchApache-2.0

The fastai deep learning library

28,2277,642CSS, PythonSep 9, 20174 weeks agoApache-2.0
colabdeep-learningfastaigpumachine-learning+3 more

Similar Papers

Supervised Fine-tuning in turn Improves Visual Foundation Models

88% match1 repo382024

Skip Tuning: Pre-trained Vision-Language Models are Effective and Efficient Adapters Themselves

88% match1 repo312026

Less Could Be Better: Parameter-efficient Fine-tuning Advances Medical Vision Foundation Models

87% match1 repo232024

Conditional Prototype Rectification Prompt Learning

86% match1 repo62024

Regularized Mask Tuning: Uncovering Hidden Knowledge in Pre-trained Vision-Language Models

86% match2026

IAP: Improving Continual Learning of Vision-Language Models via Instance-Aware Prompting

86% match1 repo52025

Cite this paper

@article{howard2018universal,
  title  = {Universal Language Model Fine-tuning for Text Classification},
  author = {Jeremy Howard and Sebastian Ruder},
  year   = {2018},
  eprint = {1801.06146},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/1801.06146}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/1801.06146.svg)](https://codeofpaper.com/paper/1801.06146)

Discussion