OLMoE: Open Mixture-of-Experts Language Models

Niklas Muennighoff, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Jacob Morrison +19 more
9/3/2024
cs.CLcs.AIcs.LG
moe

No official code found. Best implementation: XueFuzhao/OpenMoE (1,695 stars, last commit 2024-03-08). 1 more below.

Abstract

We introduce OLMoE, a fully open, state-of-the-art language model leveraging sparse Mixture-of-Experts (MoE). OLMoE-1B-7B has 7 billion (B) parameters but uses only 1B per input token. We pretrain it on 5 trillion tokens and further adapt it to create OLMoE-1B-7B-Instruct. Our models outperform all available models with similar active parameters, even surpassing larger ones like Llama2-13B-Chat and DeepSeekMoE-16B. We present various experiments on MoE training, analyze routing in our model showing high specialization, and open-source all aspects of our work: model weights, training data, code, and logs.

View on arXivView PDF

Code Implementations(2)

allenai/OLMoECommunity100%
Apache-2.0

OLMoE: Open Mixture-of-Experts Language Models

1,091122Shell, PythonJun 22, 20241 years agoApache-2.0
XueFuzhao/OpenMoECommunity70%

A family of open-sourced Mixture-of-Experts (MoE) Large Language Models

1,69586Shell, PythonAug 8, 20232 years ago

Similar Papers

OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models

88% match1 repo1.7k2024

MoSE: Mixture of Slimmable Experts for Efficient and Adaptive Language Models

87% match1 repo22026

LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training

87% match1 repo1.0k2024

dots.llm1 Technical Report

87% match2025

FLAME-MoE: A Transparent End-to-End Research Platform for Mixture-of-Experts Language Models

87% match1 repo432025

Mixture of Experts in Large Language Models

86% match1 repo5082025

Cite this paper

@article{muennighoff2024olmoe,
  title  = {OLMoE: Open Mixture-of-Experts Language Models},
  author = {Niklas Muennighoff and Luca Soldaini and Dirk Groeneveld and Kyle Lo and Jacob Morrison and Sewon Min and Weijia Shi and Pete Walsh and Oyvind Tafjord and Nathan Lambert and Yuling Gu and Shane Arora and Akshita Bhagia and Dustin Schwenk and David Wadden and Alexander Wettig and Binyuan Hui and Tim Dettmers and Douwe Kiela and Ali Farhadi and Noah A. Smith and Pang Wei Koh and Amanpreet Singh and Hannaneh Hajishirzi},
  year   = {2024},
  eprint = {2409.02060},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2409.02060v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2409.02060.svg)](https://codeofpaper.com/paper/2409.02060)

Discussion