Flexible and Effective Mixing of Large Language Models into a Mixture of Domain Experts

Rhui Dih Lee, Laura Wynter, Raghu Kiran Ganti

8/30/2024

cs.AIcs.CL

No confident code implementation found for this paper. 3 possible matches can be shown below.

Abstract

We present a toolkit for creating low-cost Mixture-of-Domain-Experts (MOE) from trained models. The toolkit can be used for creating a mixture from models or from adapters. We perform extensive tests and offer guidance on defining the architecture of the resulting MOE using the toolkit. A public repository is available.

View on arXiv View PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. 3 weaker matches are hidden by default — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

A Survey on Mixture of Experts in Large Language Models

87% match1 repo5082024

Mixture of Experts in Large Language Models

87% match1 repo5082025

Nexus: Specialization meets Adaptability for Efficiently Training Mixture of Experts

86% match2024

HMoE: Heterogeneous Mixture of Experts for Language Modeling

86% match2024

Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Experts

85% match2024

Scaling Fine-Grained MoE Beyond 50B Parameters: Empirical Evaluation and Practical Insights

85% match2025

Cite this paper

@article{lee2024flexible,
  title  = {Flexible and Effective Mixing of Large Language Models into a Mixture of Domain Experts},
  author = {Rhui Dih Lee and Laura Wynter and Raghu Kiran Ganti},
  year   = {2024},
  eprint = {2408.17280},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2408.17280v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

[![Code of Paper](https://codeofpaper.com/badge/2408.17280.svg)](https://codeofpaper.com/paper/2408.17280)