A Gated Residual Kolmogorov-Arnold Networks for Mixtures of Experts

Hugo Inzirillo, Remi Genet
9/23/2024
cs.LGcs.NE

No official code found. Best implementation: remigenet/KAMoE (6 stars, last commit 2024-11-01).

Abstract

This paper introduces KAMoE, a novel Mixture of Experts (MoE) framework based on Gated Residual Kolmogorov-Arnold Networks (GRKAN). We propose GRKAN as an alternative to the traditional gating function, aiming to enhance efficiency and interpretability in MoE modeling. Through extensive experiments on digital asset markets and real estate valuation, we demonstrate that KAMoE consistently outperforms traditional MoE architectures across various tasks and model types. Our results show that GRKAN exhibits superior performance compared to standard Gating Residual Networks, particularly in LSTM-based models for sequential tasks. We also provide insights into the trade-offs between model complexity and performance gains in MoE and KAMoE architectures.

View on arXivView PDF

Code Implementations(1)

remigenet/KAMoECommunity100%

Kolmogorov Arnold Mixture of Experts

63Python, Jupyter NotebookJul 18, 20241 years agoNOASSERTION

Similar Papers

LLM-Based Routing in Mixture of Experts: A Novel Framework for Trading

82% match2025

On Expert Estimation in Hierarchical Mixture of Experts: Beyond Softmax Gating Functions

82% match2024

MoB: Mixture of Bidders

82% match2025

S'MoRE: Structural Mixture of Residual Experts for Parameter-Efficient LLM Fine-tuning

82% match1 repo62025

Mixture of Experts in Large Language Models

81% match1 repo5082025

HoME: Hierarchy of Multi-Gate Experts for Multi-Task Learning at Kuaishou

81% match2024

Cite this paper

@article{inzirillo2024gated,
  title  = {A Gated Residual Kolmogorov-Arnold Networks for Mixtures of Experts},
  author = {Hugo Inzirillo and Remi Genet},
  year   = {2024},
  eprint = {2409.15161},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2409.15161v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2409.15161.svg)](https://codeofpaper.com/paper/2409.15161)

Discussion