Static Batching of Irregular Workloads on GPUs: Framework and Application to Efficient MoE Model Inference

Yinghan Li, Yifei Li, Jiejing Zhang, Bujiao Chen, Xiaotong Chen +11 more
1/27/2025
cs.DCcs.LG
moe

No confident code implementation found for this paper. 2 possible matches can be shown below.

Abstract

It has long been a problem to arrange and execute irregular workloads on massively parallel devices. We propose a general framework for statically batching irregular workloads into a single kernel with a runtime task mapping mechanism on GPUs. We further apply this framework to Mixture-of-Experts (MoE) model inference and implement an optimized and efficient CUDA kernel. Our MoE kernel achieves up to 91% of the peak Tensor Core throughput on NVIDIA H800 GPU and 95% on NVIDIA H20 GPU.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. 2 weaker matches are hidden by default — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

A Scheduling Framework for Efficient MoE Inference on Edge GPU-NDP Systems

85% match2026

Accelerating MoE Model Inference with Expert Sharding

85% match2025

MoE-Gen: High-Throughput MoE Inference on a Single GPU with Module-Based Batching

85% match2025

MoE-Inference-Bench: Performance Evaluation of Mixture of Expert Large Language and Vision Models

83% match2025

HybriMoE: Hybrid CPU-GPU Scheduling and Cache Management for Efficient MoE Inference

82% match1 repo1182025

Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models

82% match1 repo2682024

Cite this paper

@article{li2025static,
  title  = {Static Batching of Irregular Workloads on GPUs: Framework and Application to Efficient MoE Model Inference},
  author = {Yinghan Li and Yifei Li and Jiejing Zhang and Bujiao Chen and Xiaotong Chen and Lian Duan and Yejun Jin and Zheng Li and Xuanyu Liu and Haoyu Wang and Wente Wang and Yajie Wang and Jiacheng Yang and Peiyang Zhang and Laiwen Zheng and Wenyuan Yu},
  year   = {2025},
  eprint = {2501.16103},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2501.16103v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2501.16103.svg)](https://codeofpaper.com/paper/2501.16103)

Discussion