Domino: Eliminating Communication in LLM Training via Generic Tensor Slicing and Overlapping

Guanhua Wang, Chengming Zhang, Zheyu Shen, Ang Li, Olatunji Ruwase
9/23/2024
cs.DCcs.AIcs.LG

No official code found. Best implementation: deepspeedai/DeepSpeed (43,229 stars, Apache-2.0, PyTorch, last commit 2026-04-29).

Abstract

Given the popularity of generative AI, Large Language Models (LLMs) often consume hundreds or thousands of GPUs for parallelizing and accelerating the training process. Communication overhead becomes more pronounced when training LLMs at scale. To eliminate communication overhead in distributed LLM training, we propose Domino, which provides a generic scheme to hide communication behind computation. By breaking data dependency of a single batch training into smaller independent pieces, Domino pipelines these independent pieces training and provides generic strategy of fine-grained communication and computation overlapping. Extensive results show that, comparing with Megatron-LM, Domino achieves up to 1.3x speedup for LLM training on Nvidia DGX-H100 GPUs.

View on arXivView PDF

Code Implementations(1)

deepspeedai/DeepSpeedCommunity100%
PyTorchApache-2.0

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

43,2295,039C, C++Jan 23, 20205 months agoApache-2.0
billion-parameterscompressiondata-parallelismdeep-learninggpu+8 more

Similar Papers

Tensor-Parallelism with Partially Synchronized Activations

83% match2025

Photon: Federated LLM Pre-Training

83% match2024

Towards Low-bit Communication for Tensor Parallel LLM Inference

82% match2024

ISO: Overlap of Computation and Communication within Seqenence For LLM Inference

82% match2024

ByteScale: Efficient Scaling of LLM Training with a 2048K Context Length on More Than 12,000 GPUs

82% match2025

SDP4Bit: Toward 4-bit Communication Quantization in Sharded Data Parallelism for LLM Training

82% match2 repos442024

Cite this paper

@article{wang2024domino,
  title  = {Domino: Eliminating Communication in LLM Training via Generic Tensor Slicing and Overlapping},
  author = {Guanhua Wang and Chengming Zhang and Zheyu Shen and Ang Li and Olatunji Ruwase},
  year   = {2024},
  eprint = {2409.15241},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2409.15241v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2409.15241.svg)](https://codeofpaper.com/paper/2409.15241)

Discussion