OmniLearn: A Framework for Distributed Deep Learning over Heterogeneous Clusters

Sahil Tyagi, Prateek Sharma
3/21/2025
cs.LG

No official code found. Best implementation: sahiltyagi4/OmniLearn (0 stars, MIT, last commit 2025-07-13).

Abstract

Deep learning systems are optimized for clusters with homogeneous resources. However, heterogeneity is prevalent in computing infrastructure across edge, cloud and HPC. When training neural networks using stochastic gradient descent techniques on heterogeneous resources, performance degrades due to stragglers and stale updates. In this work, we develop an adaptive batch-scaling framework called OmniLearn to mitigate the effects of heterogeneity in distributed training. Our approach is inspired by proportional controllers to balance computation across heterogeneous servers, and works under varying resource availability. By dynamically adjusting worker mini-batches at runtime, OmniLearn reduces training time by 14-85%. We also investigate asynchronous training, where our techniques improve accuracy by up to 6.9%.

View on arXivView PDF

Code Implementations(1)

sahiltyagi4/OmniLearnCommunity100%
MIT

OmniLearn: A Framework for Distributed Deep Learning over Heterogeneous Clusters

00Shell, PythonMar 11, 20251 years agoMIT

Similar Papers

A Dynamic Weighting Strategy to Mitigate Worker Node Failure in Distributed Deep Learning

83% match2024

Ravnest: Decentralized Asynchronous Training on Heterogeneous Devices

82% match2024

ScaDLES: Scalable Deep Learning over Streaming data at the Edge

82% match1 repo2023

Snake Learning: A Communication- and Computation-Efficient Distributed Learning Framework for 6G

82% match2024

ScaleFL: Resource-Adaptive Federated Learning with Heterogeneous Clients

82% match2026

A Survey of Distributed Learning in Cloud, Mobile, and Edge Settings

82% match2024

Cite this paper

@article{tyagi2025omnilearn,
  title  = {OmniLearn: A Framework for Distributed Deep Learning over Heterogeneous Clusters},
  author = {Sahil Tyagi and Prateek Sharma},
  year   = {2025},
  eprint = {2503.17469},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2503.17469v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2503.17469.svg)](https://codeofpaper.com/paper/2503.17469)

Discussion