Kimad: Adaptive Gradient Compression with Bandwidth Awareness

Jihao Xin, Ivan Ilin, Shunkang Zhang, Marco Canini, Peter Richtárik
12/13/2023
cs.LGcs.DCcs.IT

No confident code implementation found for this paper.

Abstract

In distributed training, communication often emerges as a bottleneck. In response, we introduce Kimad, a solution that offers adaptive gradient compression. By consistently monitoring bandwidth, Kimad refines compression ratios to match specific neural network layer requirements. Our exhaustive tests and proofs confirm Kimad's outstanding performance, establishing it as a benchmark in adaptive compression for distributed deep learning.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

Contractive error feedback for gradient compression

87% match2023

Flexible Communication for Optimal Distributed Learning over Unpredictable Networks

87% match1 repo2023

GraVAC: Adaptive Compression for Communication-Efficient Distributed DL Training

86% match1 repo2023

AB-Training: A Communication-Efficient Approach for Distributed Low-Rank Learning

86% match2024

Adaptive Step-Size Methods for Compressed SGD

85% match2022

Beyond Throughput and Compression Ratios: Towards High End-to-end Utility of Gradient Compression

84% match2024

Cite this paper

@article{xin2023kimad,
  title  = {Kimad: Adaptive Gradient Compression with Bandwidth Awareness},
  author = {Jihao Xin and Ivan Ilin and Shunkang Zhang and Marco Canini and Peter Richtárik},
  year   = {2023},
  eprint = {2312.08053},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2312.08053v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2312.08053.svg)](https://codeofpaper.com/paper/2312.08053)

Discussion