DeInfoReg: A Decoupled Learning Framework for Better Training Throughput

Zih-Hao Huang, You-Teng Lin, Hung-Hsuan Chen
6/22/2025
cs.LGcs.AIcs.DC

Official code: ianzih/Decoupled-Supervised-Learning-for-Information-Regularization (1 star, last commit 2024-08-01).

Abstract

This paper introduces Decoupled Supervised Learning with Information Regularization (DeInfoReg), a novel approach that transforms a long gradient flow into multiple shorter ones, thereby mitigating the vanishing gradient problem. Integrating a pipeline strategy, DeInfoReg enables model parallelization across multiple GPUs, significantly improving training throughput. We compare our proposed method with standard backpropagation and other gradient flow decomposition techniques. Extensive experiments on diverse tasks and datasets demonstrate that DeInfoReg achieves superior performance and better noise resistance than traditional BP models and efficiently utilizes parallel computing resources. The code for reproducibility is available at: https://github.com/ianzih/Decoupled-Supervised-Learning-for-Information-Regularization/.

View on arXivView PDF

Code Implementations(1)

ianzih/Decoupled-Supervised-Learning-for-Information-RegularizationOfficial100%
10PythonJul 29, 20232 years ago

Similar Papers

SCPL: Enhancing Neural Network Training Throughput with Decoupled Local Losses and Model Parallelism

82% match2026

Distributed Deep Learning using Stochastic Gradient Staleness

81% match2025

Asynchronous Stochastic Gradient Descent with Decoupled Backpropagation and Layer-Wise Updates

80% match2024

Knowledge Distillation: Enhancing Neural Network Compression with Integrated Gradients

79% match2025

Learning without Global Backpropagation via Synergistic Information Distillation

79% match2025

Accelerating Large Batch Training via Gradient Signal to Noise Ratio (GSNR)

79% match2023

Cite this paper

@article{huang2025deinforeg,
  title  = {DeInfoReg: A Decoupled Learning Framework for Better Training Throughput},
  author = {Zih-Hao Huang and You-Teng Lin and Hung-Hsuan Chen},
  year   = {2025},
  eprint = {2506.18193},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2506.18193v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2506.18193.svg)](https://codeofpaper.com/paper/2506.18193)

Discussion