SWALP : Stochastic Weight Averaging in Low-Precision Training

Guandao Yang, Tianyi Zhang, Polina Kirichenko, Junwen Bai, Andrew Gordon Wilson +1 more
4/26/2019
cs.LGcs.AIstat.ML

Official code: stevenygd/SWALP (60 stars, BSD-2-Clause, last commit 2019-05-11).

Abstract

Low precision operations can provide scalability, memory savings, portability, and energy efficiency. This paper proposes SWALP, an approach to low precision training that averages low-precision SGD iterates with a modified learning rate schedule. SWALP is easy to implement and can match the performance of full-precision SGD even with all numbers quantized down to 8 bits, including the gradient accumulators. Additionally, we show that SWALP converges arbitrarily close to the optimal solution for quadratic objectives, and to a noise ball asymptotically smaller than low precision SGD in strongly convex settings.

View on arXivView PDF

Code Implementations(1)

stevenygd/SWALPOfficial100%
BSD-2-Clause

Code for paper "SWALP: Stochastic Weight Averaging forLow-Precision Training".

605Shell, PythonJun 25, 20187 years agoBSD-2-Clause

Similar Papers

Quantized Training of Gradient Boosting Decision Trees

84% match2 repos18.6k2022

Low-Precision Arithmetic for Fast Gaussian Processes

84% match1 repo162022

Q-PACE: Dynamic Precision Allocation for Quantization-Aware Training

83% match2026

Oscillation-Reduced MXFP4 Training for Vision Transformers

82% match1 repo412025

QPyTorch: A Low-Precision Arithmetic Simulation Framework

81% match1 repo2902019

Bit-by-Bit: Progressive QAT Strategy with Outlier Channel Splitting for Stable Low-Bit LLMs

81% match2026

Cite this paper

@article{yang2019swalp,
  title  = {SWALP : Stochastic Weight Averaging in Low-Precision Training},
  author = {Guandao Yang and Tianyi Zhang and Polina Kirichenko and Junwen Bai and Andrew Gordon Wilson and Christopher De Sa},
  year   = {2019},
  eprint = {1904.11943},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/1904.11943}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/1904.11943.svg)](https://codeofpaper.com/paper/1904.11943)

Discussion