MCTensor: A High-Precision Deep Learning Library with Multi-Component Floating-Point

Tao Yu, Wentao Guo, Jianan Canal Li, Tiancheng Yuan, Christopher De Sa
7/18/2022
cs.LGcs.MS

No official code found. Best implementation: ydtydr/MCTensor (8 stars, last commit 2022-08-26).

Abstract

In this paper, we introduce MCTensor, a library based on PyTorch for providing general-purpose and high-precision arithmetic for DL training. MCTensor is used in the same way as PyTorch Tensor: we implement multiple basic, matrix-level computation operators and NN modules for MCTensor with identical PyTorch interface. Our algorithms achieve high precision computation and also benefits from heavily-optimized PyTorch floating-point arithmetic. We evaluate MCTensor arithmetic against PyTorch native arithmetic for a series of tasks, where models using MCTensor in float16 would match or outperform the PyTorch model with float32 or float64 precision.

View on arXivView PDF

Code Implementations(1)

ydtydr/MCTensorCommunity100%

This is the official repo for MCTensor library

81Cython, PythonJul 18, 20224 years agoNOASSERTION

Similar Papers

Hawkeye: Reproducing GPU-Level Non-Determinism

85% match2026

Is Integer Arithmetic Enough for Deep Learning Training?

84% match2022

Executing Arithmetic: Fine-Tuning Large Language Models as Turing Machines

83% match1 repo112024

Fuzzy PyTorch: Rapid Numerical Variability Evaluation for Deep Learning Models

82% match2026

An Open-Source Framework for Efficient Numerically-Tailored Computations

82% match1 repo262024

BluTrain: A C++/CUDA Framework for AI Systems

82% match2026

Cite this paper

@article{yu2022mctensor,
  title  = {MCTensor: A High-Precision Deep Learning Library with Multi-Component Floating-Point},
  author = {Tao Yu and Wentao Guo and Jianan Canal Li and Tiancheng Yuan and Christopher De Sa},
  year   = {2022},
  eprint = {2207.08867},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2207.08867v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2207.08867.svg)](https://codeofpaper.com/paper/2207.08867)

Discussion