Mixed-Precision Training for NLP and Speech Recognition with OpenSeq2Seq

Oleksii Kuchaiev, Boris Ginsburg, Igor Gitman, Vitaly Lavrukhin, Jason Li +3 more
5/25/2018
cs.CL

Official code: NVIDIA/OpenSeq2Seq (1,560 stars, Apache-2.0, TensorFlow, last commit 2021-05-11).

Abstract

We present OpenSeq2Seq - a TensorFlow-based toolkit for training sequence-to-sequence models that features distributed and mixed-precision training. Benchmarks on machine translation and speech recognition tasks show that models built using OpenSeq2Seq give state-of-the-art performance at 1.5-3x less training time. OpenSeq2Seq currently provides building blocks for models that solve a wide range of tasks including neural machine translation, automatic speech recognition, and speech synthesis.

View on arXivView PDF

Code Implementations(1)

NVIDIA/OpenSeq2SeqOfficial100%
TensorFlowApache-2.0

Toolkit for efficient experimentation with Speech Recognition, Text2Speech and NLP

1,560372C++, PerlSep 8, 20175 years agoApache-2.0
deep-learningfloat16language-modelmixed-precisionmulti-gpu+9 more

Similar Papers

IndicBART: A Pre-trained Model for Indic Natural Language Generation

87% match1 repo522021

Training Tips for the Transformer Model

85% match1 repo17.5k2018

Attention Is All You Need

84% match1 repo17.5k2017

Reproducing Whisper-Style Training Using an Open-Source Toolkit and Publicly Available Data

84% match1 repo10.0k2023

XPhoneBERT: A Pre-trained Multilingual Model for Phoneme Representations for Text-to-Speech

84% match1 repo3582023

Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages

83% match2023

Cite this paper

@article{kuchaiev2018mixedprecision,
  title  = {Mixed-Precision Training for NLP and Speech Recognition with OpenSeq2Seq},
  author = {Oleksii Kuchaiev and Boris Ginsburg and Igor Gitman and Vitaly Lavrukhin and Jason Li and Huyen Nguyen and Carl Case and Paulius Micikevicius},
  year   = {2018},
  eprint = {1805.10387},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/1805.10387}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/1805.10387.svg)](https://codeofpaper.com/paper/1805.10387)

Discussion