ElasticTrainer: Speeding Up On-Device Training with Runtime Elastic Tensor Selection

Kai Huang, Boyuan Yang, Wei Gao
12/21/2023
cs.LGcs.AI

No official code found. Best implementation: pittisl/ElasticTrainer (14 stars, MIT, TensorFlow, last commit 2023-11-01).

Abstract

On-device training is essential for neural networks (NNs) to continuously adapt to new online data, but can be time-consuming due to the device's limited computing power. To speed up on-device training, existing schemes select trainable NN portion offline or conduct unrecoverable selection at runtime, but the evolution of trainable NN portion is constrained and cannot adapt to the current need for training. Instead, runtime adaptation of on-device training should be fully elastic, i.e., every NN substructure can be freely removed from or added to the trainable NN portion at any time in training. In this paper, we present ElasticTrainer, a new technique that enforces such elasticity to achieve the required training speedup with the minimum NN accuracy loss. Experiment results show that ElasticTrainer achieves up to 3.5x more training speedup in wall-clock time and reduces energy consumption by 2x-3x more compared to the existing schemes, without noticeable accuracy loss.

View on arXivView PDF

Code Implementations(1)

pittisl/ElasticTrainerCommunity100%
TensorFlowMIT

Code for paper "ElasticTrainer: Speeding Up On-Device Training with Runtime Elastic Tensor Selection" (MobiSys'23)

142Shell, PythonMar 31, 20232 years agoMIT
deep-learningneural-networkon-device-trainingtensorflow

Similar Papers

TODM: Train Once Deploy Many Efficient Supernet-Based RNN-T Compression For On-device ASR Models

85% match2023

SCoTTi: Save Computation at Training Time with an adaptive framework

84% match1 repo12023

System-level Impact of Non-Ideal Program-Time of Charge Trap Flash (CTF) on Deep Neural Network

83% match2024

MaxK-GNN: Extremely Fast GPU Kernel Design for Accelerating Graph Neural Networks Training

83% match1 repo422023

TsetlinWiSARD: On-Chip Training of Weightless Neural Networks using Tsetlin Automata on FPGAs

83% match2026

ZeroFL: Efficient On-Device Training for Federated Learning with Local Sparsity

83% match2022

Cite this paper

@article{huang2023elastictrainer,
  title  = {ElasticTrainer: Speeding Up On-Device Training with Runtime Elastic Tensor Selection},
  author = {Kai Huang and Boyuan Yang and Wei Gao},
  year   = {2023},
  eprint = {2312.14227},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2312.14227v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2312.14227.svg)](https://codeofpaper.com/paper/2312.14227)

Discussion