Revisiting Offline Compression: Going Beyond Factorization-based Methods for Transformer Language Models

Mohammadreza Banaei, Klaudia Bałazy, Artur Kasymov, Rémi Lebret, Jacek Tabor +1 more
2/8/2023
cs.CL

No official code found. Best implementation: MohammadrezaBanaei/auto-encoder-based-transformer-compression (3 stars, MIT, last commit 2023-02-10).

Abstract

Recent transformer language models achieve outstanding results in many natural language processing (NLP) tasks. However, their enormous size often makes them impractical on memory-constrained devices, requiring practitioners to compress them to smaller networks. In this paper, we explore offline compression methods, meaning computationally-cheap approaches that do not require further fine-tuning of the compressed model. We challenge the classical matrix factorization methods by proposing a novel, better-performing autoencoder-based framework. We perform a comprehensive ablation study of our approach, examining its different aspects over a diverse set of evaluation settings. Moreover, we show that enabling collaboration between modules across layers by compressing certain modules together positively impacts the final model performance. Experiments on various NLP tasks demonstrate that our approach significantly outperforms commonly used factorization-based offline compression methods.

View on arXivView PDF

Code Implementations(1)

MohammadrezaBanaei/auto-encoder-based-transformer-compressionCommunity100%
MIT

[EACL 2023] Revisiting Offline Compression: Going Beyond Factorization-based Methods for Transformer Language Models

30PythonJan 25, 20233 years agoMIT
autoencoderbertcompressionfactorizationlanguage-model

Similar Papers

AA-SVD : Anchored and Adaptive SVD for Large Language Model Compression

88% match2026

Compressing Transformer Language Models via Matrix Product Operator Decomposition: A Case Study on PicoGPT

86% match1 repo32026

Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment

86% match2023

DSFormer: Effective Compression of Text-Transformers by Dense-Sparse Weight Factorization

85% match2023

Retrieval-based Knowledge Transfer: An Effective Approach for Extreme Large Language Model Compression

84% match2023

Projected Compression: Trainable Projection for Efficient Transformer Compression

84% match2025

Cite this paper

@article{banaei2023revisiting,
  title  = {Revisiting Offline Compression: Going Beyond Factorization-based Methods for Transformer Language Models},
  author = {Mohammadreza Banaei and Klaudia Bałazy and Artur Kasymov and Rémi Lebret and Jacek Tabor and Karl Aberer},
  year   = {2023},
  eprint = {2302.04045},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2302.04045v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2302.04045.svg)](https://codeofpaper.com/paper/2302.04045)

Discussion