SDBERT: SparseDistilBERT, a faster and smaller BERT model

Devaraju Vinoda, Pawan Kumar Yadav
7/28/2022
cs.CLcs.ITcs.LG

No confident code implementation found for this paper.

Abstract

In this work we introduce a new transformer architecture called SparseDistilBERT (SDBERT), which is a combination of sparse attention and knowledge distillantion (KD). We implemented sparse attention mechanism to reduce quadratic dependency on input length to linear. In addition to reducing computational complexity of the model, we used knowledge distillation (KD). We were able to reduce the size of BERT model by 60% while retaining 97% performance and it only took 40% of time to train.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

MaTVLM: Hybrid Mamba-Transformer for Efficient Vision-Language Modeling

87% match1 repo622025

TinyViT: Fast Pretraining Distillation for Small Vision Transformers

86% match2 repos1.8k2022

AccelTran: A Sparsity-Aware Accelerator for Dynamic Inference with Transformers

86% match1 repo622023

ReduceFormer: Attention with Tensor Reduction by Summation

86% match2024

Efficient Long-Range Transformers: You Need to Attend More, but Not Necessarily at Every Layer

85% match2023

Fourier Transformer: Fast Long Range Modeling by Removing Sequence Redundancy with FFT Operator

85% match1 repo402023

Cite this paper

@article{vinoda2022sdbert,
  title  = {SDBERT: SparseDistilBERT, a faster and smaller BERT model},
  author = {Devaraju Vinoda and Pawan Kumar Yadav},
  year   = {2022},
  eprint = {2208.10246},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2208.10246v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2208.10246.svg)](https://codeofpaper.com/paper/2208.10246)

Discussion