SummaryMixing: A Linear-Complexity Alternative to Self-Attention for Speech Recognition and Understanding

Titouan Parcollet, Rogier van Dalen, Shucong Zhang, Sourav Bhattacharya
7/12/2023
cs.CLcs.SDeess.AS

No official code found. Best implementation: SamsungLabs/SummaryMixing (7 stars, last commit 2025-06-24).

Abstract

Modern speech processing systems rely on self-attention. Unfortunately, token mixing with self-attention takes quadratic time in the length of the speech utterance, slowing down inference and training and increasing memory consumption. Cheaper alternatives to self-attention for ASR have been developed, but they fail to consistently reach the same level of accuracy. This paper, therefore, proposes a novel linear-time alternative to self-attention. It summarises an utterance with the mean over vectors for all time steps. This single summary is then combined with time-specific information. We call this method "SummaryMixing". Introducing SummaryMixing in state-of-the-art ASR models makes it feasible to preserve or exceed previous speech recognition performance while making training and inference up to 28% faster and reducing memory use by half.

View on arXivView PDF

Code Implementations(1)

SamsungLabs/SummaryMixingCommunity100%
70PythonJun 23, 20251 years agoNOASSERTION

Similar Papers

Linear Time Complexity Conformers with SummaryMixing for Streaming Speech Recognition

88% match1 repo72024

When to Use Efficient Self Attention? Profiling Text, Speech and Image Transformer Variants

86% match2023

FitHuBERT: Going Thinner and Deeper for Knowledge Distillation of Speech Self-Supervised Learning

84% match1 repo192022

Accelerating Transducers through Adjacent Token Merging

84% match2023

Fast-HuBERT: An Efficient Training Framework for Self-Supervised Speech Representation Learning

83% match1 repo1002023

MambAttention: Mamba with Multi-Head Attention for Generalizable Single-Channel Speech Enhancement

83% match1 repo352025

Cite this paper

@article{parcollet2023summarymixing,
  title  = {SummaryMixing: A Linear-Complexity Alternative to Self-Attention for Speech Recognition and Understanding},
  author = {Titouan Parcollet and Rogier van Dalen and Shucong Zhang and Sourav Bhattacharya},
  year   = {2023},
  eprint = {2307.07421},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2307.07421v3}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2307.07421.svg)](https://codeofpaper.com/paper/2307.07421)

Discussion