Keyword Transformer: A Self-Attention Model for Keyword Spotting

Axel Berg, Mark O'Connor, Miguel Tairum Cruz
4/1/2021
eess.AScs.CLcs.LG

Official code: ARM-software/keyword-transformer (148 stars, Apache-2.0, last commit 2022-04-29).

Abstract

The Transformer architecture has been successful across many domains, including natural language processing, computer vision and speech recognition. In keyword spotting, self-attention has primarily been used on top of convolutional or recurrent encoders. We investigate a range of ways to adapt the Transformer architecture to keyword spotting and introduce the Keyword Transformer (KWT), a fully self-attentional architecture that exceeds state-of-the-art performance across multiple tasks without any pre-training or additional data. Surprisingly, this simple architecture outperforms more complex models that mix convolutional, recurrent and attentive layers. KWT can be used as a drop-in replacement for these models, setting two new benchmark records on the Google Speech Commands dataset with 98.6% and 97.7% accuracy on the 12 and 35-command tasks respectively.

View on arXivView PDF

Code Implementations(1)

ARM-software/keyword-transformerOfficial100%
Apache-2.0

Official implementation of the Keyword Transformer: https://arxiv.org/abs/2104.00769

14840Shell, PythonMay 11, 20214 years agoApache-2.0
keyword-spottingmachine-learningtransformer

Similar Papers

On-Device Constrained Self-Supervised Speech Representation Learning for Keyword Spotting via Knowledge Distillation

84% match2023

Contextual Transformer Networks for Visual Recognition

83% match2 repos5382021

TAB: Transformer Attention Bottlenecks enable User Intervention and Debugging in Vision-Language Models

83% match1 repo52024

When to Use Efficient Self Attention? Profiling Text, Speech and Image Transformer Variants

83% match2023

Honk: A PyTorch Reimplementation of Convolutional Neural Networks for Keyword Spotting

83% match1 repo5272017

Deep Residual Learning for Small-Footprint Keyword Spotting

82% match1 repo5272017

Cite this paper

@article{berg2021keyword,
  title  = {Keyword Transformer: A Self-Attention Model for Keyword Spotting},
  author = {Axel Berg and Mark O'Connor and Miguel Tairum Cruz},
  year   = {2021},
  eprint = {2104.00769},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/2104.00769}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2104.00769.svg)](https://codeofpaper.com/paper/2104.00769)

Discussion