AST: Audio Spectrogram Transformer

Yuan Gong, Yu-An Chung, James Glass
4/5/2021
cs.SDcs.AI

Official code: YuanGongND/ast (1,483 stars, BSD-3-Clause, PyTorch, last commit 2023-05-21).

Abstract

In the past decade, convolutional neural networks (CNNs) have been widely adopted as the main building block for end-to-end audio classification models, which aim to learn a direct mapping from audio spectrograms to corresponding labels. To better capture long-range global context, a recent trend is to add a self-attention mechanism on top of the CNN, forming a CNN-attention hybrid model. However, it is unclear whether the reliance on a CNN is necessary, and if neural networks purely based on attention are sufficient to obtain good performance in audio classification. In this paper, we answer the question by introducing the Audio Spectrogram Transformer (AST), the first convolution-free, purely attention-based model for audio classification. We evaluate AST on various audio classification benchmarks, where it achieves new state-of-the-art results of 0.485 mAP on AudioSet, 95.6% accuracy on ESC-50, and 98.1% accuracy on Speech Commands V2.

View on arXivView PDF

Code Implementations(1)

YuanGongND/astOfficial100%
PyTorchBSD-3-Clause

Code for the Interspeech 2021 paper "AST: Audio Spectrogram Transformer".

1,483246Shell, PythonJun 2, 20213 years agoBSD-3-Clause
audioaudio-classificationdeep-learningkeyword-spottingpytorch+3 more

Similar Papers

Audio Mamba: Bidirectional State Space Model for Audio Representation Learning

90% match1 repo1722024

SSAST: Self-Supervised Audio Spectrogram Transformer

88% match1 repo4302021

GAFX: A General Audio Feature eXtractor

86% match2022

LHGNN: Local-Higher Order Graph Neural Networks For Audio Classification and Tagging

85% match2025

A neural attention model for speech command recognition

83% match1 repo1872018

LCANets++: Robust Audio Classification using Multi-layer Neural Networks with Lateral Competition

83% match2023

Cite this paper

@article{gong2021ast,
  title  = {AST: Audio Spectrogram Transformer},
  author = {Yuan Gong and Yu-An Chung and James Glass},
  year   = {2021},
  eprint = {2104.01778},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/2104.01778}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2104.01778.svg)](https://codeofpaper.com/paper/2104.01778)

Discussion