A Human-Inspired Decoupled Architecture for Efficient Audio Representation Learning

Harunori Kawano, Takeshi Sasaki
3/27/2026
cs.SDcs.AIcs.LG

Official code: HarunoriKawano/HEAR (2 stars, MIT, last commit 2026-03-30).

Abstract

While self-supervised learning (SSL) has revolutionized audio representation, the excessive parameterization and quadratic computational cost of standard Transformers limit their deployment on resource-constrained devices. To address this bottleneck, we propose HEAR (Human-inspired Efficient Audio Representation), a novel decoupled architecture. Inspired by the human cognitive ability to isolate local acoustic features from global context, HEAR splits the processing pipeline into two dedicated modules: an Acoustic Model for local feature extraction and a Task Model for global semantic integration. Coupled with an Acoustic Tokenizer trained via knowledge distillation, our approach enables robust Masked Audio Modeling (MAM). Extensive experiments demonstrate that HEAR requires only 15M parameters and 9.47 GFLOPs for inference, operating at a fraction of the computational cost of conventional foundation models (which typically require 85M-94M parameters). Despite this high efficiency, HEAR achieves highly competitive performance across diverse audio classification benchmarks. The code and pre-trained models are available at https://github.com/HarunoriKawano/HEAR

View on arXivView PDF

Code Implementations(1)

HarunoriKawano/HEAROfficial100%
MIT

Official implementation of "A Human-Inspired Decoupled Architecture for Efficient Audio Representation Learning" (HEAR).

20PythonMar 26, 20266 months agoMIT
artificial-intelligenceaudio-recognitiondeep-learningmachine-learningspeech-recognition

Similar Papers

DinoSR: Self-Distillation and Online Clustering for Self-supervised Speech Representation Learning

84% match1 repo532026

SSAMBA: Self-Supervised Audio Representation Learning with Mamba State Space Model

84% match1 repo1432024

Prototype based Masked Audio Model for Self-Supervised Learning of Sound Event Detection

84% match1 repo1062024

Unified Speech Recognition: A Single Model for Auditory, Visual, and Audiovisual Inputs

83% match1 repo402026

HiCMAE: Hierarchical Contrastive Masked Autoencoder for Self-Supervised Audio-Visual Emotion Recognition

83% match1 repo1212024

EAT: Self-Supervised Pre-Training with Efficient Audio Transformer

83% match1 repo2462024

Cite this paper

@article{kawano2026humaninspired,
  title  = {A Human-Inspired Decoupled Architecture for Efficient Audio Representation Learning},
  author = {Harunori Kawano and Takeshi Sasaki},
  year   = {2026},
  eprint = {2603.26098},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2603.26098v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2603.26098.svg)](https://codeofpaper.com/paper/2603.26098)

Discussion