Mind's Eye: Image Recognition by EEG via Multimodal Similarity-Keeping Contrastive Learning

Chi-Sheng Chen, Chun-Shu Wei
6/5/2024
eess.SPcs.AIcs.HC

Official code: ChiShengChen/MUSE_EEG (44 stars, last commit 2025-04-13).

Abstract

Decoding images from non-invasive electroencephalographic (EEG) signals has been a grand challenge in understanding how the human brain process visual information in real-world scenarios. To cope with the issues of signal-to-noise ratio and nonstationarity, this paper introduces a MUltimodal Similarity-keeping contrastivE learning (MUSE) framework for zero-shot EEG-based image classification. We develop a series of multivariate time-series encoders tailored for EEG signals and assess the efficacy of regularized contrastive EEG-Image pretraining using an extensive visual EEG dataset. Our method achieves state-of-the-art performance, with a top-1 accuracy of 19.3% and a top-5 accuracy of 48.8% in 200-way zero-shot image classification. Furthermore, we visualize neural patterns via model interpretation, shedding light on the visual processing dynamics in the human brain. The code repository for this work is available at: https://github.com/ChiShengChen/MUSE_EEG.

View on arXivView PDF

Code Implementations(1)

ChiShengChen/MUSE_EEGOfficial100%

The official implement of Mind's eye: image recognition by EEG via multimodal similarity-keeping contrastive learning.

442PythonMay 22, 20241 years ago
contrastive-learningdeep-learningeeg-analysiseeg-signals-processingmultimodal-learning+1 more

Similar Papers

EEG-CLIP : Learning EEG representations from natural language descriptions

84% match1 repo292025

Visual Decoding and Reconstruction via EEG Embeddings with Guided Diffusion

84% match2 repos2202026

NECOMIMI: Neural-Cognitive Multimodal EEG-informed Image Generation with Diffusion Models

83% match2024

Decoding visual brain representations from electroencephalography through Knowledge Distillation and latent diffusion models

83% match2023

A Knowledge-Driven Cross-view Contrastive Learning for EEG Representation

83% match2023

CoSupFormer : A Contrastive Supervised learning approach for EEG signal Classification

83% match2025

Cite this paper

@article{chen2024minds,
  title  = {Mind's Eye: Image Recognition by EEG via Multimodal Similarity-Keeping Contrastive Learning},
  author = {Chi-Sheng Chen and Chun-Shu Wei},
  year   = {2024},
  eprint = {2406.16910},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2406.16910v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2406.16910.svg)](https://codeofpaper.com/paper/2406.16910)

Discussion