An Attentive Dual-Encoder Framework Leveraging Multimodal Visual and Semantic Information for Automatic OSAHS Diagnosis

Yingchen Wei, Xihe Qiu, Xiaoyu Tan, Jingjing Huang, Wei Chu +2 more
12/25/2024
cs.CVcs.LG

No official code found. Best implementation: luboyan6/VTA-OSAHS (1 star, MIT, last commit 2024-08-11).

Abstract

Obstructive sleep apnea-hypopnea syndrome (OSAHS) is a common sleep disorder caused by upper airway blockage, leading to oxygen deprivation and disrupted sleep. Traditional diagnosis using polysomnography (PSG) is expensive, time-consuming, and uncomfortable. Existing deep learning methods using facial image analysis lack accuracy due to poor facial feature capture and limited sample sizes. To address this, we propose a multimodal dual encoder model that integrates visual and language inputs for automated OSAHS diagnosis. The model balances data using randomOverSampler, extracts key facial features with attention grids, and converts physiological data into meaningful text. Cross-attention combines image and text data for better feature extraction, and ordered regression loss ensures stable learning. Our approach improves diagnostic efficiency and accuracy, achieving 91.3% top-1 accuracy in a four-class severity classification task, demonstrating state-of-the-art performance. Code will be released upon acceptance.

View on arXivView PDF

Code Implementations(1)

luboyan6/VTA-OSAHSCommunity100%
MIT
10PythonJul 17, 20242 years agoMIT

Similar Papers

Multimodal Sleep Stage and Sleep Apnea Classification Using Vision Transformer: A Multitask Explainable Learning Approach

81% match2025

NAP: Attention-Based Late Fusion for Automatic Sleep Staging

79% match2025

Joint enhancement of automatic chest X-ray diagnosis and radiological gaze prediction with multi-stage cooperative learning

78% match2024

Robust Multimodal Learning for Ophthalmic Disease Grading via Disentangled Representation

78% match1 repo52025

Deep Learning with HM-VGG: AI Strategies for Multi-modal Image Analysis

78% match2024

STREAMLINE: An Automated Machine Learning Pipeline for Biomedicine Applied to Examine the Utility of Photography-Based Phenotypes for OSA Prediction Across International Sleep Centers

78% match1 repo822023

Cite this paper

@article{wei2024attentive,
  title  = {An Attentive Dual-Encoder Framework Leveraging Multimodal Visual and Semantic Information for Automatic OSAHS Diagnosis},
  author = {Yingchen Wei and Xihe Qiu and Xiaoyu Tan and Jingjing Huang and Wei Chu and Yinghui Xu and Yuan Qi},
  year   = {2024},
  eprint = {2412.18919},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2412.18919v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2412.18919.svg)](https://codeofpaper.com/paper/2412.18919)

Discussion