GazeSAM: What You See is What You Segment

Bin Wang, Armstrong Aboah, Zheyuan Zhang, Ulas Bagci
4/26/2023
cs.CV

Official code: ukaukaaaa/GazeSAM (69 stars, last commit 2023-05-01).

Abstract

This study investigates the potential of eye-tracking technology and the Segment Anything Model (SAM) to design a collaborative human-computer interaction system that automates medical image segmentation. We present the \textbf{GazeSAM} system to enable radiologists to collect segmentation masks by simply looking at the region of interest during image diagnosis. The proposed system tracks radiologists' eye movement and utilizes the eye-gaze data as the input prompt for SAM, which automatically generates the segmentation mask in real time. This study is the first work to leverage the power of eye-tracking technology and SAM to enhance the efficiency of daily clinical practice. Moreover, eye-gaze data coupled with image and corresponding segmentation labels can be easily recorded for further advanced eye-tracking research. The code is available in \url{https://github.com/ukaukaaaa/GazeSAM}.

View on arXivView PDF

Code Implementations(1)

ukaukaaaa/GazeSAMOfficial100%

A human-computer interaction system that combines eye tracking with Segment Anything Model (SAM), and it enables users to segment object they are looking at in real-time. [NeurIPSW 2023]

698PythonApr 26, 20233 years ago

Similar Papers

When Eye-Tracking Meets Machine Learning: A Systematic Review on Applications in Medical Image Analysis

87% match2024

GazeGNN: A Gaze-Guided Graph Neural Network for Chest X-ray Classification

87% match1 repo222023

Multimodal Learning and Cognitive Processes in Radiology: MedGaze for Chest X-ray Scanpath Prediction

87% match2024

Integrating Eye-Gaze Data into CXR DL Approaches: A Preliminary study

84% match2023

I-AI: A Controllable & Interpretable AI System for Decoding Radiologists' Intense Focus for Accurate CXR Diagnoses

83% match1 repo72023

Gaze-Assisted Medical Image Segmentation

83% match1 repo142024

Cite this paper

@article{wang2023gazesam,
  title  = {GazeSAM: What You See is What You Segment},
  author = {Bin Wang and Armstrong Aboah and Zheyuan Zhang and Ulas Bagci},
  year   = {2023},
  eprint = {2304.13844},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2304.13844v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2304.13844.svg)](https://codeofpaper.com/paper/2304.13844)

Discussion