Fluoroformer: Scaling multiple instance learning to multiplexed images via attention-based channel fusion

Marc Harary, Eliezer M. Van Allen, William Lotter
11/13/2024
eess.IVcs.AIcs.CV

No official code found. Best implementation: lotterlab/fluoroformer (4 stars, MIT, last commit 2024-11-19).

Abstract

Though multiple instance learning (MIL) has been a foundational strategy in computational pathology for processing whole slide images (WSIs), current approaches are designed for traditional hematoxylin and eosin (H&E) slides rather than emerging multiplexed technologies. Here, we present an MIL strategy, the Fluoroformer module, that is specifically tailored to multiplexed WSIs by leveraging scaled dot-product attention (SDPA) to interpretably fuse information across disparate channels. On a cohort of 434 non-small cell lung cancer (NSCLC) samples, we show that the Fluoroformer both obtains strong prognostic performance and recapitulates immuno-oncological hallmarks of NSCLC. Our technique thereby provides a path for adapting state-of-the-art AI techniques to emerging spatial biology assays.

View on arXivView PDF

Code Implementations(1)

lotterlab/fluoroformerCommunity100%
MIT
41PythonNov 8, 20241 years agoMIT

Similar Papers

Semantics-Aware Attention Guidance for Diagnosing Whole Slide Images

86% match2024

The Whole Pathological Slide Classification via Weakly Supervised Learning

84% match2023

Advances in Multiple Instance Learning for Whole Slide Image Analysis: Techniques, Challenges, and Future Directions

84% match2024

Towards Hierarchical Regional Transformer-based Multiple Instance Learning

83% match2023

WSD-MIL: Window Scale Decay Multiple Instance Learning for Whole Slide Image Classification

83% match2025

Dual Attention Model with Reinforcement Learning for Classification of Histology Whole-Slide Images

83% match2023

Cite this paper

@article{harary2024fluoroformer,
  title  = {Fluoroformer: Scaling multiple instance learning to multiplexed images via attention-based channel fusion},
  author = {Marc Harary and Eliezer M. Van Allen and William Lotter},
  year   = {2024},
  eprint = {2411.08975},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2411.08975v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2411.08975.svg)](https://codeofpaper.com/paper/2411.08975)

Discussion