Masked-attention Mask Transformer for Universal Image Segmentation

Bowen Cheng, Ishan Misra, Alexander G. Schwing, Alexander Kirillov, Rohit Girdhar
12/2/2021
cs.CVcs.AIcs.LG

Official code: facebookresearch/Mask2Former (3,415 stars, MIT, last commit 2024-07-29).

Abstract

Image segmentation is about grouping pixels with different semantics, e.g., category or instance membership, where each choice of semantics defines a task. While only the semantics of each task differ, current research focuses on designing specialized architectures for each task. We present Masked-attention Mask Transformer (Mask2Former), a new architecture capable of addressing any image segmentation task (panoptic, instance or semantic). Its key components include masked attention, which extracts localized features by constraining cross-attention within predicted mask regions. In addition to reducing the research effort by at least three times, it outperforms the best specialized architectures by a significant margin on four popular datasets. Most notably, Mask2Former sets a new state-of-the-art for panoptic segmentation (57.8 PQ on COCO), instance segmentation (50.1 AP on COCO) and semantic segmentation (57.7 mIoU on ADE20K).

View on arXivView PDF

Code Implementations(1)

facebookresearch/Mask2FormerOfficial100%
MIT

Code release for "Masked-attention Mask Transformer for Universal Image Segmentation"

3,415528C++, CudaNov 24, 20212 years agoMIT

Similar Papers

Optimized Unet with Attention Mechanism for Multi-Scale Semantic Segmentation

86% match2025

From Drop-off to Recovery: A Mechanistic Analysis of Segmentation in MLLMs

86% match2026

Dual Attention U-Net with Feature Infusion: Pushing the Boundaries of Multiclass Defect Segmentation

86% match1 repo182023

EOV-Seg: Efficient Open-Vocabulary Panoptic Segmentation

85% match1 repo402024

Principal Component Clustering for Semantic Segmentation in Synthetic Data Generation

85% match2024

The revenge of BiSeNet: Efficient Multi-Task Image Segmentation

85% match2024

Cite this paper

@article{cheng2021maskedattention,
  title  = {Masked-attention Mask Transformer for Universal Image Segmentation},
  author = {Bowen Cheng and Ishan Misra and Alexander G. Schwing and Alexander Kirillov and Rohit Girdhar},
  year   = {2021},
  eprint = {2112.01527},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/2112.01527}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2112.01527.svg)](https://codeofpaper.com/paper/2112.01527)

Discussion