Missing Modality Robustness in Semi-Supervised Multi-Modal Semantic Segmentation

Harsh Maheshwari, Yen-Cheng Liu, Zsolt Kira
4/21/2023
cs.CVcs.LG

Official code: harshm121/M3L (44 stars, last commit 2023-11-14).

Abstract

Using multiple spatial modalities has been proven helpful in improving semantic segmentation performance. However, there are several real-world challenges that have yet to be addressed: (a) improving label efficiency and (b) enhancing robustness in realistic scenarios where modalities are missing at the test time. To address these challenges, we first propose a simple yet efficient multi-modal fusion mechanism Linear Fusion, that performs better than the state-of-the-art multi-modal models even with limited supervision. Second, we propose M3L: Multi-modal Teacher for Masked Modality Learning, a semi-supervised framework that not only improves the multi-modal performance but also makes the model robust to the realistic missing modality scenario using unlabeled data. We create the first benchmark for semi-supervised multi-modal semantic segmentation and also report the robustness to missing modalities. Our proposal shows an absolute improvement of up to 10% on robust mIoU above the most competitive baselines. Our code is available at https://github.com/harshm121/M3L

View on arXivView PDF

Code Implementations(1)

harshm121/M3LOfficial100%

Official repository for "Missing Modality Robustness in Semi-Supervised Multi-Modal Semantic Segmentation"

444Shell, PythonApr 3, 20232 years ago

Similar Papers

MAGIC++: Efficient and Resilient Modality-Agnostic Semantic Segmentation via Hierarchical Modality Selection

88% match2024

Centering the Value of Every Modality: Towards Efficient and Resilient Modality-agnostic Semantic Segmentation

87% match2026

Keep the Balance: A Parameter-Efficient Symmetrical Framework for RGB+X Semantic Segmentation

86% match1 repo212026

Learning Modality-agnostic Representation for Semantic Segmentation from Any Modalities

86% match2026

OmniSegmentor: A Flexible Multi-Modal Learning Framework for Semantic Segmentation

85% match2025

Fourier Prompt Tuning for Modality-Incomplete Scene Segmentation

85% match1 repo122024

Cite this paper

@article{maheshwari2023missing,
  title  = {Missing Modality Robustness in Semi-Supervised Multi-Modal Semantic Segmentation},
  author = {Harsh Maheshwari and Yen-Cheng Liu and Zsolt Kira},
  year   = {2023},
  eprint = {2304.10756},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2304.10756v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2304.10756.svg)](https://codeofpaper.com/paper/2304.10756)

Discussion