3FM: Multi-modal Meta-learning for Federated Tasks

Minh Tran, Roochi Shah, Zejun Gong
12/15/2023
cs.LG

No official code found. Best implementation: minhtcai/MLMF (6 stars, last commit 2025-06-25).

Abstract

We present a novel approach in the domain of federated learning (FL), particularly focusing on addressing the challenges posed by modality heterogeneity, variability in modality availability across clients, and the prevalent issue of missing data. We introduce a meta-learning framework specifically designed for multimodal federated tasks. Our approach is motivated by the need to enable federated models to robustly adapt when exposed to new modalities, a common scenario in FL where clients often differ in the number of available modalities. The effectiveness of our proposed framework is demonstrated through extensive experimentation on an augmented MNIST dataset, enriched with audio and sign language data. We demonstrate that the proposed algorithm achieves better performance than the baseline on a subset of missing modality scenarios with careful tuning of the meta-learning rates. This is a shortened report, and our work will be extended and updated soon.

View on arXivView PDF

Code Implementations(1)

minhtcai/MLMFCommunity100%

Meta-Learning Initialization for Multimodal Federated Tasks

60TeX, BibTeX StyleNov 9, 20231 years ago

Similar Papers

Overcome Modal Bias in Multi-modal Federated Learning via Balanced Modality Selection

90% match1 repo82023

Federated Pseudo Modality Generation for Incomplete Multi-Modal MRI Reconstruction

88% match2023

Multimodal Federated Learning with Missing Modality via Prototype Mask and Contrast

88% match2 repos272023

ProMoE-FL: Prototype-conditioned Mixture of Experts for Multimodal Federated Learning with Missing Modalities

87% match1 repo12026

FedMFS: Federated Multimodal Fusion Learning with Selective Modality Communication

87% match2023

Examining Modality Incongruity in Multimodal Federated Learning for Medical Vision and Language-based Disease Detection

86% match2024

Cite this paper

@article{tran2023fm,
  title  = {3FM: Multi-modal Meta-learning for Federated Tasks},
  author = {Minh Tran and Roochi Shah and Zejun Gong},
  year   = {2023},
  eprint = {2312.10179},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2312.10179v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2312.10179.svg)](https://codeofpaper.com/paper/2312.10179)

Discussion