Synthetic data shuffling accelerates the convergence of federated learning under data heterogeneity

Bo Li, Yasin Esfandiari, Mikkel N. Schmidt, Tommy S. Alstrøm, Sebastian U. Stich
6/23/2023
cs.LGcs.CVcs.DC

Official code: lyn1874/fedssyn (6 stars, MIT, last commit 2023-07-17).

Abstract

In federated learning, data heterogeneity is a critical challenge. A straightforward solution is to shuffle the clients' data to homogenize the distribution. However, this may violate data access rights, and how and when shuffling can accelerate the convergence of a federated optimization algorithm is not theoretically well understood. In this paper, we establish a precise and quantifiable correspondence between data heterogeneity and parameters in the convergence rate when a fraction of data is shuffled across clients. We prove that shuffling can quadratically reduce the gradient dissimilarity with respect to the shuffling percentage, accelerating convergence. Inspired by the theory, we propose a practical approach that addresses the data access rights issue by shuffling locally generated synthetic data. The experimental results show that shuffling synthetic data improves the performance of multiple existing federated learning algorithms by a large margin.

View on arXivView PDF

Code Implementations(1)

lyn1874/fedssynOfficial100%
MIT

Official implementation of the paper "Synthetic data shuffling accelerates the convergence of federated learning under data heterogeneity"

62Shell, PythonJul 17, 20233 years agoMIT
federatedlearningmachinelearningoptimization

Similar Papers

A New Theoretical Perspective on Data Heterogeneity in Federated Optimization

87% match2024

An Aggregation-Free Federated Learning for Tackling Data Heterogeneity

84% match2024

Tackling Computational Heterogeneity in FL: A Few Theoretical Insights

84% match2023

FedStale: leveraging stale client updates in federated learning

84% match2024

Exact and Linear Convergence for Federated Learning under Arbitrary Client Participation is Attainable

84% match1 repo92025

Momentum Benefits Non-IID Federated Learning Simply and Provably

83% match2023

Cite this paper

@article{li2023synthetic,
  title  = {Synthetic data shuffling accelerates the convergence of federated learning under data heterogeneity},
  author = {Bo Li and Yasin Esfandiari and Mikkel N. Schmidt and Tommy S. Alstrøm and Sebastian U. Stich},
  year   = {2023},
  eprint = {2306.13263},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2306.13263v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2306.13263.svg)](https://codeofpaper.com/paper/2306.13263)

Discussion