I've Got 99 Problems But FLOPS Ain't One

Alexandru M. Gherghescu, Vlad-Andrei Bădoiu, Alexandru Agache, Mihai-Valentin Dumitru, Iuliu Vasilescu +2 more
7/1/2024
cs.DCcs.CLcs.LG

No confident code implementation found for this paper. 3 possible matches can be shown below.

Abstract

Hyperscalers dominate the landscape of large network deployments, yet they rarely share data or insights about the challenges they face. In light of this supremacy, what problems can we find to solve in this space? We take an unconventional approach to find relevant research directions, starting from public plans to build a $100 billion datacenter for machine learning applications. Leveraging the language models scaling laws, we discover what workloads such a datacenter might carry and explore the challenges one may encounter in doing so, with a focus on networking research. We conclude that building the datacenter and training such models is technically possible, but this requires novel wide-area transports for inter-DC communication, a multipath transport and novel datacenter topologies for intra-datacenter communication, high speed scale-up networks and transports, outlining a rich research agenda for the networking community.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. 3 weaker matches are hidden by default — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

FoMoE: Breaking the Full-Replica Barrier with a Federation of MoEs

81% match2026

Large Language Models for Networking: Workflow, Advances and Challenges

80% match2024

Communication-Efficient Large-Scale Distributed Deep Learning: A Comprehensive Survey

79% match2024

MAD Max Beyond Single-Node: Enabling Large Machine Learning Model Acceleration on Distributed Systems

79% match2023

Hardware Scaling Trends and Diminishing Returns in Large-Scale Distributed Training

79% match2024

Rail-only: A Low-Cost High-Performance Network for Training LLMs with Trillion Parameters

79% match2023

Cite this paper

@article{gherghescu2024ive,
  title  = {I've Got 99 Problems But FLOPS Ain't One},
  author = {Alexandru M. Gherghescu and Vlad-Andrei Bădoiu and Alexandru Agache and Mihai-Valentin Dumitru and Iuliu Vasilescu and Radu Mantu and Costin Raiciu},
  year   = {2024},
  eprint = {2407.12819},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2407.12819v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2407.12819.svg)](https://codeofpaper.com/paper/2407.12819)

Discussion