SWEb: A Large Web Dataset for the Scandinavian Languages

Tobias Norlund, Tim Isbister, Amaru Cuba Gyllensten, Paul Dos Santos, Danila Petrelli +2 more
10/6/2024
cs.CL

No official code found. Best implementation: TobiasNorlund/SWEb (3 stars, MIT, last commit 2025-03-01).

Abstract

This paper presents the hitherto largest pretraining dataset for the Scandinavian languages: the Scandinavian WEb (SWEb), comprising over one trillion tokens. The paper details the collection and processing pipeline, and introduces a novel model-based text extractor that significantly reduces complexity in comparison with rule-based approaches. We also introduce a new cloze-style benchmark for evaluating language models in Swedish, and use this test to compare models trained on the SWEb data to models trained on FineWeb, with competitive results. All data, models and code are shared openly.

View on arXivView PDF

Code Implementations(1)

TobiasNorlund/SWEbCommunity65%
MIT

SWEb: A Large Web Dataset for the Scandinavian Languages

30CSS, LuaMar 1, 20251 years agoMIT

Similar Papers

ScandEval: A Benchmark for Scandinavian Natural Language Processing

85% match1 repo2002023

SnakModel: Lessons Learned from Training an Open Danish Large Language Model

84% match1 repo102024

Small Languages, Big Models: A Study of Continual Training on Languages of Norway

82% match2024

Multilingual Language Model Pretraining using Machine-translated Data

81% match2025

FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Language

81% match1 repo2472025

Building High-Quality Datasets for Portuguese LLMs: From Common Crawl Snapshots to Industrial-Grade Corpora

81% match2025

Cite this paper

@article{norlund2024sweb,
  title  = {SWEb: A Large Web Dataset for the Scandinavian Languages},
  author = {Tobias Norlund and Tim Isbister and Amaru Cuba Gyllensten and Paul Dos Santos and Danila Petrelli and Ariel Ekgren and Magnus Sahlgren},
  year   = {2024},
  eprint = {2410.04456},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2410.04456v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2410.04456.svg)](https://codeofpaper.com/paper/2410.04456)

Discussion