Cleaner Pretraining Corpus Curation with Neural Web Scraping

Zhipeng Xu, Zhenghao Liu, Yukun Yan, Zhiyuan Liu, Ge Yu +1 more
2/22/2024
cs.CL

Official code: OpenMatch/NeuScraper (228 stars, MIT, last commit 2024-08-28).

Abstract

The web contains large-scale, diverse, and abundant information to satisfy the information-seeking needs of humans. Through meticulous data collection, preprocessing, and curation, webpages can be used as a fundamental data resource for language model pretraining. However, when confronted with the progressively revolutionized and intricate nature of webpages, rule-based/feature-based web scrapers are becoming increasingly inadequate. This paper presents a simple, fast, and effective Neural web Scraper (NeuScraper) to help extract primary and clean text contents from webpages. Experimental results show that NeuScraper surpasses the baseline scrapers by achieving more than a 20% improvement, demonstrating its potential in extracting higher-quality data to facilitate the language model pretraining. All of the code is available at https://github.com/OpenMatch/NeuScraper.

View on arXivView PDF

Code Implementations(1)

OpenMatch/NeuScraperOfficial100%
MIT

[ACL 2024] This is the code repo for our ACL’24 paper "Cleaner Pretraining Corpus Curation with Neural Web Scraping".

22818Shell, PythonFeb 19, 20242 years agoMIT

Similar Papers

Leveraging Large Language Models for Web Scraping

80% match2024

AICC: Parse HTML Finer, Make Models Better -- A 7.3T AI-Ready Corpus Built by a Model-Based HTML Parser

80% match2025

Beyond a Single Extractor: Re-thinking HTML-to-Text Extraction for LLM Pretraining

79% match2026

Blu-WERP (Web Extraction and Refinement Pipeline): A Scalable Pipeline for Preprocessing Large Language Model Datasets

78% match2025

ChineseWebText: Large-scale High-quality Chinese Web Text Extracted with Effective Evaluation Model

78% match1 repo1862023

PLM-GNN: A Webpage Classification Method based on Joint Pre-trained Language Model and Graph Neural Network

77% match2023

Cite this paper

@article{xu2024cleaner,
  title  = {Cleaner Pretraining Corpus Curation with Neural Web Scraping},
  author = {Zhipeng Xu and Zhenghao Liu and Yukun Yan and Zhiyuan Liu and Ge Yu and Chenyan Xiong},
  year   = {2024},
  eprint = {2402.14652},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2402.14652v3}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2402.14652.svg)](https://codeofpaper.com/paper/2402.14652)

Discussion