Too Much Information: Keeping Training Simple for BabyLMs

Lukas Edman, Lisa Bylinina
11/3/2023
cs.CL

No confident code implementation found for this paper.

Abstract

This paper details the work of the University of Groningen for the BabyLM Challenge. We follow the idea that, like babies, language models should be introduced to simpler concepts first and build off of that knowledge to understand more complex concepts. We examine this strategy of simple-then-complex through a variety of lenses, namely context size, vocabulary, and overall linguistic complexity of the data. We find that only one, context size, is truly beneficial to training a language model. However this simple change to context size gives us improvements of 2 points on average on (Super)GLUE tasks, 1 point on MSGS tasks, and 12\% on average on BLiMP tasks. Our context-limited model outperforms the baseline that was trained on 10$\times$ the amount of data.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

Understanding In-Context Learning via Supportive Pretraining Data

87% match2023

Beyond Output Matching: Bidirectional Alignment for Enhanced In-Context Learning

87% match2023

Baby's CoThought: Leveraging Large Language Models for Enhanced Reasoning in Compact Models

87% match1 repo172023

LongQLoRA: Efficient and Effective Method to Extend Context Length of Large Language Models

87% match1 repo1682023

In-context Learning Generalizes, But Not Always Robustly: The Case of Syntax

86% match1 repo62023

Evaluating Named Entity Recognition Using Few-Shot Prompting with Large Language Models

86% match1 repo102024

Cite this paper

@article{edman2023much,
  title  = {Too Much Information: Keeping Training Simple for BabyLMs},
  author = {Lukas Edman and Lisa Bylinina},
  year   = {2023},
  eprint = {2311.01955},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2311.01955v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2311.01955.svg)](https://codeofpaper.com/paper/2311.01955)

Discussion