Not all layers are equally as important: Every Layer Counts BERT

Lucas Georges Gabriel Charpentier, David Samuel
11/3/2023
cs.CL

No official code found. Best implementation: SunnyWan59/qdelcbert (2 stars, last commit 2025-02-11).

Abstract

This paper introduces a novel modification of the transformer architecture, tailored for the data-efficient pretraining of language models. This aspect is evaluated by participating in the BabyLM challenge, where our solution won both the strict and strict-small tracks. Our approach allows each transformer layer to select which outputs of previous layers to process. The empirical results verify the potential of this simple modification and show that not all layers are equally as important.

View on arXivView PDF

Code Implementations(1)

SunnyWan59/qdelcbertCommunity62%

My own implementation of the babyLM ELC BERT, given by the 2023 paper "Not all layers are equally as important: Every Layer Counts BERT" by Charpentier and Samuel

20PythonJan 9, 20251 years ago

Similar Papers

Increasing The Performance of Cognitively Inspired Data-Efficient Language Models via Implicit Structure Building

87% match2023

Transformer Layers as Painters

86% match1 repo32024

Layerwise Importance Analysis of Feed-Forward Networks in Transformer-based Language Models

86% match2025

No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Models

85% match2026

No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Models

85% match1 repo812023

Efficient GPT Model Pre-training using Tensor Train Matrix Representation

85% match2023

Cite this paper

@article{charpentier2023layers,
  title  = {Not all layers are equally as important: Every Layer Counts BERT},
  author = {Lucas Georges Gabriel Charpentier and David Samuel},
  year   = {2023},
  eprint = {2311.02265},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2311.02265v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2311.02265.svg)](https://codeofpaper.com/paper/2311.02265)

Discussion