Neural Architecture Search for Sentence Classification with BERT

Philip Kenneweg, Sarah Schröder, Barbara Hammer
3/27/2024
cs.AI

No confident code implementation found for this paper.

Abstract

Pre training of language models on large text corpora is common practice in Natural Language Processing. Following, fine tuning of these models is performed to achieve the best results on a variety of tasks. In this paper we question the common practice of only adding a single output layer as a classification head on top of the network. We perform an AutoML search to find architectures that outperform the current single layer at only a small compute cost. We validate our classification architecture on a variety of NLP benchmarks from the GLUE dataset.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

On Importance of Layer Pruning for Smaller BERT Models and Low Resource Languages

84% match2025

It's All in The [MASK]: Simple Instruction-Tuning Enables BERT-like Masked Language Models As Generative Classifiers

83% match2025

Attention is Not Always What You Need: Towards Efficient Classification of Domain-Specific Text

83% match2023

Tiny language models

83% match1 repo32025

Cost-Aware Model Selection for Text Classification: Multi-Objective Trade-offs Between Fine-Tuned Encoders and LLM Prompting in Production

83% match2026

Scaling Performance of Large Language Model Pretraining

83% match2025

Cite this paper

@article{kenneweg2024neural,
  title  = {Neural Architecture Search for Sentence Classification with BERT},
  author = {Philip Kenneweg and Sarah Schröder and Barbara Hammer},
  year   = {2024},
  eprint = {2403.18547},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2403.18547v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2403.18547.svg)](https://codeofpaper.com/paper/2403.18547)

Discussion