Parameter-Efficient Transformer Embeddings

Henry Ndubuaku, Mouad Talhi
5/4/2025
cs.CLcs.AIcs.LG

No official code found. Best implementation: HenryNdubuaku/pete (10 stars, last commit 2026-02-06).

Abstract

Embedding layers in transformer-based NLP models typically account for the largest share of model parameters, scaling with vocabulary size but not yielding performance gains proportional to scale. We propose an alternative approach in which token embedding vectors are first generated deterministically, directly from the token IDs using a Fourier expansion of their normalized values, followed by a lightweight multilayer perceptron (MLP) that captures higher-order interactions. We train standard transformers and our architecture on natural language inference tasks (SNLI and MNLI), and evaluate zero-shot performance on sentence textual similarity (STS-B). Our results demonstrate that the proposed method achieves competitive performance using significantly fewer parameters, trains faster, and operates effectively without the need for dropout. This proof-of-concept study highlights the potential for scalable, memory-efficient language models and motivates further large-scale experimentation based on our findings.

View on arXivView PDF

Code Implementations(1)

HenryNdubuaku/peteCommunity100%

Parameter-efficient transformer embeddings replace learned embeddings with hardware-aware polynomial expansions of token IDs.

100PythonNov 2, 20248 months ago
arxivarxiv-papersembeddingsllmsnlp

Similar Papers

Efficient transformer with reinforced position embedding for language models

86% match2024

Transformer Layer Injection: A Novel Approach for Efficient Upscaling of Large Language Models

86% match2024

Scaling Transformer to 1M tokens and beyond with RMT

85% match2 repos7762023

Advancements in Natural Language Processing: Exploring Transformer-Based Architectures for Text Understanding

85% match2025

Over-Tokenized Transformer: Vocabulary is Generally Worth Scaling

85% match2025

Hierarchical Autoregressive Transformers: Combining Byte- and Word-Level Processing for Robust, Adaptable Language Models

85% match2025

Cite this paper

@article{ndubuaku2025parameterefficient,
  title  = {Parameter-Efficient Transformer Embeddings},
  author = {Henry Ndubuaku and Mouad Talhi},
  year   = {2025},
  eprint = {2505.02266},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2505.02266v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2505.02266.svg)](https://codeofpaper.com/paper/2505.02266)

Discussion