Avey-B

Devang Acharya, Mohammad Hammoud
2/17/2026
cs.CLcs.AI

No official code found. Best implementation: rimads/avey-b (32 stars, Apache-2.0, last commit 2026-02-21).

Abstract

Compact pretrained bidirectional encoders remain the backbone of industrial NLP under tight compute and memory budgets. Their effectiveness stems from self-attention's ability to deliver high-quality bidirectional contextualization with sequence-level parallelism, as popularized by BERT-style architectures. Recently, Avey was introduced as an autoregressive, attention-free alternative that naturally admits an encoder-only adaptation. In this paper, we reformulate Avey for the encoder-only paradigm and propose several innovations to its architecture, including decoupled static and dynamic parameterizations, stability-oriented normalization, and neural compression. Results show that this reformulated architecture compares favorably to four widely used Transformer-based encoders, consistently outperforming them on standard token-classification and information-retrieval benchmarks while scaling more efficiently to long contexts.

View on arXivView PDF

Code Implementations(1)

rimads/avey-bCommunity60%
Apache-2.0

Code for the Avey-B paper (https://arxiv.org/abs/2602.15814)

321Shell, PythonNov 2, 20258 months agoApache-2.0

Similar Papers

Don't Pay Attention

88% match1 repo1632025

Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference

82% match2 repos1.8k2024

It's All in The [MASK]: Simple Instruction-Tuning Enables BERT-like Masked Language Models As Generative Classifiers

80% match2025

Multi-Task Pre-Finetuning of Lightweight Transformer Encoders for Text Classification and NER

80% match2025

NarrowBERT: Accelerating Masked Language Model Pretraining and Inference

79% match1 repo82023

Focus on the Core: Efficient Attention via Pruned Token Compression for Document Classification

79% match2024

Cite this paper

@article{acharya2026aveyb,
  title  = {Avey-B},
  author = {Devang Acharya and Mohammad Hammoud},
  year   = {2026},
  eprint = {2602.15814},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2602.15814v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2602.15814.svg)](https://codeofpaper.com/paper/2602.15814)

Discussion