Learned Queries and Keys Are All You Need: Replacing the Value Projection with Structured Transforms

Ene Meco, Emadeldeen Hamdan, A. Enis Cetin
9/29/2026
cs.LG

No confident code implementation found for this paper. 3 possible matches can be shown below.

Abstract

To reduce the number of parameters and cache memory requirements of transformers we introduce dual-headed transformers instead of three heads. We studied Walsh-Hadamard Transform (WHT), Discrete Cosine Transform (DCT), Discrete Fourier Transform, filterbank based Shearlet Transform, and Multiplication-Avoiding (MA) operators to construct dual heads. We combine spatial patches and their orthogonal transforms (or Shearlet and MA operators) in a structure similar to the attention block. We obtained better results than triple headed transformers in ImageNet. Extensive simulation examples are presented.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. 3 weaker matches are hidden by default — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

Pit One Against Many: Leveraging Attention-head Embeddings for Parameter-efficient Multi-head Attention

80% match2023

MABViT -- Modified Attention Block Enhances Vision Transformers

79% match2023

Fully Interpretable Minimal Transformers: From Geometry to Algorithm

79% match2026

EfficientViT: Memory Efficient Vision Transformer with Cascaded Group Attention

78% match2 repos1.8k2026

Optimizing Knowledge Distillation in Transformers: Enabling Multi-Head Attention without Alignment Barriers

78% match2025

Attention-Only Transformers via Unrolled Subspace Denoising

78% match2026

Cite this paper

@article{meco2026learned,
  title  = {Learned Queries and Keys Are All You Need: Replacing the Value Projection with Structured Transforms},
  author = {Ene Meco and Emadeldeen Hamdan and A. Enis Cetin},
  year   = {2026},
  eprint = {2609.36698},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2609.36698v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2609.36698.svg)](https://codeofpaper.com/paper/2609.36698)

Discussion