LLM Vocabulary Compression for Low-Compute Environments

Sreeram Vennam, Anish Joishy, Ponnurangam Kumaraguru
11/10/2024
cs.CLcs.LG

No confident code implementation found for this paper.

Abstract

We present a method to compress the final linear layer of language models, reducing memory usage by up to 3.4x without significant performance loss. By grouping tokens based on Byte Pair Encoding (BPE) merges, we prevent materialization of the memory-intensive logits tensor. Evaluations on the TinyStories dataset show that our method performs on par with GPT-Neo and GPT2 while significantly improving throughput by up to 3x, making it suitable for low-compute environments.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

TensorSLM: Energy-efficient Embedding Compression of Sub-billion Parameter Language Models on Low-end Devices

84% match2025

When Compression Meets Model Compression: Memory-Efficient Double Compression for Large Language Models

84% match2025

Sentence-Anchored Gist Compression for Long-Context LLMs

84% match2025

Basis Selection: Low-Rank Decomposition of Pretrained Large Language Models for Target Applications

83% match2024

Lillama: Large Language Models Compression via Low-Rank Feature Distillation

83% match1 repo112024

LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression

83% match2 repos6.5k2023

Cite this paper

@article{vennam2024vocabulary,
  title  = {LLM Vocabulary Compression for Low-Compute Environments},
  author = {Sreeram Vennam and Anish Joishy and Ponnurangam Kumaraguru},
  year   = {2024},
  eprint = {2411.06371},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2411.06371v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2411.06371.svg)](https://codeofpaper.com/paper/2411.06371)

Discussion