LLMs on a Budget? Say HOLA

Zohaib Hasan Siddiqui, Jiechao Gao, Ebad Shabbir, Mohammad Anas Azeez, Rafiq Ali +2 more
6/23/2025
cs.LGcs.AIcs.CL

No official code found. Best implementation: zohaibhasan066/HOLA_Codebase (0 stars, last commit 2025-07-04).

Abstract

Running Large Language Models (LLMs) on edge devices is constrained by high compute and memory demands posing a barrier for real-time applications in sectors like healthcare, education, and embedded systems. Current solutions such as quantization, pruning, and retrieval-augmented generation (RAG) offer only partial optimizations and often compromise on speed or accuracy. We introduce HOLA, an end-to-end optimization framework for efficient LLM deployment. Internally, it leverages Hierarchical Speculative Decoding (HSD) for faster inference without quality loss. Externally, AdaComp-RAG adjusts retrieval complexity based on context needs. Together with LoBi, which blends structured pruning (LoRA) and quantization, HOLA delivers significant gains: 17.6% EMA on GSM8K, 10.5% MCA on ARC, and reduced latency and memory on edge devices like Jetson Nano--proving both scalable and production-ready.

View on arXivView PDF

Code Implementations(1)

zohaibhasan066/HOLA_CodebaseCommunity59%

This repository contains code for "LLMs on a Budget: Say HOLA", featuring HOLA—a lightweight regularization method that improves cross-domain robustness of small language models. Includes training, evaluation, hardware-aware analysis, and visualizations on GSM8K and ARC datasets for edge deployment.

02PythonJul 3, 20251 years ago

Similar Papers

Democratizing LLM Efficiency: From Hyperscale Optimizations to Universal Deployability

88% match2025

New Solutions on LLM Acceleration, Optimization, and Application

86% match2024

ScaleLLM: A Resource-Frugal LLM Serving Framework by Optimizing End-to-End Efficiency

86% match2024

Puzzle: Distillation-Based NAS for Inference-Optimized LLMs

86% match2024

Slimming Down LLMs Without Losing Their Minds

85% match2025

From Large to Super-Tiny: End-to-End Optimization for Cost-Efficient LLMs

85% match2025

Cite this paper

@article{siddiqui2025llms,
  title  = {LLMs on a Budget? Say HOLA},
  author = {Zohaib Hasan Siddiqui and Jiechao Gao and Ebad Shabbir and Mohammad Anas Azeez and Rafiq Ali and Gautam Siddharth Kashyap and Usman Naseem},
  year   = {2025},
  eprint = {2506.18952},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2506.18952v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2506.18952.svg)](https://codeofpaper.com/paper/2506.18952)

Discussion