Leveraging Speculative Sampling and KV-Cache Optimizations Together for Generative AI using OpenVINO

Haim Barad, Ekaterina Aidova, Yury Gorbachev
11/8/2023
cs.LGcs.AIcs.PF

Official code: openvinotoolkit/openvino_notebooks (3,190 stars, Apache-2.0, last commit 2026-04-30).

Abstract

Inference optimizations are critical for improving user experience and reducing infrastructure costs and power consumption. In this article, we illustrate a form of dynamic execution known as speculative sampling to reduce the overall latency of text generation and compare it with standard autoregressive sampling. This can be used together with model-based optimizations (e.g. quantization) to provide an optimized solution. Both sampling methods make use of KV caching. A Jupyter notebook and some sample executions are provided.

View on arXivView PDF

Code Implementations(1)

openvinotoolkit/openvino_notebooksOfficial100%
Apache-2.0

📚 Jupyter notebook tutorials for OpenVINO™

3,1901,026CSS, HTMLMar 11, 20215 months agoApache-2.0
computer-visiondeep-learninginferencemachine-learningopenvino

Similar Papers

Accelerated AI Inference via Dynamic Execution Methods

85% match2024

BASS: Batched Attention-optimized Speculative Sampling

83% match2024

Speculative Sampling with Reinforcement Learning

80% match2026

QuantSpec: Self-Speculative Decoding with Hierarchical Quantized KV Cache

80% match2025

Speculative Decoding and Beyond: An In-Depth Survey of Techniques

80% match2025

The N-Grammys: Accelerating Autoregressive Inference with Learning-Free Batched Speculation

80% match2024

Cite this paper

@article{barad2023leveraging,
  title  = {Leveraging Speculative Sampling and KV-Cache Optimizations Together for Generative AI using OpenVINO},
  author = {Haim Barad and Ekaterina Aidova and Yury Gorbachev},
  year   = {2023},
  eprint = {2311.04951},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2311.04951v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2311.04951.svg)](https://codeofpaper.com/paper/2311.04951)

Discussion