Optimization of Armv9 architecture general large language model inference performance based on Llama.cpp

Longhao Chen, Yina Zhao, Qiangjun Xie, Qinghua Sheng
6/16/2024
cs.PLcs.AIcs.AR

No confident code implementation found for this paper. 1 possible match can be shown below.

Abstract

This article optimizes the inference performance of the Qwen-1.8B model by performing Int8 quantization, vectorizing some operators in llama.cpp, and modifying the compilation script to improve the compiler optimization level. On the Yitian 710 experimental platform, the prefill performance is increased by 1.6 times, the decoding performance is increased by 24 times, the memory usage is reduced to 1/5 of the original, and the accuracy loss is almost negligible.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. 1 weaker match is hidden by default — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

Patched MOA: optimizing inference for diverse software development tasks

88% match1 repo4.3k2024

Inference Optimizations for Large Language Models: Effects, Challenges, and Practical Considerations

87% match2024

BaseRT: Best-in-Class LLM Inference on Apple Silicon via Native Metal

86% match2 repos82026

MoE-Lens: Towards the Hardware Limit of High-Throughput MoE LLM Serving Under Resource Constraints

85% match2025

Accelerating Masked Diffusion Large Language Models: A Survey of Efficient Inference Techniques

85% match2026

GUIDE: A Global Unified Inference Engine for Deploying Large Language Models in Heterogeneous Environments

85% match2024

Cite this paper

@article{chen2024optimization,
  title  = {Optimization of Armv9 architecture general large language model inference performance based on Llama.cpp},
  author = {Longhao Chen and Yina Zhao and Qiangjun Xie and Qinghua Sheng},
  year   = {2024},
  eprint = {2406.10816},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2406.10816v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2406.10816.svg)](https://codeofpaper.com/paper/2406.10816)

Discussion