Confidential Computing on NVIDIA Hopper GPUs: A Performance Benchmark Study

Jianwei Zhu, Hang Yin, Peng Deng, Aline Almeida, Shunfan Zhou
9/6/2024
cs.DCcs.AIcs.PF
benchmark-dataset

No official code found. Best implementation: Phala-Network/hopper-llm-benchmark (5 stars, last commit 2024-11-06).

Abstract

This report evaluates the performance impact of enabling Trusted Execution Environments (TEE) on NVIDIA Hopper GPUs for large language model (LLM) inference tasks. We benchmark the overhead introduced by TEE mode across various LLMs and token lengths, with a particular focus on the bottleneck caused by CPU-GPU data transfers via PCIe. Our results indicate that while there is minimal computational overhead within the GPU, the overall performance penalty is primarily attributable to data transfer. For the majority of typical LLM queries, the overhead remains below 7%, with larger models and longer sequences experiencing nearly zero overhead.

View on arXivView PDF

Code Implementations(1)

Phala-Network/hopper-llm-benchmarkCommunity72%

Confidential Computing on nVIDIA Hopper GPUs: A Performance Benchmark Study

50TeX, PythonSep 6, 20241 years ago

Similar Papers

MoE-Lens: Towards the Hardware Limit of High-Throughput MoE LLM Serving Under Resource Constraints

87% match2025

SparQ Attention: Bandwidth-Efficient LLM Inference

86% match1 repo362023

Local Inference of LLMs on Long Contexts: A Data Systems Perspective

86% match2026

ES-dLLM: Efficient Inference for Diffusion Large Language Models by Early-Skipping

86% match2026

GUIDE: A Global Unified Inference Engine for Deploying Large Language Models in Heterogeneous Environments

86% match2024

Optimizing Small Language Models for NL2SQL via Chain-of-Thought Fine-Tuning

86% match2026

Cite this paper

@article{zhu2024confidential,
  title  = {Confidential Computing on NVIDIA Hopper GPUs: A Performance Benchmark Study},
  author = {Jianwei Zhu and Hang Yin and Peng Deng and Aline Almeida and Shunfan Zhou},
  year   = {2024},
  eprint = {2409.03992v4},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2409.03992v4}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2409.03992v4.svg)](https://codeofpaper.com/paper/2409.03992v4)

Discussion