---
title: "Discussion on \"The Numbers: Benchmarking My LLM Gateway on a H100\" | Hashnode"
description: "Discussion on \"The Numbers: Benchmarking My LLM Gateway on a H100\". Real H100 benchmark of a production LLM gateway vs a naive HuggingFace baseline continuous batching, streaming TTFT, graceful 429s."
url: "https://hashnode.com/posts/the-numbers-benchmarking-my-llm-gateway-on-a-h100/6a37c1660984820a823b1126"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [The Numbers: Benchmarking My LLM Gateway on a H100](https://orhunkupeli.hashnode.dev/the-numbers-benchmarking-my-llm-gateway-on-a-h100?utm_source=hashnode&utm_medium=feed)

Real H100 benchmark of a production LLM gateway vs a naive HuggingFace baseline continuous batching, streaming TTFT, graceful 429s.

Jun 21 · 5 min read · orhunkupeli.hashnode.dev  
Read the article

-   [#llm-gateway](https://hashnode.com/tag/llm-gateway)
-   [#vllm](https://hashnode.com/tag/vllm)
-   [#nvidia-a100](https://hashnode.com/tag/nvidia-a100)
-   [#llm-benchmarking](https://hashnode.com/tag/llm-benchmarking)
-   [#continuous-batching](https://hashnode.com/tag/continuous-batching)
-   [#qwen25](https://hashnode.com/tag/qwen25)
-   [#ai-infrastructure](https://hashnode.com/tag/ai-infrastructure)
-   [#llmops](https://hashnode.com/tag/llmops)
-   [#time-to-first-token](https://hashnode.com/tag/time-to-first-token)
-   [#modelserving](https://hashnode.com/tag/modelserving)
-   [#ai-inference-optimization](https://hashnode.com/tag/ai-inference-optimization)
-   [#awq-quantization](https://hashnode.com/tag/awq-quantization)

## Written by

[Orhun Küpeli](https://hashnode.com/@orhunkupeli)

@orhunkupeli · Munich, Germany

  
[View profile](https://hashnode.com/@orhunkupeli)

## Responses

Markdown supported · ⌘↵ to post

No responses yet.