Serving · KernelIndex

32 comparison groups · 42 feasible · 0 excluded

Ranked only within one model, workload, protocol, hardware, and quality bar; a run missing a bounded metric is left out, not guessed.

Llama-2-70B (MLPerf reference) · llama2-70b-99 · Server

4× NVIDIA GB200 · mlperf_closed_99

Ranked only inside this comparison group · 3 feasible · TTFT p99 reported for 0 of 3

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

NVIDIA · (4x GB200-186GB_aarch64, TensorRT)

TensorRT 10.11, CUDA 12.9

49,360

not reported

4× NVIDIA GB200 · mlperf_loadgen v5.1 · mlperf_closed_99 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0070Run detail →

Nebius · Nebius GB200 (4x GB200-186GB_aarch64, TensorRT)

TensorRT 10.11, CUDA 12.9

49,216

not reported

4× NVIDIA GB200 · mlperf_loadgen v5.1 · mlperf_closed_99 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0074Run detail →

Azure · ND_GB200_v6

TensorRT 10.11, CUDA 12.9

46,017

not reported

4× NVIDIA GB200 · mlperf_loadgen v5.1 · mlperf_closed_99 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0008Run detail →

Llama-2-70B (MLPerf reference) · llama2-70b-99 · Offline

4× NVIDIA GB200 · mlperf_closed_99

Ranked only inside this comparison group · 3 feasible · TTFT p99 reported for 0 of 3

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

Azure · ND_GB200_v6

TensorRT 10.11, CUDA 12.9

52,062

not reported

4× NVIDIA GB200 · mlperf_loadgen v5.1 · mlperf_closed_99 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0008Run detail →

NVIDIA · (4x GB200-186GB_aarch64, TensorRT)

TensorRT 10.11, CUDA 12.9

51,737

Llama-3.1-405B (MLPerf reference) · llama3.1-405b · Offline

4× NVIDIA GB200 · mlperf_closed_99

Ranked only inside this comparison group · 2 feasible · TTFT p99 reported for 0 of 2

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

Nebius · Nebius GB200 (4x GB200-186GB_aarch64, TensorRT)

TensorRT 10.11, CUDA 12.9

856

not reported

4× NVIDIA GB200 · mlperf_loadgen v5.1 · mlperf_closed_99 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0074Run detail →

Azure · ND_GB200_v6

TensorRT 10.11, CUDA 12.9

847

DeepSeek-R1 (MLPerf reference) · deepseek-r1 · Server

4× NVIDIA GB200 × 18 nodes · mlperf_closed_99

Ranked only inside this comparison group · 2 feasible · TTFT p99 reported for 0 of 2

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

NVIDIA · NVIDIA GB200 NVL72 (72x GB200-186GB_aarch64, TensorRT)

TensorRT 10.14, CUDA 13.1, cuDNN 9.17, TensorRT-LLM feat/1.2-mlpinf, NVIDIA Dynamo mlperf-v6.0-dynamo-v0.8.0, vLLM CentM

336,106

not reported

4× NVIDIA GB200 × 18 nodes · mlperf_loadgen v6.0 · mlperf_closed_99 · Reported evidence · observed 2026-03-26

MLPerf Inference · entry v6.0/6.0-0075Run detail →

Google · NVIDIA GB200 NVL72 (72x GB200-186GB_aarch64, TensorRT)

Llama-3.1-405B (MLPerf reference) · llama3.1-405b · Server

4× NVIDIA GB200 · mlperf_closed_99

Ranked only inside this comparison group · 2 feasible · TTFT p99 reported for 0 of 2

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

Nebius · Nebius GB200 (4x GB200-186GB_aarch64, TensorRT)

TensorRT 10.11, CUDA 12.9

596

not reported

4× NVIDIA GB200 · mlperf_loadgen v5.1 · mlperf_closed_99 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0074Run detail →

Azure · ND_GB200_v6

TensorRT 10.11, CUDA 12.9

584

Llama-2-70B (MLPerf reference) · llama2-70b-99 · Server

4× NVIDIA GB200 · mlperf_closed_99_9

Ranked only inside this comparison group · 2 feasible · TTFT p99 reported for 0 of 2

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

NVIDIA · (4x GB200-186GB_aarch64, TensorRT)

TensorRT 10.11, CUDA 12.9

49,360

not reported

4× NVIDIA GB200 · mlperf_loadgen v5.1 · mlperf_closed_99_9 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0070Run detail →

Nebius · Nebius GB200 (4x GB200-186GB_aarch64, TensorRT)

TensorRT 10.11, CUDA 12.9

49,216

DeepSeek-R1 (MLPerf reference) · deepseek-r1 · Offline

4× NVIDIA GB200 × 18 nodes · mlperf_closed_99

Ranked only inside this comparison group · 2 feasible · TTFT p99 reported for 0 of 2

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

NVIDIA · NVIDIA GB200 NVL72 (72x GB200-186GB_aarch64, TensorRT)

TensorRT 10.14, CUDA 13.1, cuDNN 9.17, TensorRT-LLM feat/1.2-mlpinf, NVIDIA Dynamo mlperf-v6.0-dynamo-v0.8.0, vLLM CentM

486,141

not reported

4× NVIDIA GB200 × 18 nodes · mlperf_loadgen v6.0 · mlperf_closed_99 · Reported evidence · observed 2026-03-26

MLPerf Inference · entry v6.0/6.0-0075Run detail →

Google · NVIDIA GB200 NVL72 (72x GB200-186GB_aarch64, TensorRT)

Llama-2-70B (MLPerf reference) · llama2-70b-99 · Offline

4× NVIDIA GB200 · mlperf_closed_99_9

Ranked only inside this comparison group · 2 feasible · TTFT p99 reported for 0 of 2

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

NVIDIA · (4x GB200-186GB_aarch64, TensorRT)

TensorRT 10.11, CUDA 12.9

51,737

not reported

4× NVIDIA GB200 · mlperf_loadgen v5.1 · mlperf_closed_99_9 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0070Run detail →

Nebius · Nebius GB200 (4x GB200-186GB_aarch64, TensorRT)

TensorRT 10.11, CUDA 12.9

49,684

DeepSeek-R1 (MLPerf reference) · deepseek-r1 · Offline

4× NVIDIA GB200 × 18 nodes · mlperf_closed_99

Ranked only inside this comparison group · 1 feasible · TTFT p99 reported for 0 of 1

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

NVIDIA · (72x GB200-186GB_aarch64, TensorRT)

TensorRT 10.11, CUDA 12.9

289,712

not reported

4× NVIDIA GB200 × 18 nodes · mlperf_loadgen v5.1 · mlperf_closed_99 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0071Run detail →

Llama-2-70B (MLPerf reference) · llama2-70b-99 · Offline

4× NVIDIA GB200 × 18 nodes · mlperf_closed_99_9

Ranked only inside this comparison group · 1 feasible · TTFT p99 reported for 0 of 1

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

NVIDIA · NVIDIA GB200 NVL72 (72x GB200-186GB_aarch64, TensorRT)

TensorRT 10.14, CUDA 13.1, cuDNN 9.17, TensorRT-LLM feat/1.2-mlpinf, NVIDIA Dynamo mlperf-v6.0-dynamo-v0.8.0, vLLM CentM

888,054

not reported

4× NVIDIA GB200 × 18 nodes · mlperf_loadgen v6.0 · mlperf_closed_99_9 · Reported evidence · observed 2026-03-26

MLPerf Inference · entry v6.0/6.0-0075Run detail →

Llama-2-70B (MLPerf reference) · llama2-70b-99 · Interactive

4× NVIDIA GB200 · mlperf_closed_99

Ranked only inside this comparison group · 1 feasible · TTFT p99 reported for 0 of 1

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

NVIDIA · (4x GB200-186GB_aarch64, TensorRT)

TensorRT 10.11, CUDA 12.9

29,746

not reported

4× NVIDIA GB200 · mlperf_loadgen v5.1 · mlperf_closed_99 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0070Run detail →

DeepSeek-R1 (MLPerf reference) · deepseek-r1 · Interactive

4× NVIDIA GB200 × 18 nodes · mlperf_closed_99

Ranked only inside this comparison group · 1 feasible · TTFT p99 reported for 0 of 1

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

NVIDIA · NVIDIA GB200 NVL72 (72x GB200-186GB_aarch64, TensorRT)

TensorRT 10.14, CUDA 13.1, cuDNN 9.17, TensorRT-LLM feat/1.2-mlpinf, NVIDIA Dynamo mlperf-v6.0-dynamo-v0.8.0, vLLM CentM

240,318

not reported

4× NVIDIA GB200 × 18 nodes · mlperf_loadgen v6.0 · mlperf_closed_99 · Reported evidence · observed 2026-03-26

MLPerf Inference · entry v6.0/6.0-0075Run detail →

20 more comparison groups. Narrow by model, workload, or hardware to see them.