Serving · KernelIndex

6 comparison groups · 15 feasible · 0 excluded

Ranked only within one model, workload, protocol, hardware, and quality bar; a run missing a bounded metric is left out, not guessed.

llama3.1-405b · Interactive

8× NVIDIA B200-SXM-180GB · mlperf_closed_99

Ranked only inside this comparison group · 10 feasible · TTFT p99 reported for 0 of 10

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

Nebius · Nebius B200 (8x B200-SXM-180GB, TensorRT)

TensorRT 10.11, CUDA 12.9

771

not reported

8× NVIDIA B200-SXM-180GB · mlperf_loadgen v5.1 · mlperf_closed_99 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0073Run detail →

Lenovo · ThinkSystem SR780a V3 (8x B200-SXM-180GB, TensorRT)

TensorRT 10.8, CUDA 12.8

753

not reported

8× NVIDIA B200-SXM-180GB · mlperf_loadgen v5.1 · mlperf_closed_99 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0062Run detail →

Dell · Dell PowerEdge XE9685L (8x B200-SXM-180GB, TensorRT)

TensorRT 10.11, CUDA 12.8

752

not reported

8× NVIDIA B200-SXM-180GB · mlperf_loadgen v5.1 · mlperf_closed_99 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0028Run detail →

Supermicro · SYS-422GA-NBRT-LCC (8x B200-SXM-180GB, TensorRT)

TensorRT 10.11, CUDA 12.9

751

not reported

8× NVIDIA B200-SXM-180GB · mlperf_loadgen v5.1 · mlperf_closed_99 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0087Run detail →

Dell · Dell PowerEdge XE9680L (8x B200-SXM-180GB, TensorRT)

TensorRT 10.11, CUDA 12.9

751

not reported

8× NVIDIA B200-SXM-180GB · mlperf_loadgen v5.1 · mlperf_closed_99 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0024Run detail →

NVIDIA · NVIDIA DGX B200 (8x B200-SXM-180GB, TensorRT)

TensorRT 10.11, CUDA 12.9

751

not reported

8× NVIDIA B200-SXM-180GB · mlperf_loadgen v5.1 · mlperf_closed_99 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0069Run detail →

Supermicro · SYS-422GS-NBRT-LCC (8x B200-SXM-180GB, TensorRT)

TensorRT 10.11, CUDA 12.9

751

not reported

8× NVIDIA B200-SXM-180GB · mlperf_loadgen v5.1 · mlperf_closed_99 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0088Run detail →

Google · NVIDIA DGX B200 (8x B200-SXM-180GB, TensorRT)

TensorRT 10.11, CUDA 12.9

750

not reported

8× NVIDIA B200-SXM-180GB · mlperf_loadgen v5.1 · mlperf_closed_99 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0047Run detail →

Show all 10 configurations ›

Supermicro · SYS-A21GE-NBRT (8x B200-SXM-180GB, TensorRT)

TensorRT 10.11, CUDA 12.9

750

not reported

8× NVIDIA B200-SXM-180GB · mlperf_loadgen v5.1 · mlperf_closed_99 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0089Run detail →

GigaComputing · G894-SD1

TensorRT 10.11, CUDA 12.9

719

not reported

8× NVIDIA B200-SXM-180GB · mlperf_loadgen v5.1 · mlperf_closed_99 · Reported evidence · observed 2025-09-04

llama3.1-405b · Interactive

8× NVIDIA B300-SXM-270GB · mlperf_closed_99

Ranked only inside this comparison group · 1 feasible · TTFT p99 reported for 0 of 1

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

GigaComputing · G894-SD3-AAX7

TensorRT 10.14, CUDA 13.1

765

not reported

8× NVIDIA B300-SXM-270GB · mlperf_loadgen v6.0 · mlperf_closed_99 · Reported evidence · observed 2026-03-26

MLPerf Inference · entry v6.0/6.0-0039Run detail →

llama3.1-405b · Interactive

4× NVIDIA GB300 × 18 nodes · mlperf_closed_99

Ranked only inside this comparison group · 1 feasible · TTFT p99 reported for 0 of 1

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

NVIDIA · NVIDIA GB300 NVL72 (72x GB300-288GB_aarch64, TensorRT)

TensorRT 10.14, CUDA 13.1, cuDNN 9.17, TensorRT-LLM feat/1.2-mlpinf, NVIDIA Dynamo mlperf-v6.0-dynamo-v0.8.0, vLLM CentM

18,365

not reported

4× NVIDIA GB300 × 18 nodes · mlperf_loadgen v6.0 · mlperf_closed_99 · Reported evidence · observed 2026-03-26

MLPerf Inference · entry v6.0/6.0-0078Run detail →

llama3.1-405b · Interactive

4× NVIDIA GB200 × 18 nodes · mlperf_closed_99

Ranked only inside this comparison group · 1 feasible · TTFT p99 reported for 0 of 1

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

NVIDIA · NVIDIA GB200 NVL72 (72x GB200-186GB_aarch64, TensorRT)

TensorRT 10.14, CUDA 13.1, cuDNN 9.17, TensorRT-LLM feat/1.2-mlpinf, NVIDIA Dynamo mlperf-v6.0-dynamo-v0.8.0, vLLM CentM

14,010

not reported

4× NVIDIA GB200 × 18 nodes · mlperf_loadgen v6.0 · mlperf_closed_99 · Reported evidence · observed 2026-03-26

MLPerf Inference · entry v6.0/6.0-0075Run detail →

llama3.1-405b · Interactive

8× NVIDIA H200-SXM-141GB · mlperf_closed_99

Ranked only inside this comparison group · 1 feasible · TTFT p99 reported for 0 of 1

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

Nebius · Nebius H200 (8x H200-SXM-141GB, TensorRT)

TensorRT 10.11, CUDA 12.9

203

not reported

8× NVIDIA H200-SXM-141GB · mlperf_loadgen v5.1 · mlperf_closed_99 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0075Run detail →

llama3.1-405b · Interactive

4× NVIDIA GB200 × 18 nodes · mlperf_closed_99

Ranked only inside this comparison group · 1 feasible · TTFT p99 reported for 0 of 1

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

NVIDIA · (72x GB200-186GB_aarch64, TensorRT)

TensorRT 10.11, CUDA 12.9

9,921

not reported

4× NVIDIA GB200 × 18 nodes · mlperf_loadgen v5.1 · mlperf_closed_99 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0071Run detail →