Serving · KernelIndex

13 comparison groups · 30 feasible · 0 excluded

Ranked only within one model, workload, protocol, hardware, and quality bar; a run missing a bounded metric is left out, not guessed.

llama3.1-405b · Server

8× NVIDIA B200-SXM-180GB · mlperf_closed_99

Ranked only inside this comparison group · 13 feasible · TTFT p99 reported for 0 of 13

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

Nebius · Nebius B200 (8x B200-SXM-180GB, TensorRT)

TensorRT 10.11, CUDA 12.9

1,280

not reported

8× NVIDIA B200-SXM-180GB · mlperf_loadgen v5.1 · mlperf_closed_99 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0073Run detail →

Lenovo · ThinkSystem SR780a V3 (8x B200-SXM-180GB, TensorRT)

TensorRT 10.8, CUDA 12.8

1,249

not reported

8× NVIDIA B200-SXM-180GB · mlperf_loadgen v5.1 · mlperf_closed_99 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0062Run detail →

Google · NVIDIA DGX B200 (8x B200-SXM-180GB, TensorRT)

TensorRT 10.11, CUDA 12.9

1,247

not reported

8× NVIDIA B200-SXM-180GB · mlperf_loadgen v5.1 · mlperf_closed_99 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0047Run detail →

Lambda · NVIDIA DGX B200 (8x B200-SXM-180GB, TensorRT)

TensorRT 10.11, CUDA 12.9

1,247

not reported

8× NVIDIA B200-SXM-180GB · mlperf_loadgen v5.1 · mlperf_closed_99 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0060Run detail →

Dell · Dell PowerEdge XE9685L (8x B200-SXM-180GB, TensorRT)

TensorRT 10.11, CUDA 12.8

1,247

not reported

8× NVIDIA B200-SXM-180GB · mlperf_loadgen v5.1 · mlperf_closed_99 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0028Run detail →

GigaComputing · G894-SD1

TensorRT 10.11, CUDA 12.9

1,245

not reported

8× NVIDIA B200-SXM-180GB · mlperf_loadgen v5.1 · mlperf_closed_99 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0046Run detail →

Supermicro · SYS-422GS-NBRT-LCC (8x B200-SXM-180GB, TensorRT)

TensorRT 10.11, CUDA 12.9

1,245

not reported

8× NVIDIA B200-SXM-180GB · mlperf_loadgen v5.1 · mlperf_closed_99 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0088Run detail →

Broadcom_Supermicro · Supermicro SYS-422GA-NBRT-LCC (8x B200-SXM-180GB, TensorRT)

TensorRT 10.11, CUDA 12.8

1,244

not reported

8× NVIDIA B200-SXM-180GB · mlperf_loadgen v5.1 · mlperf_closed_99 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0009Run detail →

Show all 13 configurations ›

Oracle · BM.GPU.B200.8

TensorRT 10.11, CUDA 12.9

1,244

not reported

8× NVIDIA B200-SXM-180GB · mlperf_loadgen v5.1 · mlperf_closed_99 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0076Run detail →

Dell · Dell PowerEdge XE9680L (8x B200-SXM-180GB, TensorRT)

TensorRT 10.11, CUDA 12.9

1,243

not reported

llama3.1-405b · Server

8× NVIDIA B300-SXM-270GB · mlperf_closed_99

Ranked only inside this comparison group · 4 feasible · TTFT p99 reported for 0 of 4

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

Quanta_Cloud_Technology · QuantaGrid D75H-10U (8x B300-SXM-270GB, TensorRT)

TensorRT 10.14, CUDA 13.1

1,484

not reported

8× NVIDIA B300-SXM-270GB · mlperf_loadgen v6.0 · mlperf_closed_99 · Reported evidence · observed 2026-03-26

MLPerf Inference · entry v6.0/6.0-0089Run detail →

NVIDIA · NVIDIA DGX B300 (8x B300-SXM-270GB, TensorRT)

TensorRT 10.14, CUDA 13.1, cuDNN 9.17, TensorRT-LLM feat/1.2-mlpinf, NVIDIA Dynamo mlperf-v6.0-dynamo-v0.8.0, vLLM CentM

llama3.1-405b · Server

4× NVIDIA GB200 · mlperf_closed_99

Ranked only inside this comparison group · 2 feasible · TTFT p99 reported for 0 of 2

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

Nebius · Nebius GB200 (4x GB200-186GB_aarch64, TensorRT)

TensorRT 10.11, CUDA 12.9

596

not reported

4× NVIDIA GB200 · mlperf_loadgen v5.1 · mlperf_closed_99 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0074Run detail →

Azure · ND_GB200_v6

TensorRT 10.11, CUDA 12.9

584

llama3.1-405b · Server

8× NVIDIA H200-SXM-141GB · mlperf_closed_99

Ranked only inside this comparison group · 2 feasible · TTFT p99 reported for 0 of 2

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

Quanta_Cloud_Technology · QuantaGrid D74H-7U (8x H200-SXM-141GB, TensorRT)

TensorRT 10.8, CUDA 12.8

296

not reported

8× NVIDIA H200-SXM-141GB · mlperf_loadgen v5.1 · mlperf_closed_99 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0079Run detail →

Nebius · Nebius H200 (8x H200-SXM-141GB, TensorRT)

TensorRT 10.11, CUDA 12.9

295

llama3.1-405b · Server

4× NVIDIA GB300 × 18 nodes · mlperf_closed_99

Ranked only inside this comparison group · 1 feasible · TTFT p99 reported for 0 of 1

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

NVIDIA · NVIDIA GB300 NVL72 (72x GB300-288GB_aarch64, TensorRT)

TensorRT 10.14, CUDA 13.1, cuDNN 9.17, TensorRT-LLM feat/1.2-mlpinf, NVIDIA Dynamo mlperf-v6.0-dynamo-v0.8.0, vLLM CentM

18,628

not reported

4× NVIDIA GB300 × 18 nodes · mlperf_loadgen v6.0 · mlperf_closed_99 · Reported evidence · observed 2026-03-26

MLPerf Inference · entry v6.0/6.0-0078Run detail →

llama3.1-405b · Server

8× NVIDIA H100-SXM-80GB × 4 nodes · mlperf_closed_99

Ranked only inside this comparison group · 1 feasible · TTFT p99 reported for 0 of 1

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

Cisco · HPF HGX System (32xH100-SXM-80GB, TensorRT)

TensorRT 10.8, CUDA 12.8

1,127

not reported

8× NVIDIA H100-SXM-80GB × 4 nodes · mlperf_loadgen v5.1 · mlperf_closed_99 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0016Run detail →

llama3.1-405b · Server

4× NVIDIA GB200 × 18 nodes · mlperf_closed_99

Ranked only inside this comparison group · 1 feasible · TTFT p99 reported for 0 of 1

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

NVIDIA · NVIDIA GB200 NVL72 (72x GB200-186GB_aarch64, TensorRT)

TensorRT 10.14, CUDA 13.1, cuDNN 9.17, TensorRT-LLM feat/1.2-mlpinf, NVIDIA Dynamo mlperf-v6.0-dynamo-v0.8.0, vLLM CentM

14,134

not reported

4× NVIDIA GB200 × 18 nodes · mlperf_loadgen v6.0 · mlperf_closed_99 · Reported evidence · observed 2026-03-26

MLPerf Inference · entry v6.0/6.0-0075Run detail →

llama3.1-405b · Server

8× NVIDIA H200-SXM-141GB × 2 nodes · mlperf_closed_99

Ranked only inside this comparison group · 1 feasible · TTFT p99 reported for 0 of 1

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

Cisco · Cisco UCS C885A M8 (16x H200-SXM-141GB, TensorRT)

TensorRT 10.8, CUDA 12.8

584

not reported

8× NVIDIA H200-SXM-141GB × 2 nodes · mlperf_loadgen v5.1 · mlperf_closed_99 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0013Run detail →

llama3.1-405b · Server

8× NVIDIA B200-SXM-180GB · mlperf_closed_99

Ranked only inside this comparison group · 1 feasible · TTFT p99 reported for 0 of 1

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

Google · NVIDIA DGX B200 (8x B200-SXM-180GB, TensorRT)

TensorRT 10.14, CUDA 13.1

984

not reported

8× NVIDIA B200-SXM-180GB · mlperf_loadgen v6.0 · mlperf_closed_99 · Reported evidence · observed 2026-03-26

MLPerf Inference · entry v6.0/6.0-0040Run detail →

llama3.1-405b · Server

4× NVIDIA GB200 × 18 nodes · mlperf_closed_99

Ranked only inside this comparison group · 1 feasible · TTFT p99 reported for 0 of 1

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

NVIDIA · (72x GB200-186GB_aarch64, TensorRT)

TensorRT 10.11, CUDA 12.9

11,614

not reported

4× NVIDIA GB200 × 18 nodes · mlperf_loadgen v5.1 · mlperf_closed_99 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0071Run detail →

llama3.1-405b · Server

8× Virtualized NVIDIA H200-SXM-141GB · mlperf_closed_99

Ranked only inside this comparison group · 1 feasible · TTFT p99 reported for 0 of 1

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

Dell_Broadcom · Dell PowerEdge XE9680 (8x H200-SXM-141GB,VMware ESXi 9.0.0)

TensorRT 10.11, CUDA 12.8

277

not reported

8× Virtualized NVIDIA H200-SXM-141GB · mlperf_loadgen v5.1 · mlperf_closed_99 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0029Run detail →

llama3.1-405b · Server

4× NVIDIA GB300 × 18 nodes · mlperf_closed_99

Ranked only inside this comparison group · 1 feasible · TTFT p99 reported for 0 of 1

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

NVIDIA · (72x GB300-288GB_aarch64, TensorRT)

TensorRT 10.13, CUDA 13.0

12,248

not reported

4× NVIDIA GB300 × 18 nodes · mlperf_loadgen v5.1 · mlperf_closed_99 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0072Run detail →

1 more comparison groups. Narrow by model, workload, or hardware to see them.