Serving · KernelIndex

73 comparison groups · 163 feasible · 0 excluded

Ranked only within one model, workload, protocol, hardware, and quality bar; a run missing a bounded metric is left out, not guessed.

llama2-70b-99 · Server

8× NVIDIA B200-SXM-180GB · mlperf_closed_99_9

Ranked only inside this comparison group · 11 feasible · TTFT p99 reported for 0 of 11

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

Nebius · Nebius B200 (8x B200-SXM-180GB, TensorRT)

TensorRT 10.11, CUDA 12.9

101,611

not reported

8× NVIDIA B200-SXM-180GB · mlperf_loadgen v5.1 · mlperf_closed_99_9 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0073Run detail →

Dell · Dell PowerEdge XE9685L (8x B200-SXM-180GB, TensorRT)

TensorRT 10.11, CUDA 12.8

99,214

not reported

8× NVIDIA B200-SXM-180GB · mlperf_loadgen v5.1 · mlperf_closed_99_9 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0028Run detail →

Lenovo · ThinkSystem SR680a V3 (8x B200-SXM-180GB, TensorRT)

TensorRT 10.8, CUDA 12.8

99,204

not reported

8× NVIDIA B200-SXM-180GB · mlperf_loadgen v5.1 · mlperf_closed_99_9 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0061Run detail →

Supermicro · SYS-422GA-NBRT-LCC (8x B200-SXM-180GB, TensorRT)

TensorRT 10.11, CUDA 12.9

99,196

not reported

8× NVIDIA B200-SXM-180GB · mlperf_loadgen v5.1 · mlperf_closed_99_9 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0087Run detail →

Supermicro · AS-4126GS-NBR-LCC (8x B200-SXM-180GB, TensorRT)

TensorRT 10.11, CUDA 12.9

99,182

not reported

8× NVIDIA B200-SXM-180GB · mlperf_loadgen v5.1 · mlperf_closed_99_9 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0086Run detail →

Supermicro · SYS-422GS-NBRT-LCC (8x B200-SXM-180GB, TensorRT)

TensorRT 10.11, CUDA 12.9

99,180

not reported

8× NVIDIA B200-SXM-180GB · mlperf_loadgen v5.1 · mlperf_closed_99_9 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0088Run detail →

Google · NVIDIA DGX B200 (8x B200-SXM-180GB, TensorRT)

TensorRT 10.11, CUDA 12.9

99,170

not reported

8× NVIDIA B200-SXM-180GB · mlperf_loadgen v5.1 · mlperf_closed_99_9 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0047Run detail →

Dell · Dell PowerEdge XE9680L (8x B200-SXM-180GB, TensorRT)

TensorRT 10.11, CUDA 12.9

99,139

not reported

8× NVIDIA B200-SXM-180GB · mlperf_loadgen v5.1 · mlperf_closed_99_9 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0024Run detail →

Show all 11 configurations ›

Supermicro · SYS-A21GE-NBRT (8x B200-SXM-180GB, TensorRT)

TensorRT 10.11, CUDA 12.9

99,128

not reported

8× NVIDIA B200-SXM-180GB · mlperf_loadgen v5.1 · mlperf_closed_99_9 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0089Run detail →

NVIDIA · NVIDIA DGX B200 (8x B200-SXM-180GB, TensorRT)

TensorRT 10.11, CUDA 12.9

99,123

not reported

llama2-70b-99 · Server

8× NVIDIA B200-SXM-180GB · mlperf_closed_99

Ranked only inside this comparison group · 10 feasible · TTFT p99 reported for 0 of 10

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

Nebius · Nebius B200 (8x B200-SXM-180GB, TensorRT)

TensorRT 10.11, CUDA 12.9

101,611

not reported

8× NVIDIA B200-SXM-180GB · mlperf_loadgen v5.1 · mlperf_closed_99 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0073Run detail →

Lambda · NVIDIA DGX B200 (8x B200-SXM-180GB, TensorRT)

TensorRT 10.11, CUDA 12.9

99,994

llama2-70b-99 · Server

8× AMD Instinct MI325X 256GB HBM3e · mlperf_closed_99_9

Ranked only inside this comparison group · 8 feasible · TTFT p99 reported for 0 of 8

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

GigaComputing · G893-ZX1-AAX2

vLLM 0.6.5.dev964+mlperf50, Pytorch 2.7.0a0+git3a58512, ROCm 6.3.1

32,139

not reported

8× AMD Instinct MI325X 256GB HBM3e · mlperf_loadgen v5.1 · mlperf_closed_99_9 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0045Run detail →

AMD · QuantaGrid D74A-7U

vLLM 0.9.0.2.dev108+g71faa1880.d20250731.rocm641, Pytorch 2.7.0+gitf717b2a, ROCm 6.4.1.60401-83~22.04

llama2-70b-99 · Server

8× AMD Instinct MI325X 256GB HBM3e · mlperf_closed_99

Ranked only inside this comparison group · 8 feasible · TTFT p99 reported for 0 of 8

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

GigaComputing · G893-ZX1-AAX2

vLLM 0.6.5.dev964+mlperf50, Pytorch 2.7.0a0+git3a58512, ROCm 6.3.1

32,139

not reported

8× AMD Instinct MI325X 256GB HBM3e · mlperf_loadgen v5.1 · mlperf_closed_99 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0045Run detail →

AMD · QuantaGrid D74A-7U

vLLM 0.9.0.2.dev108+g71faa1880.d20250731.rocm641, Pytorch 2.7.0+gitf717b2a, ROCm 6.4.1.60401-83~22.04

llama2-70b-99 · Server

8× NVIDIA H200-SXM-141GB · mlperf_closed_99_9

Ranked only inside this comparison group · 7 feasible · TTFT p99 reported for 0 of 7

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

Nebius · Nebius H200 (8x H200-SXM-141GB, TensorRT)

TensorRT 10.11, CUDA 12.9

34,029

not reported

8× NVIDIA H200-SXM-141GB · mlperf_loadgen v5.1 · mlperf_closed_99_9 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0075Run detail →

Quanta_Cloud_Technology · QuantaGrid D74H-7U (8x H200-SXM-141GB, TensorRT)

TensorRT 10.8, CUDA 12.8

33,353

llama2-70b-99 · Server

8× AMD Instinct MI355X 288GB HBM3e · mlperf_closed_99

Ranked only inside this comparison group · 7 feasible · TTFT p99 reported for 0 of 7

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

AMD · smci355-ccs-aus-m09-17 (AS -4126GS-NMR-LCC)

PyTorch 2.9.1+git8907517, ROCm 7.0.0

100,282

not reported

8× AMD Instinct MI355X 288GB HBM3e · mlperf_loadgen v6.0 · mlperf_closed_99 · Reported evidence · observed 2026-03-26

MLPerf Inference · entry v6.0/6.0-0002Run detail →

ORACLE · BM.GPU.MI355X.8

Oracle submission (software not stated)

98,239

llama2-70b-99 · Server

8× AMD Instinct MI355X 288GB HBM3e · mlperf_closed_99_9

Ranked only inside this comparison group · 7 feasible · TTFT p99 reported for 0 of 7

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

AMD · smci355-ccs-aus-m09-17 (AS -4126GS-NMR-LCC)

PyTorch 2.9.1+git8907517, ROCm 7.0.0

100,282

not reported

8× AMD Instinct MI355X 288GB HBM3e · mlperf_loadgen v6.0 · mlperf_closed_99_9 · Reported evidence · observed 2026-03-26

MLPerf Inference · entry v6.0/6.0-0002Run detail →

ORACLE · BM.GPU.MI355X.8

Oracle submission (software not stated)

98,239

llama2-70b-99 · Server

8× NVIDIA H200-SXM-141GB · mlperf_closed_99

Ranked only inside this comparison group · 7 feasible · TTFT p99 reported for 0 of 7

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

ASUSTeK · ASUSTeK ESC N8 H200 (8x H200-SXM-141GB, TensorRT)

TensorRT 10.11, CUDA 12.9

34,194

not reported

8× NVIDIA H200-SXM-141GB · mlperf_loadgen v5.1 · mlperf_closed_99 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0007Run detail →

Nebius · Nebius H200 (8x H200-SXM-141GB, TensorRT)

TensorRT 10.11, CUDA 12.9

34,029

llama2-70b-99 · Server

8× NVIDIA RTX PRO 6000 Blackwell Server Edition · mlperf_closed_99

Ranked only inside this comparison group · 6 feasible · TTFT p99 reported for 0 of 6

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

HPE · HPE ProLiant Compute DL380a Gen12 (8x NVIDIA RTX PRO 6000 Blackwell Server Edition, TensorRT)

TensorRT 10.14.1.48, CUDA 13.0

28,728

not reported

8× NVIDIA RTX PRO 6000 Blackwell Server Edition · mlperf_loadgen v6.0 · mlperf_closed_99 · Reported evidence · observed 2026-03-26

MLPerf Inference · entry v6.0/6.0-0047Run detail →

ASUSTeK · ESC8000A_E13P_RTX_PRO_6000_PCIE_96GBx8

TensorRT 10.14, CUDA 13.1

llama2-70b-99 · Server

8× NVIDIA B300-SXM-270GB · mlperf_closed_99

Ranked only inside this comparison group · 5 feasible · TTFT p99 reported for 0 of 5

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

Cisco · Cisco UCS C880A M8 (8x NVIDIA B300-SXM-270GB, TensorRT)

TensorRT 10.13, CUDA 13.0

114,504

not reported

8× NVIDIA B300-SXM-270GB · mlperf_loadgen v6.0 · mlperf_closed_99 · Reported evidence · observed 2026-03-26

MLPerf Inference · entry v6.0/6.0-0010Run detail →

Dell · PowerEdge XE9780L (8x B300-SXM-270GB, TensorRT)

TensorRT 10.14, CUDA 13.1

108,392

llama2-70b-99 · Server

8× AMD Instinct MI300X 192GB HBM3 · mlperf_closed_99

Ranked only inside this comparison group · 4 feasible · TTFT p99 reported for 0 of 4

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

MangoBoost · Supermicro AS-8125GS-TNMR2 (8x MI300X-192GB MangoBoost-LLMBoost)

Mango LLMBoost AI Enterprise Platform, vllm-0.9.0.2.dev108+g71faa1880.d20250717.rocm641, ROCm 6.12.12

24,861

not reported

8× AMD Instinct MI300X 192GB HBM3 · mlperf_loadgen v5.1 · mlperf_closed_99 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0066Run detail →

Dell · Dell Poweredge XE9680

vLLM 0.6.5.dev964+mlperf50, Pytorch 2.7.0a0+git3a58512, ROCm 6.3.1

llama2-70b-99 · Server

8× NVIDIA RTX PRO 6000 Blackwell Server Edition · mlperf_closed_99_9

Ranked only inside this comparison group · 4 feasible · TTFT p99 reported for 0 of 4

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

HPE · HPE ProLiant Compute DL380a Gen12 (8x NVIDIA RTX PRO 6000 Blackwell Server Edition, TensorRT)

TensorRT 10.14.1.48, CUDA 13.0

28,717

not reported

8× NVIDIA RTX PRO 6000 Blackwell Server Edition · mlperf_loadgen v6.0 · mlperf_closed_99_9 · Reported evidence · observed 2026-03-26

MLPerf Inference · entry v6.0/6.0-0047Run detail →

Lenovo · ThinkSystem SR675i V3 (8x NVIDIA RTX PRO 6000 Blackwell Server Edition, TensorRT)

61 more comparison groups. Narrow by model, workload, or hardware to see them.