Serving · KernelIndex

9 comparison groups · 16 feasible · 0 excluded

Ranked only within one model, workload, protocol, hardware, and quality bar; a run missing a bounded metric is left out, not guessed.

mixtral-8x7b · Server

8× AMD Instinct MI325X 256GB HBM3e · mlperf_closed_99

Ranked only inside this comparison group · 6 feasible · TTFT p99 reported for 0 of 6

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

AMD · QuantaGrid D74A-7U

vLLM 0.9.0.2.dev108+g71faa1880.d20250731.rocm641, Pytorch 2.7.0+gitf717b2a, ROCm 6.4.1.60401-83~22.04

59,784

not reported

8× AMD Instinct MI325X 256GB HBM3e · mlperf_loadgen v5.1 · mlperf_closed_99 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0002Run detail →

Quanta_Cloud_Technology · D75T-7U_8xMI325X

vLLM 0.6.5.dev964+mlperf50, Pytorch 2.7.0a0+git3a58512, ROCm 6.3.1

59,041

not reported

8× AMD Instinct MI325X 256GB HBM3e · mlperf_loadgen v5.1 · mlperf_closed_99 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0081Run detail →

GigaComputing · G893-ZX1-AAX2

vLLM 0.6.5.dev964+mlperf50, Pytorch 2.7.0a0+git3a58512, ROCm 6.3.1

58,243

not reported

8× AMD Instinct MI325X 256GB HBM3e · mlperf_loadgen v5.1 · mlperf_closed_99 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0045Run detail →

MiTAC · G8825Z5

vLLM 0.6.5.dev964+mlperf50, Pytorch 2.7.0a0+git3a58512, ROCm 6.3.1

57,087

not reported

8× AMD Instinct MI325X 256GB HBM3e · mlperf_loadgen v5.1 · mlperf_closed_99 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0068Run detail →

Vultr · Supermicro AS -8126GS-TNMR (8x MI325X)

vLLM 0.6.5.dev964+mlperf50, Pytorch 2.7.0a0+git3a58512, ROCm 6.3.1

56,654

not reported

8× AMD Instinct MI325X 256GB HBM3e · mlperf_loadgen v5.1 · mlperf_closed_99 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0096Run detail →

ASUSTeK · ESC_A8A_MI325X_256GBx8

vLLM 0.6.5.dev964+mlperf50, Pytorch 2.7.0a0+git3a58512, ROCm 6.3.1

55,800

not reported

8× AMD Instinct MI325X 256GB HBM3e · mlperf_loadgen v5.1 · mlperf_closed_99 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0006Run detail →

mixtral-8x7b · Server

8× AMD Instinct MI300X 192GB HBM3 · mlperf_closed_99

Ranked only inside this comparison group · 2 feasible · TTFT p99 reported for 0 of 2

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

AMD · Supermicro AS-8125GS-TNMR2

vLLM 0.9.0.2.dev108+g71faa1880.d20250730.rocm641, Pytorch 2.7.0+gitf717b2a, ROCm 6.4.1.60401-83~22.04

47,804

not reported

8× AMD Instinct MI300X 192GB HBM3 · mlperf_loadgen v5.1 · mlperf_closed_99 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0001Run detail →

Dell · Dell Poweredge XE9680

vLLM 0.6.5.dev964+mlperf50, Pytorch 2.7.0a0+git3a58512, ROCm 6.3.1

mixtral-8x7b · Server

8× NVIDIA H200-SXM-141GB · mlperf_closed_99

Ranked only inside this comparison group · 2 feasible · TTFT p99 reported for 0 of 2

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

HPE · HPE Cray XD670 (8x H200-SXM-141GB)

TensorRT 10.11, CUDA 12.9

60,955

not reported

8× NVIDIA H200-SXM-141GB · mlperf_loadgen v5.1 · mlperf_closed_99 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0049Run detail →

ASUSTeK · ASUSTeK ESC N8 H200 (8x H200-SXM-141GB, TensorRT)

TensorRT 10.11, CUDA 12.9

59,993

mixtral-8x7b · Server

8× NVIDIA RTX PRO 6000 Blackwell Server Edition · mlperf_closed_99

Ranked only inside this comparison group · 1 feasible · TTFT p99 reported for 0 of 1

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

HPE · HPE ProLiant Compute DL380a Gen12 (8x NVIDIA RTX PRO 6000 Blackwell Server Edition, TensorRT)

TensorRT 10.11.0.33, CUDA 13.0

32,151

not reported

8× NVIDIA RTX PRO 6000 Blackwell Server Edition · mlperf_loadgen v6.0 · mlperf_closed_99 · Reported evidence · observed 2026-03-26

MLPerf Inference · entry v6.0/6.0-0046Run detail →

mixtral-8x7b · Server

8× NVIDIA RTX PRO 6000 Blackwell Server Edition · mlperf_closed_99

Ranked only inside this comparison group · 1 feasible · TTFT p99 reported for 0 of 1

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

HPE · HPE ProLiant Compute DL380a Gen12 (8x RTX Pro 6000 Blackwell Server Edition)

TensorRT 10.11.0.33, CUDA 12.9

30,134

not reported

8× NVIDIA RTX PRO 6000 Blackwell Server Edition · mlperf_loadgen v5.1 · mlperf_closed_99 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0051Run detail →

mixtral-8x7b · Server

2× NVIDIA RTX PRO 6000 Blackwell Server Edition · mlperf_closed_99

Ranked only inside this comparison group · 1 feasible · TTFT p99 reported for 0 of 1

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

HPE · HPE ProLiant DL385 Gen11 (2x NVIDIA RTX PRO 6000 Blackwell Server Edition-600W, TensorRT)

TensorRT 10.11.0.33, CUDA 13.0

7,743

not reported

2× NVIDIA RTX PRO 6000 Blackwell Server Edition · mlperf_loadgen v6.0 · mlperf_closed_99 · Reported evidence · observed 2026-03-26

MLPerf Inference · entry v6.0/6.0-0051Run detail →

mixtral-8x7b · Server

10× NVIDIA RTX PRO 6000 Blackwell Server Edition · mlperf_closed_99

Ranked only inside this comparison group · 1 feasible · TTFT p99 reported for 0 of 1

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

HPE · HPE ProLiant Compute DL380a Gen12 (10x NVIDIA RTX PRO 6000 Blackwell Server Edition, TensorRT)

TensorRT 10.11.0.33, CUDA 13.0

39,234

not reported

10× NVIDIA RTX PRO 6000 Blackwell Server Edition · mlperf_loadgen v6.0 · mlperf_closed_99 · Reported evidence · observed 2026-03-26

MLPerf Inference · entry v6.0/6.0-0044Run detail →

mixtral-8x7b · Server

4× NVIDIA RTX PRO 6000 Blackwell Server Edition · mlperf_closed_99

Ranked only inside this comparison group · 1 feasible · TTFT p99 reported for 0 of 1

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

HPE · HPE ProLiant DL385 Gen11 (4x NVIDIA RTX PRO 6000 Blackwell Server Edition-450W, TensorRT)

TensorRT 10.11.0.33, CUDA 13.0

13,846

not reported

4× NVIDIA RTX PRO 6000 Blackwell Server Edition · mlperf_loadgen v6.0 · mlperf_closed_99 · Reported evidence · observed 2026-03-26

MLPerf Inference · entry v6.0/6.0-0053Run detail →

mixtral-8x7b · Server

8× NVIDIA B200-SXM-180GB · mlperf_closed_99

Ranked only inside this comparison group · 1 feasible · TTFT p99 reported for 0 of 1

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

HPE · HPE ProLiant Compute XD685 (8x NVIDIA B200 180GB, TensorRT)

TensorRT 10.11, CUDA 13.0

127,209

not reported

8× NVIDIA B200-SXM-180GB · mlperf_loadgen v6.0 · mlperf_closed_99 · Reported evidence · observed 2026-03-26

MLPerf Inference · entry v6.0/6.0-0048Run detail →