Serving · KernelIndex

28 comparison groups · 48 feasible · 0 excluded

Ranked only within one model, workload, protocol, hardware, and quality bar; a run missing a bounded metric is left out, not guessed.

Llama-2-70B (MLPerf reference) · llama2-70b-99 · Server

8× AMD Instinct MI300X 192GB HBM3 · mlperf_closed_99

Ranked only inside this comparison group · 4 feasible · TTFT p99 reported for 0 of 4

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

MangoBoost · Supermicro AS-8125GS-TNMR2 (8x MI300X-192GB MangoBoost-LLMBoost)

Mango LLMBoost AI Enterprise Platform, vllm-0.9.0.2.dev108+g71faa1880.d20250717.rocm641, ROCm 6.12.12

24,861

not reported

8× AMD Instinct MI300X 192GB HBM3 · mlperf_loadgen v5.1 · mlperf_closed_99 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0066Run detail →

Dell · Dell Poweredge XE9680

vLLM 0.6.5.dev964+mlperf50, Pytorch 2.7.0a0+git3a58512, ROCm 6.3.1

24,748

not reported

8× AMD Instinct MI300X 192GB HBM3 · mlperf_loadgen v5.1 · mlperf_closed_99 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0027Run detail →

AMD · Supermicro AS-8125GS-TNMR2

vLLM 0.9.0.2.dev108+g71faa1880.d20250730.rocm641, Pytorch 2.7.0+gitf717b2a, ROCm 6.4.1.60401-83~22.04

24,594

not reported

8× AMD Instinct MI300X 192GB HBM3 · mlperf_loadgen v5.1 · mlperf_closed_99 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0001Run detail →

Dell_MangoBoost · Dell PowerEdge XE9680 (8x MI300X-192GB MangoBoost-LLMBoost)

Mango LLMBoost AI Enterprise Platform, vllm-0.9.0.2.dev108+g71faa1880.d20250717.rocm641, ROCm 6.12.12

24,533

not reported

8× AMD Instinct MI300X 192GB HBM3 · mlperf_loadgen v5.1 · mlperf_closed_99 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0031Run detail →

Llama-2-70B (MLPerf reference) · llama2-70b-99 · Offline

8× AMD Instinct MI300X 192GB HBM3 · mlperf_closed_99_9

Ranked only inside this comparison group · 4 feasible · TTFT p99 reported for 0 of 4

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

MangoBoost · Supermicro AS-8125GS-TNMR2 (8x MI300X-192GB MangoBoost-LLMBoost)

Mango LLMBoost AI Enterprise Platform, vllm-0.9.0.2.dev108+g71faa1880.d20250717.rocm641, ROCm 6.12.12

27,854

not reported

8× AMD Instinct MI300X 192GB HBM3 · mlperf_loadgen v5.1 · mlperf_closed_99_9 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0066Run detail →

AMD · Supermicro AS-8125GS-TNMR2

Llama-2-70B (MLPerf reference) · llama2-70b-99 · Offline

8× AMD Instinct MI300X 192GB HBM3 · mlperf_closed_99

Ranked only inside this comparison group · 4 feasible · TTFT p99 reported for 0 of 4

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

MangoBoost · Supermicro AS-8125GS-TNMR2 (8x MI300X-192GB MangoBoost-LLMBoost)

Mango LLMBoost AI Enterprise Platform, vllm-0.9.0.2.dev108+g71faa1880.d20250717.rocm641, ROCm 6.12.12

27,854

not reported

8× AMD Instinct MI300X 192GB HBM3 · mlperf_loadgen v5.1 · mlperf_closed_99 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0066Run detail →

AMD · Supermicro AS-8125GS-TNMR2

Llama-2-70B (MLPerf reference) · llama2-70b-99 · Server

8× AMD Instinct MI300X 192GB HBM3 · mlperf_closed_99_9

Ranked only inside this comparison group · 4 feasible · TTFT p99 reported for 0 of 4

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

MangoBoost · Supermicro AS-8125GS-TNMR2 (8x MI300X-192GB MangoBoost-LLMBoost)

Mango LLMBoost AI Enterprise Platform, vllm-0.9.0.2.dev108+g71faa1880.d20250717.rocm641, ROCm 6.12.12

24,861

not reported

8× AMD Instinct MI300X 192GB HBM3 · mlperf_loadgen v5.1 · mlperf_closed_99_9 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0066Run detail →

Dell · Dell Poweredge XE9680

Llama-2-70B (MLPerf reference) · llama2-70b-99 · Interactive

8× AMD Instinct MI300X 192GB HBM3 · mlperf_closed_99

Ranked only inside this comparison group · 2 feasible · TTFT p99 reported for 0 of 2

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

AMD · Supermicro AS-8125GS-TNMR2

vLLM 0.9.0.2.dev108+g71faa1880.d20250730.rocm641, Pytorch 2.7.0+gitf717b2a, ROCm 6.4.1.60401-83~22.04

8,840

not reported

8× AMD Instinct MI300X 192GB HBM3 · mlperf_loadgen v5.1 · mlperf_closed_99 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0001Run detail →

Dell · Dell Poweredge XE9680

vLLM 0.6.5.dev964+mlperf50, Pytorch 2.7.0a0+git3a58512, ROCm 6.3.1

Mixtral-8x7B (MLPerf reference) · mixtral-8x7b · Server

8× AMD Instinct MI300X 192GB HBM3 · mlperf_closed_99

Ranked only inside this comparison group · 2 feasible · TTFT p99 reported for 0 of 2

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

AMD · Supermicro AS-8125GS-TNMR2

vLLM 0.9.0.2.dev108+g71faa1880.d20250730.rocm641, Pytorch 2.7.0+gitf717b2a, ROCm 6.4.1.60401-83~22.04

47,804

not reported

8× AMD Instinct MI300X 192GB HBM3 · mlperf_loadgen v5.1 · mlperf_closed_99 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0001Run detail →

Dell · Dell Poweredge XE9680

vLLM 0.6.5.dev964+mlperf50, Pytorch 2.7.0a0+git3a58512, ROCm 6.3.1

Mixtral-8x7B (MLPerf reference) · mixtral-8x7b · Offline

8× AMD Instinct MI300X 192GB HBM3 · mlperf_closed_99

Ranked only inside this comparison group · 2 feasible · TTFT p99 reported for 0 of 2

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

Dell · Dell Poweredge XE9680

vLLM 0.6.5.dev964+mlperf50, Pytorch 2.7.0a0+git3a58512, ROCm 6.3.1

53,334

not reported

8× AMD Instinct MI300X 192GB HBM3 · mlperf_loadgen v5.1 · mlperf_closed_99 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0027Run detail →

AMD · Supermicro AS-8125GS-TNMR2

vLLM 0.9.0.2.dev108+g71faa1880.d20250730.rocm641, Pytorch 2.7.0+gitf717b2a, ROCm 6.4.1.60401-83~22.04

Llama-2-70B (MLPerf reference) · llama2-70b-99 · Server

8× AMD Instinct MI300X 192GB HBM3 · mlperf_closed_99

Ranked only inside this comparison group · 2 feasible · TTFT p99 reported for 0 of 2

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

MangoBoost · MangoBoost Mi300X (8x MI300X, LLMBoost)

ROCm 6.10.5

25,463

not reported

8× AMD Instinct MI300X 192GB HBM3 · mlperf_loadgen v6.0 · mlperf_closed_99 · Reported evidence · observed 2026-03-26

MLPerf Inference · entry v6.0/6.0-0069Run detail →

Dell · PowerEdge XE9680 (8x MI300X)

vLLM 0.9.0.2.dev108+g71faa1880.d20260213, Pytorch 2.7.0+gitf717b2a, ROCm 6.4.1

Llama-2-70B (MLPerf reference) · llama2-70b-99 · Server

8× AMD Instinct MI300X 192GB HBM3 · mlperf_closed_99_9

Ranked only inside this comparison group · 2 feasible · TTFT p99 reported for 0 of 2

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

MangoBoost · MangoBoost Mi300X (8x MI300X, LLMBoost)

ROCm 6.10.5

25,463

not reported

8× AMD Instinct MI300X 192GB HBM3 · mlperf_loadgen v6.0 · mlperf_closed_99_9 · Reported evidence · observed 2026-03-26

MLPerf Inference · entry v6.0/6.0-0069Run detail →

Dell · PowerEdge XE9680 (8x MI300X)

vLLM 0.9.0.2.dev108+g71faa1880.d20260213, Pytorch 2.7.0+gitf717b2a, ROCm 6.4.1

Llama-2-70B (MLPerf reference) · llama2-70b-99 · Interactive

8× AMD Instinct MI300X 192GB HBM3 · mlperf_closed_99_9

Ranked only inside this comparison group · 2 feasible · TTFT p99 reported for 0 of 2

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

AMD · Supermicro AS-8125GS-TNMR2

vLLM 0.9.0.2.dev108+g71faa1880.d20250730.rocm641, Pytorch 2.7.0+gitf717b2a, ROCm 6.4.1.60401-83~22.04

8,840

not reported

8× AMD Instinct MI300X 192GB HBM3 · mlperf_loadgen v5.1 · mlperf_closed_99_9 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0001Run detail →

Dell · Dell Poweredge XE9680

vLLM 0.6.5.dev964+mlperf50, Pytorch 2.7.0a0+git3a58512, ROCm 6.3.1

Llama-2-70B (MLPerf reference) · llama2-70b-99 · Offline

8× AMD Instinct MI300X 192GB HBM3 · mlperf_closed_99_9

Ranked only inside this comparison group · 2 feasible · TTFT p99 reported for 0 of 2

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

MangoBoost · MangoBoost Mi300X (8x MI300X, LLMBoost)

ROCm 6.10.5

27,599

not reported

8× AMD Instinct MI300X 192GB HBM3 · mlperf_loadgen v6.0 · mlperf_closed_99_9 · Reported evidence · observed 2026-03-26

MLPerf Inference · entry v6.0/6.0-0069Run detail →

Dell · PowerEdge XE9680 (8x MI300X)

vLLM 0.9.0.2.dev108+g71faa1880.d20260213, Pytorch 2.7.0+gitf717b2a, ROCm 6.4.1

Llama-2-70B (MLPerf reference) · llama2-70b-99 · Offline

8× AMD Instinct MI300X 192GB HBM3 · mlperf_closed_99

Ranked only inside this comparison group · 2 feasible · TTFT p99 reported for 0 of 2

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

MangoBoost · MangoBoost Mi300X (8x MI300X, LLMBoost)

ROCm 6.10.5

27,599

not reported

8× AMD Instinct MI300X 192GB HBM3 · mlperf_loadgen v6.0 · mlperf_closed_99 · Reported evidence · observed 2026-03-26

MLPerf Inference · entry v6.0/6.0-0069Run detail →

Dell · PowerEdge XE9680 (8x MI300X)

vLLM 0.9.0.2.dev108+g71faa1880.d20260213, Pytorch 2.7.0+gitf717b2a, ROCm 6.4.1

16 more comparison groups. Narrow by model, workload, or hardware to see them.