Kernels Serving

Serving

722 results · all from MLPerf Inference, shown as published

32 comparison groups · 78 feasible · 0 excluded

Ranked only within one model, workload, protocol, hardware, and quality bar; a run missing a bounded metric is left out, not guessed.

llama2-70b-99 · Interactive

8× NVIDIA B200-SXM-180GB · mlperf_closed_99_9

Ranked only inside this comparison group · 8 feasible · TTFT p99 reported for 0 of 8

#

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

1

GigaComputing · G894-SD1

TensorRT 10.11, CUDA 12.9

62,851

not reported

not reported

8

8× NVIDIA B200-SXM-180GB · mlperf_loadgen v5.1 · mlperf_closed_99_9 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0046Run detail →

2

Supermicro · SYS-422GA-NBRT-LCC (8x B200-SXM-180GB, TensorRT)

TensorRT 10.11, CUDA 12.9

60,140

not reported

not reported

8

8× NVIDIA B200-SXM-180GB · mlperf_loadgen v5.1 · mlperf_closed_99_9 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0087Run detail →

3

Dell · Dell PowerEdge XE9680L (8x B200-SXM-180GB, TensorRT)

TensorRT 10.11, CUDA 12.9

60,138

not reported

not reported

8

8× NVIDIA B200-SXM-180GB · mlperf_loadgen v5.1 · mlperf_closed_99_9 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0024Run detail →

4

Supermicro · SYS-422GS-NBRT-LCC (8x B200-SXM-180GB, TensorRT)

TensorRT 10.11, CUDA 12.9

60,120

not reported

not reported

8

8× NVIDIA B200-SXM-180GB · mlperf_loadgen v5.1 · mlperf_closed_99_9 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0088Run detail →

5

Nebius · Nebius B200 (8x B200-SXM-180GB, TensorRT)

TensorRT 10.11, CUDA 12.9

59,623

not reported

not reported

8

8× NVIDIA B200-SXM-180GB · mlperf_loadgen v5.1 · mlperf_closed_99_9 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0073Run detail →

6

NVIDIA · NVIDIA DGX B200 (8x B200-SXM-180GB, TensorRT)

TensorRT 10.11, CUDA 12.9

59,545

not reported

not reported

8

8× NVIDIA B200-SXM-180GB · mlperf_loadgen v5.1 · mlperf_closed_99_9 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0069Run detail →

7

Supermicro · SYS-A21GE-NBRT (8x B200-SXM-180GB, TensorRT)

TensorRT 10.11, CUDA 12.9

57,418

not reported

not reported

8

8× NVIDIA B200-SXM-180GB · mlperf_loadgen v5.1 · mlperf_closed_99_9 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0089Run detail →

8

Google · NVIDIA DGX B200 (8x B200-SXM-180GB, TensorRT)

TensorRT 10.11, CUDA 12.9

52,141

not reported

not reported

8

8× NVIDIA B200-SXM-180GB · mlperf_loadgen v5.1 · mlperf_closed_99_9 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0047Run detail →

llama2-70b-99 · Interactive

8× NVIDIA B200-SXM-180GB · mlperf_closed_99

Ranked only inside this comparison group · 6 feasible · TTFT p99 reported for 0 of 6

#

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

1

GigaComputing · G894-SD1

TensorRT 10.11, CUDA 12.9

62,851

not reported

not reported

8

8× NVIDIA B200-SXM-180GB · mlperf_loadgen v5.1 · mlperf_closed_99 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0046Run detail →

2

Dell · Dell PowerEdge XE9680L (8x B200-SXM-180GB, TensorRT)

TensorRT 10.11, CUDA 12.9

60,138

llama2-70b-99 · Interactive

8× AMD Instinct MI355X 288GB HBM3e · mlperf_closed_99

Ranked only inside this comparison group · 6 feasible · TTFT p99 reported for 0 of 6

#

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

1

AMD · smci355-ccs-aus-m09-17 (AS -4126GS-NMR-LCC)

PyTorch 2.9.1+git8907517, ROCm 7.0.0

73,608

not reported

not reported

8

8× AMD Instinct MI355X 288GB HBM3e · mlperf_loadgen v6.0 · mlperf_closed_99 · Reported evidence · observed 2026-03-26

MLPerf Inference · entry v6.0/6.0-0002Run detail →

2

GigaComputing · G893-ZX1-AAX4

PyTorch 2.9.1+git8907517, ROCm 7.0.0

72,547

llama2-70b-99 · Interactive

8× AMD Instinct MI355X 288GB HBM3e · mlperf_closed_99_9

Ranked only inside this comparison group · 6 feasible · TTFT p99 reported for 0 of 6

#

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

1

AMD · smci355-ccs-aus-m09-17 (AS -4126GS-NMR-LCC)

PyTorch 2.9.1+git8907517, ROCm 7.0.0

73,608

not reported

not reported

8

8× AMD Instinct MI355X 288GB HBM3e · mlperf_loadgen v6.0 · mlperf_closed_99_9 · Reported evidence · observed 2026-03-26

MLPerf Inference · entry v6.0/6.0-0002Run detail →

2

GigaComputing · G893-ZX1-AAX4

PyTorch 2.9.1+git8907517, ROCm 7.0.0

72,547

llama2-70b-99 · Interactive

8× AMD Instinct MI325X 256GB HBM3e · mlperf_closed_99_9

Ranked only inside this comparison group · 6 feasible · TTFT p99 reported for 0 of 6

#

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

1

MiTAC · G8825Z5

vLLM 0.6.5.dev964+mlperf50, Pytorch 2.7.0a0+git3a58512, ROCm 6.3.1

18,846

not reported

not reported

8

8× AMD Instinct MI325X 256GB HBM3e · mlperf_loadgen v5.1 · mlperf_closed_99_9 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0068Run detail →

2

GigaComputing · G893-ZX1-AAX2

vLLM 0.6.5.dev964+mlperf50, Pytorch 2.7.0a0+git3a58512, ROCm 6.3.1

llama2-70b-99 · Interactive

8× AMD Instinct MI325X 256GB HBM3e · mlperf_closed_99

Ranked only inside this comparison group · 6 feasible · TTFT p99 reported for 0 of 6

#

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

1

MiTAC · G8825Z5

vLLM 0.6.5.dev964+mlperf50, Pytorch 2.7.0a0+git3a58512, ROCm 6.3.1

18,846

not reported

not reported

8

8× AMD Instinct MI325X 256GB HBM3e · mlperf_loadgen v5.1 · mlperf_closed_99 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0068Run detail →

2

GigaComputing · G893-ZX1-AAX2

vLLM 0.6.5.dev964+mlperf50, Pytorch 2.7.0a0+git3a58512, ROCm 6.3.1

llama2-70b-99 · Interactive

8× NVIDIA H200-SXM-141GB · mlperf_closed_99

Ranked only inside this comparison group · 5 feasible · TTFT p99 reported for 0 of 5

#

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

1

Nebius · Nebius H200 (8x H200-SXM-141GB, TensorRT)

TensorRT 10.11, CUDA 12.9

23,080

not reported

not reported

8

8× NVIDIA H200-SXM-141GB · mlperf_loadgen v5.1 · mlperf_closed_99 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0075Run detail →

2

Dell · Dell PowerEdge XE9680 (8xH200-SXM-141GB, TensorRT)

TensorRT 10.11, CUDA 12.8

21,916

llama2-70b-99 · Interactive

8× NVIDIA H200-SXM-141GB · mlperf_closed_99_9

Ranked only inside this comparison group · 5 feasible · TTFT p99 reported for 0 of 5

#

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

1

Nebius · Nebius H200 (8x H200-SXM-141GB, TensorRT)

TensorRT 10.11, CUDA 12.9

23,080

not reported

not reported

8

8× NVIDIA H200-SXM-141GB · mlperf_loadgen v5.1 · mlperf_closed_99_9 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0075Run detail →

2

Dell · Dell PowerEdge XE9680 (8xH200-SXM-141GB, TensorRT)

TensorRT 10.11, CUDA 12.8

21,916

llama2-70b-99 · Interactive

8× NVIDIA B300-SXM-270GB · mlperf_closed_99

Ranked only inside this comparison group · 2 feasible · TTFT p99 reported for 0 of 2

#

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

1

Dell · PowerEdge XE9780L (8x B300-SXM-270GB, TensorRT)

TensorRT 10.14, CUDA 13.1

70,724

not reported

not reported

8

8× NVIDIA B300-SXM-270GB · mlperf_loadgen v6.0 · mlperf_closed_99 · Reported evidence · observed 2026-03-26

MLPerf Inference · entry v6.0/6.0-0024Run detail →

2

GigaComputing · G894-SD3-AAX7

TensorRT 10.14, CUDA 13.1

65,465

llama2-70b-99 · Interactive

8× AMD Instinct MI300X 192GB HBM3 · mlperf_closed_99

Ranked only inside this comparison group · 2 feasible · TTFT p99 reported for 0 of 2

#

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

1

AMD · Supermicro AS-8125GS-TNMR2

vLLM 0.9.0.2.dev108+g71faa1880.d20250730.rocm641, Pytorch 2.7.0+gitf717b2a, ROCm 6.4.1.60401-83~22.04

8,840

not reported

not reported

8

8× AMD Instinct MI300X 192GB HBM3 · mlperf_loadgen v5.1 · mlperf_closed_99 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0001Run detail →

2

Dell · Dell Poweredge XE9680

vLLM 0.6.5.dev964+mlperf50, Pytorch 2.7.0a0+git3a58512, ROCm 6.3.1

llama2-70b-99 · Interactive

8× NVIDIA B300-SXM-270GB · mlperf_closed_99_9

Ranked only inside this comparison group · 2 feasible · TTFT p99 reported for 0 of 2

#

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

1

Dell · PowerEdge XE9780L (8x B300-SXM-270GB, TensorRT)

TensorRT 10.14, CUDA 13.1

70,724

not reported

not reported

8

8× NVIDIA B300-SXM-270GB · mlperf_loadgen v6.0 · mlperf_closed_99_9 · Reported evidence · observed 2026-03-26

MLPerf Inference · entry v6.0/6.0-0024Run detail →

2

GigaComputing · G894-SD3-AAX7

TensorRT 10.14, CUDA 13.1

65,465

llama2-70b-99 · Interactive

8× AMD Instinct MI350X 288GB HBM3e · mlperf_closed_99

Ranked only inside this comparison group · 2 feasible · TTFT p99 reported for 0 of 2

#

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

1

MiTAC · G8825Z5

Pytorch 2.9.1 , ROCm 7.0

54,731

not reported

not reported

8

8× AMD Instinct MI350X 288GB HBM3e · mlperf_loadgen v6.0 · mlperf_closed_99 · Reported evidence · observed 2026-03-26

MLPerf Inference · entry v6.0/6.0-0071Run detail →

2

Cisco · Cisco UCS 8xMI350X

vLLM

51,617

20 more comparison groups. Narrow by model, workload, or hardware to see them.