Serving · KernelIndex

14 comparison groups · 14 feasible · 0 excluded

Ranked only within one model, workload, protocol, hardware, and quality bar; a run missing a bounded metric is left out, not guessed.

Llama-2-70B (MLPerf reference) · llama2-70b-99 · Offline

8× NVIDIA H100-SXM-80GB × 2 nodes · mlperf_closed_99_9

Ranked only inside this comparison group · 1 feasible · TTFT p99 reported for 0 of 1

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

Cisco · HPF HGX System (16xH100-SXM-80GB, TensorRT)

TensorRT 10.8, CUDA 12.8

62,352

not reported

8× NVIDIA H100-SXM-80GB × 2 nodes · mlperf_loadgen v5.1 · mlperf_closed_99_9 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0015Run detail →

Llama-2-70B (MLPerf reference) · llama2-70b-99 · Offline

8× NVIDIA H100-SXM-80GB × 4 nodes · mlperf_closed_99_9

Ranked only inside this comparison group · 1 feasible · TTFT p99 reported for 0 of 1

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

Cisco · HPF HGX System (32xH100-SXM-80GB, TensorRT)

TensorRT 10.8, CUDA 12.8

124,879

not reported

8× NVIDIA H100-SXM-80GB × 4 nodes · mlperf_loadgen v5.1 · mlperf_closed_99_9 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0016Run detail →

Llama-3.1-405B (MLPerf reference) · llama3.1-405b · Server

8× NVIDIA H100-SXM-80GB × 4 nodes · mlperf_closed_99

Ranked only inside this comparison group · 1 feasible · TTFT p99 reported for 0 of 1

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

Cisco · HPF HGX System (32xH100-SXM-80GB, TensorRT)

TensorRT 10.8, CUDA 12.8

1,127

not reported

8× NVIDIA H100-SXM-80GB × 4 nodes · mlperf_loadgen v5.1 · mlperf_closed_99 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0016Run detail →

Llama-2-70B (MLPerf reference) · llama2-70b-99 · Offline

8× NVIDIA H100-SXM-80GB × 2 nodes · mlperf_closed_99

Ranked only inside this comparison group · 1 feasible · TTFT p99 reported for 0 of 1

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

Cisco · HPF HGX System (16xH100-SXM-80GB, TensorRT)

TensorRT 10.8, CUDA 12.8

62,352

not reported

8× NVIDIA H100-SXM-80GB × 2 nodes · mlperf_loadgen v5.1 · mlperf_closed_99 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0015Run detail →

Llama-2-70B (MLPerf reference) · llama2-70b-99 · Server

8× NVIDIA H100-SXM-80GB × 4 nodes · mlperf_closed_99_9

Ranked only inside this comparison group · 1 feasible · TTFT p99 reported for 0 of 1

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

Cisco · HPF HGX System (32xH100-SXM-80GB, TensorRT)

TensorRT 10.8, CUDA 12.8

122,274

not reported

8× NVIDIA H100-SXM-80GB × 4 nodes · mlperf_loadgen v5.1 · mlperf_closed_99_9 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0016Run detail →

Llama-2-70B (MLPerf reference) · llama2-70b-99 · Server

8× NVIDIA H100-SXM-80GB × 2 nodes · mlperf_closed_99

Ranked only inside this comparison group · 1 feasible · TTFT p99 reported for 0 of 1

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

Cisco · HPF HGX System (16xH100-SXM-80GB, TensorRT)

TensorRT 10.8, CUDA 12.8

61,101

not reported

8× NVIDIA H100-SXM-80GB × 2 nodes · mlperf_loadgen v5.1 · mlperf_closed_99 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0015Run detail →

Llama-3.1-405B (MLPerf reference) · llama3.1-405b · Offline

8× NVIDIA H100-SXM-80GB × 2 nodes · mlperf_closed_99

Ranked only inside this comparison group · 1 feasible · TTFT p99 reported for 0 of 1

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

Cisco · HPF HGX System (16xH100-SXM-80GB, TensorRT)

TensorRT 10.8, CUDA 12.8

783

not reported

8× NVIDIA H100-SXM-80GB × 2 nodes · mlperf_loadgen v5.1 · mlperf_closed_99 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0015Run detail →

Llama-3.1-8B (MLPerf reference) · llama3.1-8b · Server

1× NVIDIA H100-SXM-80GB · mlperf_closed_99

Ranked only inside this comparison group · 1 feasible · TTFT p99 reported for 0 of 1

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

RedHat · Dell PowerEdge XE8640 (1xH100-SXM-80GB, vLLM)

vLLM 0.10.0, RHEL 9.6

5,104

not reported

1× NVIDIA H100-SXM-80GB · mlperf_loadgen v5.1 · mlperf_closed_99 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0083Run detail →

Llama-3.1-8B (MLPerf reference) · llama3.1-8b · Offline

1× NVIDIA H100-SXM-80GB · mlperf_closed_99

Ranked only inside this comparison group · 1 feasible · TTFT p99 reported for 0 of 1

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

RedHat · Dell PowerEdge XE8640 (1xH100-SXM-80GB, vLLM)

vLLM 0.10.0, RHEL 9.6

5,777

not reported

1× NVIDIA H100-SXM-80GB · mlperf_loadgen v5.1 · mlperf_closed_99 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0083Run detail →

Llama-3.1-405B (MLPerf reference) · llama3.1-405b · Offline

8× NVIDIA H100-SXM-80GB × 4 nodes · mlperf_closed_99

Ranked only inside this comparison group · 1 feasible · TTFT p99 reported for 0 of 1

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

Cisco · HPF HGX System (32xH100-SXM-80GB, TensorRT)

TensorRT 10.8, CUDA 12.8

1,562

not reported

8× NVIDIA H100-SXM-80GB × 4 nodes · mlperf_loadgen v5.1 · mlperf_closed_99 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0016Run detail →

Llama-2-70B (MLPerf reference) · llama2-70b-99 · Server

8× NVIDIA H100-SXM-80GB × 4 nodes · mlperf_closed_99

Ranked only inside this comparison group · 1 feasible · TTFT p99 reported for 0 of 1

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

Cisco · HPF HGX System (32xH100-SXM-80GB, TensorRT)

TensorRT 10.8, CUDA 12.8

122,274

not reported

8× NVIDIA H100-SXM-80GB × 4 nodes · mlperf_loadgen v5.1 · mlperf_closed_99 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0016Run detail →

Llama-2-70B (MLPerf reference) · llama2-70b-99 · Offline

8× NVIDIA H100-SXM-80GB × 4 nodes · mlperf_closed_99

Ranked only inside this comparison group · 1 feasible · TTFT p99 reported for 0 of 1

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

Cisco · HPF HGX System (32xH100-SXM-80GB, TensorRT)

TensorRT 10.8, CUDA 12.8

124,879

not reported

8× NVIDIA H100-SXM-80GB × 4 nodes · mlperf_loadgen v5.1 · mlperf_closed_99 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0016Run detail →

2 more comparison groups. Narrow by model, workload, or hardware to see them.