Skip to content

Search Models Records GPUs Docs

More

Records GPUs Docs

Kernels Serving

Serving

722 results · all from MLPerf Inference, shown as published

Scope

Model

Workload

Hardware

Max GPUs

Objective

Rank by

Scenario limits

TTFT p99 ≤ms

TPOT p99 ≤ms

Feed Serving Projects Challenges Contribute Sources Methodology Legal GitHub Atom llms.txta46b010

2 comparison groups · 2 feasible · 0 excluded

Ranked only within one model, workload, protocol, hardware, and quality bar; a run missing a bounded metric is left out, not guessed.

Llama-3.1-8B (MLPerf reference) · llama3.1-8b · Offline

1× NVIDIA L40S · mlperf_closed_99

Ranked only inside this comparison group · 1 feasible · TTFT p99 reported for 0 of 1

#

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

1

RedHat · IBM Cloud gx3 instance 1xL40S-PCIE-48GB ,vLLM

vLLM 0.10.0, RHEL 9.6

1,642

not reported

not reported

1

›

1× NVIDIA L40S · mlperf_loadgen v5.1 · mlperf_closed_99 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0082Run detail →

Llama-3.1-8B (MLPerf reference) · llama3.1-8b · Server

1× NVIDIA L40S · mlperf_closed_99

Ranked only inside this comparison group · 1 feasible · TTFT p99 reported for 0 of 1

#

Configuration

Stack

tokens/s

TTFT p99

TPOT p99

GPUs

1

RedHat · IBM Cloud gx3 instance 1xL40S-PCIE-48GB ,vLLM

vLLM 0.10.0, RHEL 9.6

1,207

not reported

not reported

1

›

1× NVIDIA L40S · mlperf_loadgen v5.1 · mlperf_closed_99 · Reported evidence · observed 2025-09-04

MLPerf Inference · entry v5.1/5.1-0082Run detail →