Kernels Serving

Serving

722 results · all from MLPerf Inference, shown as published

Best reported throughput per model and scenario. Open a row for every configuration, the Pareto frontier, and exclusions. MLPerf measures token throughput only; the TTFT and TPOT limits above compare against each scenario's declared rules, never against latency KernelIndex has not seen measured.

Model

Benchmark

Best tokens/s

Stack

Hardware

Configs

Llama-2-70B (MLPerf reference)llama2-70b-99 · Interactive814,128TensorRT 10.14, CUDA 13.1, cuDNN 9.17, TensorRT-LLM feat/1.2-mlpinf, NVIDIA Dynamo mlperf-v6.0-dynamo-v0.8.0, vLLM CentM72× NVIDIA GB30042 Llama-2-70B (MLPerf reference)llama2-70b-99 · Offline1,126,850TensorRT 10.14, CUDA 13.1, cuDNN 9.17, TensorRT-LLM feat/1.2-mlpinf, NVIDIA Dynamo mlperf-v6.0-dynamo-v0.8.0, vLLM CentM72× NVIDIA GB30090 Llama-2-70B (MLPerf reference)llama2-70b-99 · Server1,016,380PyTorch 2.9.1+git8907517, ROCm 7.0.088× AMD Instinct MI355X 288GB HBM3e88 Llama-3.1-8B (MLPerf reference)llama3.1-8b · Interactive128,750TensorRT 10.14, CUDA 13.18× NVIDIA B200-SXM-180GB10 Llama-3.1-8B (MLPerf reference)llama3.1-8b · Offline166,745TensorRT 10.14, CUDA 13.18× NVIDIA B300-SXM-270GB41 Llama-3.1-8B (MLPerf reference)llama3.1-8b · Server148,067TensorRT 10.14, CUDA 13.18× NVIDIA B300-SXM-270GB41 Llama-3.1-405B (MLPerf reference)llama3.1-405b · Interactive18,365TensorRT 10.14, CUDA 13.1, cuDNN 9.17, TensorRT-LLM feat/1.2-mlpinf, NVIDIA Dynamo mlperf-v6.0-dynamo-v0.8.0, vLLM CentM72× NVIDIA GB30015 Llama-3.1-405B (MLPerf reference)llama3.1-405b · Offline19,512TensorRT 10.14, CUDA 13.1, cuDNN 9.17, TensorRT-LLM feat/1.2-mlpinf, NVIDIA Dynamo mlperf-v6.0-dynamo-v0.8.0, vLLM CentM72× NVIDIA GB30030 Llama-3.1-405B (MLPerf reference)llama3.1-405b · Server18,628TensorRT 10.14, CUDA 13.1, cuDNN 9.17, TensorRT-LLM feat/1.2-mlpinf, NVIDIA Dynamo mlperf-v6.0-dynamo-v0.8.0, vLLM CentM72× NVIDIA GB30030 GPT-OSS-120B (MLPerf reference)gpt-oss-120b · Interactive677,199TensorRT 10.14, CUDA 13.1, cuDNN 9.17, TensorRT-LLM feat/1.2-mlpinf, NVIDIA Dynamo mlperf-v6.0-dynamo-v0.8.0, vLLM CentM72× NVIDIA GB3004 GPT-OSS-120B (MLPerf reference)gpt-oss-120b · Offline1,046,150TensorRT 10.14, CUDA 13.1, cuDNN 9.17, Driver 580.105.08, TensorRT-LLM feat/1.2-mlpinf, NVIDIA Dynamo mlperf-v6.0-dynamo72× NVIDIA GB30028 GPT-OSS-120B (MLPerf reference)gpt-oss-120b · Server1,096,770TensorRT 10.14, CUDA 13.1, cuDNN 9.17, Driver 580.105.08, TensorRT-LLM feat/1.2-mlpinf, NVIDIA Dynamo mlperf-v6.0-dynamo72× NVIDIA GB30028 DeepSeek-R1 (MLPerf reference)deepseek-r1 · Interactive250,634TensorRT 10.14, CUDA 13.1, cuDNN 9.17, TensorRT-LLM feat/1.2-mlpinf, NVIDIA Dynamo mlperf-v6.0-dynamo-v0.8.0, vLLM CentM72× NVIDIA GB3003 DeepSeek-R1 (MLPerf reference)deepseek-r1 · Offline2,494,310TensorRT 10.14, CUDA 13.1, cuDNN 9.17, TensorRT-LLM feat/1.2-mlpinf, NVIDIA Dynamo mlperf-v6.0-dynamo-v0.8.0, vLLM CentM288× NVIDIA GB30026 DeepSeek-R1 (MLPerf reference)deepseek-r1 · Server1,555,110TensorRT 10.14, CUDA 13.1, cuDNN 9.17, TensorRT-LLM feat/1.2-mlpinf, NVIDIA Dynamo mlperf-v6.0-dynamo-v0.8.0, vLLM CentM288× NVIDIA GB30026 Mixtral-8x7B (MLPerf reference)mixtral-8x7b · Offline126,468TensorRT 10.11, CUDA 13.08× NVIDIA B200-SXM-180GB16 Mixtral-8x7B (MLPerf reference)mixtral-8x7b · Server127,209TensorRT 10.11, CUDA 13.08× NVIDIA B200-SXM-180GB16