Skip to content
KernelIndex
Search⌘K
Sign in

RMSNorm, Megatron layout

45 eligible runs
rmsnorm

RMSNorm over the hidden dimension of a seq×batch×hidden activation (Megatron sbh layout) with a learned hidden-size weight; the benchmark runs bf16.

Fastest reported · usable today
12.2µsmedian · 1.05× faster than baseline
LigerLiger-Kernel · BSD-2-Clause · triton

Reported evidence · last observed 2026-06-10. Reported by source; not independently reproduced.

pip install "liger-kernel==0.8.0"
View source →Run detail →

Fastest by languagetriton · Liger · 12.2 µs · #1python · PyTorch · 12.7 µs · 1.05×

Current records

Workloadseq = 4096 · batch = 1 · hidden = 1024 · bf16seq = 4096 · batch = 1 · hidden = 2048 · bf16seq = 4096 · batch = 1 · hidden = 4096 · bf16seq = 4096 · batch = 1 · hidden = 8192 · bf16seq = 4096 · batch = 1 · hidden = 16384 · bf16
HardwareBest knownImplementationRuns
NVIDIA H100 · env 112.2 µsLiger3NVIDIA H100 · env 274.4 µsPyTorch3NVIDIA H100 · env 374.6 µsPyTorch3

Not measured on B200 for this workload. Challenges →

Source-native comparison · GPU NVIDIA H100 · Workload seq = 4096 · batch = 1 · hidden = 1024 · bf16 · Protocol Liger-Kernel benchmark scripts · median · 3 results · last observed 2026-06-10Record history →
Estimated floor 2.50 µs · record 4.85× above itestimate, not evidence ›
DRAM 2.50 µs · bandwidth-bound on H100 SXM
every declared tensor crosses HBM exactly once (3,350 GB/s, H100 SXM datasheet)
no arithmetic formula for this family: bandwidth floor only
headroom-v1: a lower bound from declared tensors and datasheet peaks. A kernel can sit well above it for good reasons.
#
Implementation
Latency
vs #1
Trust
Observed
1
LigerLiger-Kernel
12.2µs
1.00×
Reported · BSD-2-Clause · source
2026-06-10

1.05× faster than the baseline. Measured exactly what you asked. Reported by source; not independently reproduced.

source mirroredBSD-2-ClauseinstallableView source →Run detail →
2
PyTorchbaseline
12.7µs
1.05×
Reported · BSD-3-Clause · source
2026-06-10

Measured exactly what you asked. The source's designated baseline implementation. Reported by source; not independently reproduced.

source mirroredBSD-3-Clauseno install recipeView source →Run detail →
3
PyTorchbaseline
12.8µs
1.05×
Reported · BSD-3-Clause · source
2026-06-10

Measured exactly what you asked. The source's designated baseline implementation. Reported by source; not independently reproduced.

source mirroredBSD-3-Clauseno install recipeView source →Run detail →

Scaling by hidden

50.0 µs100.0 µs150.0 µs1024204840968k16khidden →12.2 µs17.6 µs29.2 µs50.9 µs95.0 µsLiger12.7 µs19.0 µs32.5 µs74.5 µs149.2 µsPyTorch
LigerPyTorchbest per workload · NVIDIA H100 · Liger-Kernel benchmark scripts held constant

Implementations

Implementation
Runtime
Best latency
Evidence
Availability
PyTorch
python
12.7µs
1.05×
Reported
BSD-3-Clause · source
LigerLiger-Kernel
triton
12.2µs
1.00×
Reported
BSD-2-Clause · source

Semantics

Inputs and outputs
xfloat [seq, batch, hidden]
weightfloat [hidden]
yfloat [seq, batch, hidden]
Axes and behavior
seqvariable
batchvariable
hiddenvariable
determinismunspecified
constraintsNo mutation or aliasing
Identity
sha2562719cde9f651…
Sources: Liger-Kernel benchmarks (2026-06-10) · BSD-2-Clauselast observed 2026-06-10How records are decidedJSON