Skip to content
KernelIndex
Search⌘K
Sign in

Fused add + RMSNorm

54 eligible runs
rmsnorm

Residual add followed by RMSNorm, returning the normalized activation and the post-add residual.

Fastest reported · usable today
20.5µsmedian · 2.22× faster than baseline
Liger fusedLiger-Kernel · BSD-2-Clause · triton

Reported evidence · last observed 2026-04-07. Reported by source; not independently reproduced.

pip install "liger-kernel==0.7.0"
View source →Run detail →

Fastest by languagetriton · Liger fused · 20.5 µs · #1python · Transformers · 45.5 µs · 2.22×

Current records

Workloadrows = 2048 · hidden = 1024 · fp32rows = 2048 · hidden = 8192 · fp32rows = 2048 · hidden = 16384 · fp32rows = 2048 · hidden = 32768 · fp32rows = 2048 · hidden = 2048 · fp32rows = 2048 · hidden = 4096 · fp32
HardwareBest knownImplementationRuns
NVIDIA H100 · env 120.5 µsLiger fused3NVIDIA H100 · env 2113.6 µsTransformers3NVIDIA H100 · env 3229.6 µsTransformers3

Not measured on B200 for this workload. Challenges →

Source-native comparison · GPU NVIDIA H100 · Workload rows = 2048 · hidden = 1024 · fp32 · Protocol Liger-Kernel benchmark scripts · median · 3 results · last observed 2026-04-07Record history →
Estimated floor 5.01 µs · record 4.1× above itestimate, not evidence ›
DRAM 5.01 µs · bandwidth-bound on H100 SXM
every declared tensor crosses HBM exactly once (3,350 GB/s, H100 SXM datasheet)
no arithmetic formula for this family: bandwidth floor only
headroom-v1: a lower bound from declared tensors and datasheet peaks. A kernel can sit well above it for good reasons.
#
Implementation
Latency
vs #1
Trust
Observed
1
Liger fusedLiger-Kernel
20.5µs
1.00×
Reported · BSD-2-Clause · source
2026-04-07

2.22× faster than the baseline. Measured exactly what you asked. Reported by source; not independently reproduced.

source mirroredBSD-2-ClauseinstallableView source →Run detail →
2
TransformersbaselineHugging Face Transformers
45.5µs
2.22×
Reported · Apache-2.0 · source
2026-04-07

Measured exactly what you asked. The source's designated baseline implementation. Reported by source; not independently reproduced.

source mirroredApache-2.0no install recipeView source →Run detail →
3
Liger fusedLiger-Kernel
46.1µs
2.24×
Reported · BSD-2-Clause · source
2026-04-07

1.01× slower than the baseline. Measured exactly what you asked. Reported by source; not independently reproduced.

source mirroredBSD-2-ClauseinstallableView source →Run detail →

Scaling by hidden

100.0 µs1.00 ms1024204840968k16k32khidden →20.5 µs28.2 µs49.5 µs94.6 µs183.8 µs433.8 µsLiger fused45.5 µs72.2 µs149.1 µs271.1 µs509.2 µs982.8 µsTransformers
Liger fusedTransformersbest per workload · NVIDIA H100 · Liger-Kernel benchmark scripts held constant · log scale

Implementations

Implementation
Runtime
Best latency
Evidence
Availability
TransformersHugging Face Transformers
python
45.5µs
2.22×
Reported
Apache-2.0 · source
Liger fusedLiger-Kernel
triton
20.5µs
1.00×
Reported
BSD-2-Clause · source

Semantics

Inputs and outputs
xfloat [rows, hidden]
residualfloat [rows, hidden]
weightfloat [hidden]
yfloat [rows, hidden]
residual_outfloat [rows, hidden]
Axes and behavior
rowsvariable
hiddenvariable
determinismunspecified
constraintsNo mutation or aliasing
Identity
sha256770a9f88bcb4…
Sources: Liger-Kernel benchmarks (2026-04-07) · BSD-2-Clauselast observed 2026-04-07How records are decidedJSON