LayerNorm
50 eligible runs
layernorm
Layer normalization of a rows×hidden activation with learned weight and bias.
Fastest reported · usable today
18.8µsmedian · 1.25× faster than baseline
LigerLiger-Kernel · BSD-2-Clause · triton
Reported evidence · last observed 2025-07-17. Reported by source; not independently reproduced.
pip install "liger-kernel==0.6.0"Fastest by languagetriton · Liger · 18.8 µs · #1python · Transformers · 23.6 µs · 1.25×
Current records
Workloadrows = 4096 · hidden = 1024 · fp32rows = 4096 · hidden = 2048 · fp32rows = 4096 · hidden = 4096 · fp32rows = 4096 · hidden = 16384 · fp32rows = 4096 · hidden = 8192 · fp32
HardwareBest knownImplementationRuns
NVIDIA H100 · env 118.8 µsLiger2NVIDIA A100 · env 130.3 µsLiger2NVIDIA H100 · env 260.5 µsTransformers2NVIDIA H100 · env 380.3 µsTransformers2NVIDIA A100 · env 2118.8 µsTransformers2Not measured on B200 for this workload. Challenges →
Source-native comparison · GPU NVIDIA H100 · Workload rows = 4096 · hidden = 1024 · fp32 · Protocol Liger-Kernel benchmark scripts · median · 2 results · last observed 2025-07-17Record history →
Estimated floor 5.01 µs · record 3.76× above itestimate, not evidence ›
DRAM 5.01 µs · bandwidth-bound on H100 SXM
every declared tensor crosses HBM exactly once (3,350 GB/s, H100 SXM datasheet)
no arithmetic formula for this family: bandwidth floor only
headroom-v1: a lower bound from declared tensors and datasheet peaks. A kernel can sit well above it for good reasons.
#
Implementation
Latency
vs #1
Trust
Observed
1LigerLiger-Kernel18.8µs1.00×Reported · BSD-2-Clause · source2025-07-17stale
1.25× faster than the baseline. Measured exactly what you asked. Reported by source; not independently reproduced.
source mirroredBSD-2-ClauseinstallableView source →Run detail →
223.6µs1.25×Reported · Apache-2.0 · source2025-07-17stale
Measured exactly what you asked. The source's designated baseline implementation. Reported by source; not independently reproduced.
source mirroredApache-2.0no install recipeView source →Run detail →
Scaling by hidden
Implementations
Implementation
Runtime
Best latency
Evidence
Availability
Semantics
Inputs and outputs
xfloat [rows, hidden]
weightfloat [hidden]
biasfloat [hidden]
yfloat [rows, hidden]
Axes and behavior
rowsvariable
hiddenvariable
determinismunspecified
constraintsNo mutation or aliasing
Identity
sha256bc260cced62b…
Sources: Liger-Kernel benchmarks (2025-07-17) · BSD-2-Clauselast observed 2025-07-17How records are decidedJSON