Fused add RMSNorm h4096
Fused Add + RMSNorm with hidden_size=4096 for Llama-3.1-8B. Epsilon is fixed at 1e-5.
Reported evidence · last observed 2026-07-23. Reported by source; not independently reproduced. No public source. License unknown.
Current records
Workloadsuite of 14 cases · mean latency28 cases
Not measured on H100 for this workload. Challenges →
17.10µs89% SOL1.00×Reported · license unknown · no source2026-07-23
Measured exactly what you asked. Reported by source; not independently reproduced.
27.11µs88% SOL1.00×Reported · license unknown · no source2026-08-03
Measured exactly what you asked. Reported by source; not independently reproduced.
37.11µs88% SOL1.00×Reported · license unknown · no source2026-08-19
Measured exactly what you asked. Reported by source; not independently reproduced.
47.13µs88% SOL1.00×Reported · license unknown · no source2026-08-03
Measured exactly what you asked. Reported by source; not independently reproduced.
57.18µs88% SOL1.01×Reported · license unknown · no source2026-07-22
Measured exactly what you asked. Reported by source; not independently reproduced.
67.20µs89% SOL1.01×Reported · license unknown · no source2026-08-01
Measured exactly what you asked. Reported by source; not independently reproduced.
77.21µs88% SOL1.02×Reported · license unknown · no source2026-08-01
Measured exactly what you asked. Reported by source; not independently reproduced.
87.22µs88% SOL1.02×Reported · license unknown · no source2026-08-21
Measured exactly what you asked. Reported by source; not independently reproduced.
97.24µs89% SOL1.02×Reported · license unknown · no source2026-08-01
Measured exactly what you asked. Reported by source; not independently reproduced.
107.25µs88% SOL1.02×Reported · license unknown · no source2026-08-05
Measured exactly what you asked. Reported by source; not independently reproduced.
117.25µs89% SOL1.02×Reported · license unknown · no source2026-07-30
Measured exactly what you asked. Reported by source; not independently reproduced.
127.25µs88% SOL1.02×Reported · license unknown · no source2026-08-03
Measured exactly what you asked. Reported by source; not independently reproduced.
137.27µs88% SOL1.02×Reported · license unknown · no source2026-08-01
Measured exactly what you asked. Reported by source; not independently reproduced.
147.29µs88% SOL1.03×Reported · license unknown · no source2026-07-31
Measured exactly what you asked. Reported by source; not independently reproduced.
157.42µs87% SOL1.05×Reported · license unknown · no source2026-08-02
Measured exactly what you asked. Reported by source; not independently reproduced.
167.47µs83% SOL1.05×Reported · license unknown · no source2026-07-31
Measured exactly what you asked. Reported by source; not independently reproduced.
177.61µs83% SOL1.07×Reported · license unknown · no source2026-07-30
Measured exactly what you asked. Reported by source; not independently reproduced.
187.90µs83% SOL1.11×Reported · license unknown · no source2026-07-30
Measured exactly what you asked. Reported by source; not independently reproduced.
197.91µs82% SOL1.11×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
207.92µs82% SOL1.12×Reported · license unknown · no source2026-07-29
Measured exactly what you asked. Reported by source; not independently reproduced.
217.93µs84% SOL1.12×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
228.00µs81% SOL1.13×Reported · license unknown · no source2026-07-29
Measured exactly what you asked. Reported by source; not independently reproduced.
238.02µs81% SOL1.13×Reported · license unknown · no source2026-07-29
Measured exactly what you asked. Reported by source; not independently reproduced.
248.07µs81% SOL1.14×Reported · license unknown · no source2026-07-17
Measured exactly what you asked. Reported by source; not independently reproduced.
258.09µs79% SOL1.14×Reported · license unknown · no source2026-07-25
Measured exactly what you asked. Reported by source; not independently reproduced.
268.13µs80% SOL1.15×Reported · license unknown · no source2026-07-28
Measured exactly what you asked. Reported by source; not independently reproduced.
278.21µs81% SOL1.16×Reported · license unknown · no source2026-07-23
Measured exactly what you asked. Reported by source; not independently reproduced.
288.21µs79% SOL1.16×Reported · license unknown · no source2026-07-27
Measured exactly what you asked. Reported by source; not independently reproduced.
298.41µs80% SOL1.18×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
308.46µs82% SOL1.19×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
318.53µs80% SOL1.20×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
328.55µs80% SOL1.21×Reported · license unknown · no source2026-07-14
Measured exactly what you asked. Reported by source; not independently reproduced.
339.16µs78% SOL1.29×Reported · license unknown · no source2026-07-29
Measured exactly what you asked. Reported by source; not independently reproduced.
349.54µs79% SOL1.34×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
3510.8µs76% SOL1.53×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
3612.1µs67% SOL1.70×Reported · license unknown · no source2026-07-28
Measured exactly what you asked. Reported by source; not independently reproduced.
Implementations
Show all 62 implementations ›Showing all 62 implementations ⌄
Semantics
Also indexed as 002-fused-add-rmsnorm-h4096 · reviewed as the same computation; every definition's cohorts are shown here, ranked separately.