Post norm residual
Post-normalization residual connection pattern: output = residual + RMSNorm(sublayer_output). RMSNorm applies learned weight scaling after normalization. Used after both attention and MLP blocks in Olmo3 architecture.
Reported evidence · last observed 2026-06-07. Reported by source; not independently reproduced. No public source. License unknown.
Current records
Workloadsuite of 16 cases · mean latencyhidden_size = 409616 cases
Not measured on H100 for this workload. Challenges →
115.5µs80% SOL1.00×Reported · license unknown · no source2026-06-07
Measured exactly what you asked. Reported by source; not independently reproduced.
215.9µs78% SOL1.02×Reported · license unknown · no source2026-07-01
Measured exactly what you asked. Reported by source; not independently reproduced.
316.3µs78% SOL1.05×Reported · license unknown · no source2026-07-01
Measured exactly what you asked. Reported by source; not independently reproduced.
416.5µs78% SOL1.06×Reported · license unknown · no source2026-06-25
Measured exactly what you asked. Reported by source; not independently reproduced.
517.2µs76% SOL1.11×Reported · license unknown · no source2026-07-05
Measured exactly what you asked. Reported by source; not independently reproduced.
617.5µs76% SOL1.13×Reported · license unknown · no source2026-05-20
Measured exactly what you asked. Reported by source; not independently reproduced.
718.5µs74% SOL1.19×Reported · license unknown · no source2026-04-25
Measured exactly what you asked. Reported by source; not independently reproduced.
818.5µs74% SOL1.19×Reported · license unknown · no source2026-07-08
Measured exactly what you asked. Reported by source; not independently reproduced.
919.2µs73% SOL1.24×Reported · license unknown · no source2026-04-14
Measured exactly what you asked. Reported by source; not independently reproduced.
1019.2µs73% SOL1.24×Reported · license unknown · no source2026-07-07
Measured exactly what you asked. Reported by source; not independently reproduced.
1121.8µs68% SOL1.40×Reported · license unknown · no source2026-05-07
Measured exactly what you asked. Reported by source; not independently reproduced.
1227.6µs60% SOL1.78×Reported · license unknown · no source2026-04-15
Measured exactly what you asked. Reported by source; not independently reproduced.
1339.6µs50% SOL2.55×Reported · license unknown · no source2026-05-28
Measured exactly what you asked. Reported by source; not independently reproduced.
1443.7µs46% SOL2.82×Reported · license unknown · no source2026-04-15
Measured exactly what you asked. Reported by source; not independently reproduced.
1543.7µs46% SOL2.82×Reported · license unknown · no source2026-04-15
Measured exactly what you asked. Reported by source; not independently reproduced.
16252.7µs11% SOL16.28×Reported · license unknown · no source2026-04-23
Measured exactly what you asked. Reported by source; not independently reproduced.