Fused residual RMS MLP
Fused operation combining residual connection, RMSNorm, and complete SwiGLU MLP block. This pattern appears at every decoder layer after the attention block where the attention output residual is added, then immediately normalized, and fed into the MLP (gate_proj, up_proj, SiLU activation, element-wise multiply, down_proj).
Reported evidence · last observed 2026-08-14. Reported by source; not independently reproduced. No public source. License unknown.
Current records
Workloadsuite of 16 cases · mean latencyhidden_size = 16384 · intermediate_size = 5324816 cases
Not measured on H100 for this workload. Challenges →
18.53ms64% SOL1.00×Reported · license unknown · no source2026-08-14
Measured exactly what you asked. Reported by source; not independently reproduced.
28.57ms63% SOL1.00×Reported · license unknown · no source2026-08-15
Measured exactly what you asked. Reported by source; not independently reproduced.
38.57ms63% SOL1.00×Reported · license unknown · no source2026-08-15
Measured exactly what you asked. Reported by source; not independently reproduced.
48.59ms63% SOL1.01×Reported · license unknown · no source2026-07-28
Measured exactly what you asked. Reported by source; not independently reproduced.
58.60ms63% SOL1.01×Reported · license unknown · no source2026-07-31
Measured exactly what you asked. Reported by source; not independently reproduced.
68.62ms62% SOL1.01×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
78.70ms62% SOL1.02×Reported · license unknown · no source2026-07-18
Measured exactly what you asked. Reported by source; not independently reproduced.
88.75ms61% SOL1.03×Reported · license unknown · no source2026-07-24
Measured exactly what you asked. Reported by source; not independently reproduced.
98.76ms61% SOL1.03×Reported · license unknown · no source2026-07-21
Measured exactly what you asked. Reported by source; not independently reproduced.
108.78ms60% SOL1.03×Reported · license unknown · no source2026-07-20
Measured exactly what you asked. Reported by source; not independently reproduced.
118.81ms60% SOL1.03×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
128.82ms60% SOL1.03×Reported · license unknown · no source2026-08-12
Measured exactly what you asked. Reported by source; not independently reproduced.
138.83ms60% SOL1.04×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
148.85ms60% SOL1.04×Reported · license unknown · no source2026-07-14
Measured exactly what you asked. Reported by source; not independently reproduced.
158.88ms59% SOL1.04×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
168.89ms59% SOL1.04×Reported · license unknown · no source2026-08-15
Measured exactly what you asked. Reported by source; not independently reproduced.
178.92ms59% SOL1.05×Reported · license unknown · no source2026-07-30
Measured exactly what you asked. Reported by source; not independently reproduced.
188.95ms59% SOL1.05×Reported · license unknown · no source2026-07-30
Measured exactly what you asked. Reported by source; not independently reproduced.
199.01ms58% SOL1.06×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
209.22ms56% SOL1.08×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.