Flux multi head RMSNorm QK
Per-head RMSNorm applied to query and key tensors after projection in Flux attention. Applies RMSNorm independently to each attention head's Q and K vectors, normalizing across head_dim (128) for each of the 48 heads. This dual variant processes both Q and K tensors together to enable kernel fusion opportunities.
Reported evidence · last observed 2026-08-17. Reported by source; not independently reproduced. No public source. License unknown.
Current records
Workloadsuite of 16 cases · mean latencyhead_dim = 128 · num_heads = 4816 cases
Not measured on H100 for this workload. Challenges →
123.0µs62% SOL1.00×Reported · license unknown · no source2026-08-17
Measured exactly what you asked. Reported by source; not independently reproduced.
223.5µs60% SOL1.02×Reported · license unknown · no source2026-08-12
Measured exactly what you asked. Reported by source; not independently reproduced.
323.6µs60% SOL1.03×Reported · license unknown · no source2026-08-23
Measured exactly what you asked. Reported by source; not independently reproduced.
423.9µs60% SOL1.04×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
524.7µs59% SOL1.08×Reported · license unknown · no source2026-08-12
Measured exactly what you asked. Reported by source; not independently reproduced.
625.0µs58% SOL1.09×Reported · license unknown · no source2026-07-15
Measured exactly what you asked. Reported by source; not independently reproduced.
725.1µs58% SOL1.10×Reported · license unknown · no source2026-08-16
Measured exactly what you asked. Reported by source; not independently reproduced.
825.2µs58% SOL1.10×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
925.2µs58% SOL1.10×Reported · license unknown · no source2026-08-14
Measured exactly what you asked. Reported by source; not independently reproduced.
1025.2µs58% SOL1.10×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
1125.2µs58% SOL1.10×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
1225.2µs58% SOL1.10×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
1325.3µs58% SOL1.10×Reported · license unknown · no source2026-08-14
Measured exactly what you asked. Reported by source; not independently reproduced.
1425.3µs58% SOL1.10×Reported · license unknown · no source2026-08-14
Measured exactly what you asked. Reported by source; not independently reproduced.
1525.5µs57% SOL1.11×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
1626.9µs55% SOL1.17×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
1730.6µs51% SOL1.33×Reported · license unknown · no source2026-07-20
Measured exactly what you asked. Reported by source; not independently reproduced.