NVFP4 fused decoder layer pre norm attention
FP4-quantized fused decoder layer pre-normalization and complete attention computation. Combines RMSNorm on input hidden states, Q/K/V projections with FP4 quantization, per-head RMSNorm on Q and K, RoPE positional embeddings, Grouped Query Attention with KV repeat, and output projection with FP4 quantization.
Reported evidence · last observed 2026-08-20. Reported by source; not independently reproduced. No public source. License unknown.
Current records
Workloadsuite of 16 cases · mean latencyhead_dim = 128 · q_out_half = 2560 · hidden_size = 5120 · kv_out_half = 512 · q_out_features = 5120 · kv_out_features = 1024 · hidden_size_half = 2560 · q_out_scale_blocks = 320 · hidden_scale_blocks = 320 · kv_out_scale_blocks = 64 · num_attention_heads = 40 · num_key_value_heads = 816 cases
Not measured on H100 for this workload. Challenges →
1589.1µs94% SOL1.00×Reported · license unknown · no source2026-08-20
Measured exactly what you asked. Reported by source; not independently reproduced.
2639.2µs93% SOL1.09×Reported · license unknown · no source2026-07-26
Measured exactly what you asked. Reported by source; not independently reproduced.
3661.2µs93% SOL1.12×Reported · license unknown · no source2026-08-10
Measured exactly what you asked. Reported by source; not independently reproduced.
4664.6µs93% SOL1.13×Reported · license unknown · no source2026-07-13
Measured exactly what you asked. Reported by source; not independently reproduced.
5717.3µs93% SOL1.22×Reported · license unknown · no source2026-08-03
Measured exactly what you asked. Reported by source; not independently reproduced.
6832.6µs91% SOL1.41×Reported · license unknown · no source2026-08-03
Measured exactly what you asked. Reported by source; not independently reproduced.
7873.3µs91% SOL1.48×Reported · license unknown · no source2026-07-16
Measured exactly what you asked. Reported by source; not independently reproduced.
8998.0µs89% SOL1.69×Reported · license unknown · no source2026-08-20
Measured exactly what you asked. Reported by source; not independently reproduced.
9999.3µs89% SOL1.70×Reported · license unknown · no source2026-08-18
Measured exactly what you asked. Reported by source; not independently reproduced.
101.01ms89% SOL1.72×Reported · license unknown · no source2026-08-10
Measured exactly what you asked. Reported by source; not independently reproduced.
111.05ms89% SOL1.77×Reported · license unknown · no source2026-08-17
Measured exactly what you asked. Reported by source; not independently reproduced.
121.05ms89% SOL1.78×Reported · license unknown · no source2026-07-14
Measured exactly what you asked. Reported by source; not independently reproduced.
131.07ms89% SOL1.82×Reported · license unknown · no source2026-08-07
Measured exactly what you asked. Reported by source; not independently reproduced.
141.09ms88% SOL1.86×Reported · license unknown · no source2026-08-02
Measured exactly what you asked. Reported by source; not independently reproduced.
151.10ms88% SOL1.87×Reported · license unknown · no source2026-08-23
Measured exactly what you asked. Reported by source; not independently reproduced.
161.11ms88% SOL1.88×Reported · license unknown · no source2026-08-14
Measured exactly what you asked. Reported by source; not independently reproduced.
171.15ms88% SOL1.95×Reported · license unknown · no source2026-08-19
Measured exactly what you asked. Reported by source; not independently reproduced.
181.15ms88% SOL1.96×Reported · license unknown · no source2026-07-17
Measured exactly what you asked. Reported by source; not independently reproduced.
191.16ms88% SOL1.97×Reported · license unknown · no source2026-08-14
Measured exactly what you asked. Reported by source; not independently reproduced.
201.26ms87% SOL2.13×Reported · license unknown · no source2026-08-08
Measured exactly what you asked. Reported by source; not independently reproduced.
211.39ms85% SOL2.36×Reported · license unknown · no source2026-07-19
Measured exactly what you asked. Reported by source; not independently reproduced.
221.40ms85% SOL2.37×Reported · license unknown · no source2026-07-19
Measured exactly what you asked. Reported by source; not independently reproduced.
231.41ms85% SOL2.40×Reported · license unknown · no source2026-08-04
Measured exactly what you asked. Reported by source; not independently reproduced.
241.65ms83% SOL2.80×Reported · license unknown · no source2026-07-25
Measured exactly what you asked. Reported by source; not independently reproduced.
251.66ms83% SOL2.81×Reported · license unknown · no source2026-07-25
Measured exactly what you asked. Reported by source; not independently reproduced.
261.74ms82% SOL2.95×Reported · license unknown · no source2026-07-23
Measured exactly what you asked. Reported by source; not independently reproduced.
273.83ms66% SOL6.50×Reported · license unknown · no source2026-07-13
Measured exactly what you asked. Reported by source; not independently reproduced.
284.59ms61% SOL7.79×Reported · license unknown · no source2026-07-22
Measured exactly what you asked. Reported by source; not independently reproduced.