NVFP4 attention output projection
FP4-quantized attention output projection (o_proj) that transforms concatenated multi-head attention outputs back to hidden dimension. Uses FP4 block-scaled GEMM with 16 elements per FP8 scale factor. Computes C = (A * scale_a * global_a) @ (B^T * scale_b * global_b) where A is the input activation and B is the weight matrix.
Reported evidence · last observed 2026-07-21. Reported by source; not independently reproduced. No public source. License unknown.
Current records
Workloadsuite of 16 cases · mean latencysf_k = 256 · input_dim = 4096 · hidden_size = 2688 · sf_vec_size = 16 · hidden_size_padded = 281616 cases
Not measured on H100 for this workload. Challenges →
162.2µs82% SOL1.00×Reported · license unknown · no source2026-07-21
Measured exactly what you asked. Reported by source; not independently reproduced.
266.5µs82% SOL1.07×Reported · license unknown · no source2026-08-01
Measured exactly what you asked. Reported by source; not independently reproduced.
372.8µs80% SOL1.17×Reported · license unknown · no source2026-07-18
Measured exactly what you asked. Reported by source; not independently reproduced.
475.3µs79% SOL1.21×Reported · license unknown · no source2026-07-13
Measured exactly what you asked. Reported by source; not independently reproduced.
575.7µs79% SOL1.22×Reported · license unknown · no source2026-07-16
Measured exactly what you asked. Reported by source; not independently reproduced.
676.0µs79% SOL1.22×Reported · license unknown · no source2026-08-10
Measured exactly what you asked. Reported by source; not independently reproduced.
790.4µs75% SOL1.45×Reported · license unknown · no source2026-08-05
Measured exactly what you asked. Reported by source; not independently reproduced.
8122.0µs69% SOL1.96×Reported · license unknown · no source2026-07-13
Measured exactly what you asked. Reported by source; not independently reproduced.
9129.2µs68% SOL2.08×Reported · license unknown · no source2026-08-03
Measured exactly what you asked. Reported by source; not independently reproduced.
10141.7µs65% SOL2.28×Reported · license unknown · no source2026-07-13
Measured exactly what you asked. Reported by source; not independently reproduced.
11146.6µs65% SOL2.36×Reported · license unknown · no source2026-08-11
Measured exactly what you asked. Reported by source; not independently reproduced.
12160.9µs62% SOL2.59×Reported · license unknown · no source2026-07-22
Measured exactly what you asked. Reported by source; not independently reproduced.
13168.5µs61% SOL2.71×Reported · license unknown · no source2026-07-22
Measured exactly what you asked. Reported by source; not independently reproduced.
14458.8µs36% SOL7.38×Reported · license unknown · no source2026-07-22
Measured exactly what you asked. Reported by source; not independently reproduced.