Attention softmax with softcapping and dropout
Attention softmax with logit softcapping (tanh-based clamping to ±30.0) applied before softmax, as used in Gemma-2 models. This operation applies: tanh(logits / 30.0) * 30.0 to stabilize attention scores, then softmax normalization.
Reported evidence · last observed 2026-07-16. Reported by source; not independently reproduced. No public source. License unknown.
Current records
Workloadsuite of 16 cases · mean latencynum_heads = 8 · attn_logit_softcapping = 3016 cases
Not measured on H100 for this workload. Challenges →
17.51µs89% SOL1.00×Reported · license unknown · no source2026-07-16
Measured exactly what you asked. Reported by source; not independently reproduced.
27.76µs88% SOL1.03×Reported · license unknown · no source2026-08-03
Measured exactly what you asked. Reported by source; not independently reproduced.
37.97µs87% SOL1.06×Reported · license unknown · no source2026-07-18
Measured exactly what you asked. Reported by source; not independently reproduced.
48.06µs87% SOL1.07×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
58.42µs87% SOL1.12×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
68.43µs87% SOL1.12×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
78.46µs87% SOL1.13×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
88.71µs85% SOL1.16×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
98.72µs85% SOL1.16×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
108.84µs84% SOL1.18×Reported · license unknown · no source2026-08-24
Measured exactly what you asked. Reported by source; not independently reproduced.
119.37µs82% SOL1.25×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
1210.7µs79% SOL1.43×Reported · license unknown · no source2026-07-21
Measured exactly what you asked. Reported by source; not independently reproduced.
1315.6µs68% SOL2.08×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.