Tanh gated residual add backward
Backward pass for tanh-gated residual addition. Computes gradients for residual, hidden_states, and gate parameter. Forward: output = residual + tanh(gate) * hidden_states * mask. Backward computes: grad_residual = grad_output, grad_hidden_states = grad_output * tanh(gate) * mask, grad_gate = sum(grad_output * hidden_states * mask * sech^2(gate)).
Reported evidence · last observed 2026-08-03. Reported by source; not independently reproduced. No public source. License unknown.
Current records
Workloadsuite of 16 cases · mean latencyhidden_size = 409616 cases
Not measured on H100 for this workload. Challenges →
19.80µs93% SOL1.00×Reported · license unknown · no source2026-08-03
Measured exactly what you asked. Reported by source; not independently reproduced.
210.1µs93% SOL1.03×Reported · license unknown · no source2026-07-26
Measured exactly what you asked. Reported by source; not independently reproduced.
310.3µs92% SOL1.05×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
411.1µs90% SOL1.14×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
511.2µs91% SOL1.14×Reported · license unknown · no source2026-08-13
Measured exactly what you asked. Reported by source; not independently reproduced.
611.4µs90% SOL1.17×Reported · license unknown · no source2026-08-13
Measured exactly what you asked. Reported by source; not independently reproduced.
711.6µs89% SOL1.18×Reported · license unknown · no source2026-07-24
Measured exactly what you asked. Reported by source; not independently reproduced.
812.0µs89% SOL1.23×Reported · license unknown · no source2026-07-16
Measured exactly what you asked. Reported by source; not independently reproduced.
912.0µs89% SOL1.23×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
1012.1µs89% SOL1.23×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
1113.6µs86% SOL1.39×Reported · license unknown · no source2026-07-11
Measured exactly what you asked. Reported by source; not independently reproduced.
1215.1µs83% SOL1.54×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.
1325.1µs69% SOL2.56×Reported · license unknown · no source2026-07-10
Measured exactly what you asked. Reported by source; not independently reproduced.