SwiGLU activation, Megatron layout
SwiGLU activation over a seq×batch×(2·ffn) tensor: chunk in two on the last dimension, silu(gate)·up. The projections are outside the timed region; the benchmark runs bf16 and sweeps the per-rank FFN size.
Reported evidence · last observed 2026-07-26. Reported by source; not independently reproduced.
pip install "liger-kernel==0.8.1"Fastest by languagetriton · Liger · 23.1 µs · #1python · PyTorch · 23.2 µs · 1.00×
Current records
Not measured on B200 for this workload. Challenges →
Estimated floor 10.0 µs · record 2.31× above itestimate, not evidence ›
1LigerLiger-Kernel23.1µs1.00×Reported · BSD-2-Clause · source2026-07-26
1.00× faster than the baseline. Measured exactly what you asked. Reported by source; not independently reproduced.
2PyTorchbaseline23.2µs1.00×Reported · BSD-3-Clause · source2026-07-26
Measured exactly what you asked. The source's designated baseline implementation. Reported by source; not independently reproduced.
3LigerLiger-Kernel23.3µs1.01×Reported · BSD-2-Clause · source2026-07-26
1.01× slower than the baseline. Measured exactly what you asked. Reported by source; not independently reproduced.
4PyTorchbaseline57.8µs2.50×Reported · BSD-3-Clause · source2026-07-26
Measured exactly what you asked. The source's designated baseline implementation. Reported by source; not independently reproduced.