SwiGLU MLP
Llama-style SwiGLU MLP: silu(x·Wgate)·(x·Wup) projected back down, over batch×seq×hidden activations.
Reported evidence · last observed 2025-11-11. Reported by source; not independently reproduced.
pip install "liger-kernel==0.6.3"Fastest by languagetriton · Liger · 659.5 µs · #1python · Transformers · 674.8 µs · 1.02×
Current records
Workloadseq = 1024 · batch = 2 · hidden = 2048 · intermediate = 4096 · bf169 cases
Not measured on H100, B200 for this workload. Challenges →
Estimated floor 58.3 µs · record 11.32× above itestimate, not evidence ›
1LigerLiger-Kernel659.5µs1.00×Reported · BSD-2-Clause · source2025-11-11stale
1.02× faster than the baseline. Measured exactly what you asked. Reported by source; not independently reproduced.
2674.8µs1.02×Reported · Apache-2.0 · source2025-11-11stale
Measured exactly what you asked. The source's designated baseline implementation. Reported by source; not independently reproduced.
3LigerLiger-Kernel739.5µs1.12×Reported · BSD-2-Clause · source2025-11-11stale
1.10× slower than the baseline. Measured exactly what you asked. Reported by source; not independently reproduced.
4745.5µs1.13×Reported · Apache-2.0 · source2025-11-11stale
Measured exactly what you asked. The source's designated baseline implementation. Reported by source; not independently reproduced.