Skip to content
KernelIndex
Search⌘K
Sign in

Fused MoE expert MLP

84 eligible runs
moe

Top-k routed mixture-of-experts silu-gated MLP: tokens×hidden activations dispatched to experts holding fused gate/up and down projections, weighted by softmaxed router scores.

Fastest reported · usable today
2.10msmedian · 15.02× faster than baseline
LigerLiger-Kernel · BSD-2-Clause · triton

Reported evidence · last observed 2026-04-02. Reported by source; not independently reproduced.

pip install "liger-kernel==0.7.0"
View source →Run detail →

Fastest by languagetriton · Liger · 2.10 ms · #1python · Transformers · 31.5 ms · 15.02×

Current records

Workloadk = 8 · hidden = 2048 · tokens = 8192 · experts = 128 · intermediate = 768 · bf16k = 8 · hidden = 2048 · intermediate = 76813 cases
tokens
experts
dtype
128128bf16256128bf16512128bf161024128bf162048128bf164096128bf1681928bf16819216bf16819232bf16819264bf168192128bf1616384128bf1632768128bf16
scrolls · 13 cases total
HardwareBest knownImplementationRuns
NVIDIA H100 · env 12.10 msLiger4NVIDIA H100 · env 27.38 msLiger4NVIDIA H100 · env 39.55 msLiger4

Not measured on B200 for this workload. Challenges →

Source-native comparison · GPU NVIDIA H100 · Workload k = 8 · hidden = 2048 · tokens = 8192 · experts = 128 · intermediate = 768 · bf16 · Protocol Liger-Kernel benchmark scripts · median · 4 results · last observed 2026-04-02Record history →
Estimated floor 370.7 µs · record 5.65× above itestimate, not evidence ›
DRAM 370.7 µs · bandwidth-bound on H100 SXM
every declared tensor crosses HBM exactly once (3,350 GB/s, H100 SXM datasheet)
no arithmetic formula for this family: bandwidth floor only
headroom-v1: a lower bound from declared tensors and datasheet peaks. A kernel can sit well above it for good reasons.
#
Implementation
Latency
vs #1
Trust
Observed
1
LigerLiger-Kernel
2.10ms
1.00×
Reported · BSD-2-Clause · source
2026-04-02stale

15.02× faster than the baseline. Measured exactly what you asked. Reported by source; not independently reproduced.

source mirroredBSD-2-ClauseinstallableView source →Run detail →
2
LigerLiger-Kernel
2.23ms
1.06×
Reported · BSD-2-Clause · source
2026-04-02stale

14.14× faster than the baseline. Measured exactly what you asked. Reported by source; not independently reproduced.

source mirroredBSD-2-ClauseinstallableView source →Run detail →
3
TransformersbaselineHugging Face Transformers
31.5ms
15.02×
Reported · Apache-2.0 · source
2026-04-02stale

Measured exactly what you asked. The source's designated baseline implementation. Reported by source; not independently reproduced.

source mirroredApache-2.0no install recipeView source →Run detail →
4
TransformersbaselineHugging Face Transformers
32.0ms
15.29×
Reported · Apache-2.0 · source
2026-04-02stale

Measured exactly what you asked. The source's designated baseline implementation. Reported by source; not independently reproduced.

source mirroredApache-2.0no install recipeView source →Run detail →

Scaling by tokens

1.00 ms10.0 ms12825610242048409616k32ktokens →588.8 µs595.9 µs613.3 µs671.3 µs874.5 µs1.25 ms2.10 ms3.89 ms7.34 msLiger30.3 ms30.3 ms30.2 ms31.2 ms31.5 ms31.4 ms31.5 ms33.3 ms37.6 msTransformers
LigerTransformersbest per workload · NVIDIA H100 · Liger-Kernel benchmark scripts held constant · log scale

Implementations

Implementation
Runtime
Best latency
Evidence
Availability
TransformersHugging Face Transformers
python
4.48ms
7.60×
Reported
Apache-2.0 · source
LigerLiger-Kernel
triton
588.8µs
1.00×
Reported
BSD-2-Clause · source

Semantics

Inputs and outputs
xfloat [tokens, hidden]
gate_up_projfloat [experts, intermediate_double, hidden]
down_projfloat [experts, hidden, intermediate]
top_k_indexint32 [tokens, k]
top_k_weightsfloat [tokens, k]
yfloat [tokens, hidden]
Axes and behavior
kvariable
hiddenvariable
tokensvariable
expertsvariable
intermediatevariable
intermediate_doubleintermediate_double = intermediate * 2
determinismunspecified
constraintsNo mutation or aliasing
Identity
sha256cdde945e670f…
Sources: Liger-Kernel benchmarks (2026-04-02) · BSD-2-Clauselast observed 2026-04-02How records are decidedJSON