Skip to content
KernelIndex
Search⌘K
Sign in

SwiGLU MLP backward

29 eligible runs
mlp

Backward pass for SwiGLU MLP: computes gradients for input, gate_weight, up_weight, and down_weight through the chain output = down_proj(silu(gate_proj(x)) * up_proj(x))

Fastest reported
14.8msmean · 68% of SOL
ac4kLicense unknown · unknown

Reported evidence · last observed 2026-07-16. Reported by source; not independently reproduced. No public source. License unknown.

No public sourceRun detail →

Current records

Workloadsuite of 18 cases · mean latencyhidden_size = 16384 · intermediate_size = 5324818 cases
seq_len
batch_size
dtype
1281bf161288bf1612816bf1612832bf1612864bf1621116bf162561bf162562bf162564bf1625616bf163738bf165124bf165128bf1610242bf1610244bf1615714bf1620482bf1640961bf16
scrolls · 18 cases total
Aggregatessuite of 18 cases · mean latency
HardwareBest knownImplementationRuns
NVIDIA B200 · env 114.8 msac4k19NVIDIA B200 · env 215.1 msdoubleAI10

Not measured on H100 for this workload. Challenges →

Source-native comparison · GPU NVIDIA B200 · Workload suite of 18 cases · mean latency · Protocol SOL-ExecBench evaluation stack v1.1 · mean · 19 results · last observed 2026-08-24Record history →
#
Implementation
Latency
vs #1
Trust
Observed
1
ac4k
14.8ms68% SOL
1.00×
Reported · license unknown · no source
2026-07-16

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
2
Proud Wren @ NVIDIA
14.9ms67% SOL
1.01×
Reported · license unknown · no source
2026-07-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
3
RIAC as well
15.3ms63% SOL
1.03×
Reported · license unknown · no source
2026-08-19

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
4
Hyra
15.4ms62% SOL
1.04×
Reported · license unknown · no source
2026-07-17

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
5
doubleAI
15.7ms60% SOL
1.06×
Reported · license unknown · no source
2026-07-11

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
6
Geometric
15.8ms58% SOL
1.07×
Reported · license unknown · no source
2026-08-24

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
7
Amir M. Mir | SF Tensor
15.9ms58% SOL
1.08×
Reported · license unknown · no source
2026-07-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
8
Recursive
16.0ms58% SOL
1.08×
Reported · license unknown · no source
2026-07-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
9
Cursor
16.0ms58% SOL
1.08×
Reported · license unknown · no source
2026-07-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
10
Argus
16.2ms56% SOL
1.10×
Reported · license unknown · no source
2026-07-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
11
Geometric
16.2ms55% SOL
1.10×
Reported · license unknown · no source
2026-08-11

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
12
Geometric
16.3ms55% SOL
1.10×
Reported · license unknown · no source
2026-08-24

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
13
Geometric
16.3ms55% SOL
1.10×
Reported · license unknown · no source
2026-08-18

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
14
Geometric
16.4ms55% SOL
1.11×
Reported · license unknown · no source
2026-08-13

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
15
Geometric
16.4ms54% SOL
1.11×
Reported · license unknown · no source
2026-08-18

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
16
Geometric
16.5ms54% SOL
1.11×
Reported · license unknown · no source
2026-08-22

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
17
Geometric
16.5ms54% SOL
1.12×
Reported · license unknown · no source
2026-08-17

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
18
Geometric
16.8ms52% SOL
1.14×
Reported · license unknown · no source
2026-08-23

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
19
Geometric
31.1ms21% SOL
2.11×
Reported · license unknown · no source
2026-08-14

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

Implementations

Implementation
Runtime
Best latency
Evidence
Availability
Recursive
unknown
16.0ms
1.08×
Reported
license unknown · no source
Argus
unknown
16.2ms
1.10×
Reported
license unknown · no source
bismuth83
unknown
18.8ms
1.27×
Reported
license unknown · no source
doubleAI
unknown
15.7ms
1.06×
Reported
license unknown · no source
Proud Wren @ NVIDIA
unknown
15.7ms
1.06×
Reported
license unknown · no source
Geometric
unknown
16.2ms
1.10×
Reported
license unknown · no source
Geometric
unknown
16.3ms
1.10×
Reported
license unknown · no source
Geometric
unknown
16.3ms
1.10×
Reported
license unknown · no source
Show all 29 implementations ›
Geometric
unknown
16.4ms
1.11×
Reported
license unknown · no source
Geometric
unknown
16.4ms
1.11×
Reported
license unknown · no source
Geometric
unknown
16.5ms
1.11×
Reported
license unknown · no source
Geometric
unknown
16.5ms
1.12×
Reported
license unknown · no source
Argus
unknown
16.7ms
1.13×
Reported
license unknown · no source
Cursor
unknown
16.7ms
1.13×
Reported
license unknown · no source
Geometric
unknown
16.8ms
1.14×
Reported
license unknown · no source
doubleAI
unknown
16.9ms
1.14×
Reported
license unknown · no source
ac4k
unknown
14.8ms
1.00×
Reported
license unknown · no source
Proud Wren @ NVIDIA
unknown
14.9ms
1.01×
Reported
license unknown · no source
doubleAI
unknown
15.1ms
1.02×
Reported
license unknown · no source
RIAC as well
unknown
15.3ms
1.03×
Reported
license unknown · no source
Hyra
unknown
15.4ms
1.04×
Reported
license unknown · no source
Geometric
unknown
15.8ms
1.07×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
15.9ms
1.08×
Reported
license unknown · no source
Cursor
unknown
16.0ms
1.08×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
17.3ms
1.17×
Reported
license unknown · no source
Recursive
unknown
17.4ms
1.17×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
18.0ms
1.22×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
19.4ms
1.31×
Reported
license unknown · no source
Geometric
unknown
31.1ms
2.11×
Reported
license unknown · no source

Semantics

Inputs and outputs
grad_outputbf16 [batch_size, seq_len, hidden_size]
xbf16 [batch_size, seq_len, hidden_size]
gate_outputbf16 [batch_size, seq_len, intermediate_size]
up_outputbf16 [batch_size, seq_len, intermediate_size]
activated_gatebf16 [batch_size, seq_len, intermediate_size]
gate_weightbf16 [intermediate_size, hidden_size]
up_weightbf16 [intermediate_size, hidden_size]
down_weightbf16 [hidden_size, intermediate_size]
grad_xbf16 [batch_size, seq_len, hidden_size]
grad_gate_weightbf16 [intermediate_size, hidden_size]
grad_up_weightbf16 [intermediate_size, hidden_size]
grad_down_weightbf16 [hidden_size, intermediate_size]
Axes and behavior
seq_lenvariable
batch_sizevariable
hidden_sizeconstant = 16384
intermediate_sizeconstant = 53248
determinismunspecified
constraintsNo mutation or aliasing
Identity
alias005_swiglu_mlp_backwardmodelhermes-4-405bsha2567fe88d3a3a93…
Sources: NVIDIA SOL-ExecBench (2026-08-24)last observed 2026-08-24How records are decidedJSON