Skip to content
KernelIndex
Search⌘K
Sign in

Altup hidden state collapse with magnitude normalization

22 eligible runs
rmsnorm

Collapses multiple AltUp prediction streams back into a single hidden state by projecting each stream through learned unembed matrices, normalizing their magnitudes to match the first stream, and averaging them together.

Fastest reported
93.9µsmean · 85% of SOL
Amir M. Mir | SF TensorLicense unknown · unknown

Reported evidence · last observed 2026-07-06. Reported by source; not independently reproduced. No public source. License unknown.

No public sourceRun detail →

Current records

Workloadsuite of 16 cases · mean latencyhidden_size = 3072 · altup_num_inputs = 316 cases
seq_len
batch_size
dtype
1281bf161282bf1612816bf1612832bf1612864bf161311bf162112bf162561bf1625616bf1625664bf163738bf1644916bf1651216bf1654132bf166131bf1610248bf16
scrolls · 16 cases total
Aggregatessuite of 16 cases · mean latency
HardwareBest knownImplementationRuns
NVIDIA B200 · env 193.9 µsAmir M. Mir | SF Tensor13NVIDIA B200 · env 296.8 µsAmir M. Mir | SF Tensor9

Not measured on H100 for this workload. Challenges →

Source-native comparison · GPU NVIDIA B200 · Workload suite of 16 cases · mean latency · Protocol SOL-ExecBench evaluation stack v1.0 · mean · 13 results · last observed 2026-08-13Record history →
#
Implementation
Latency
vs #1
Trust
Observed
1
Amir M. Mir | SF Tensor
93.9µs85% SOL
1.00×
Reported · license unknown · no source
2026-07-06

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
2
doubleAI
96.0µs84% SOL
1.02×
Reported · license unknown · no source
2026-06-26

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
3
Recursive
100.1µs83% SOL
1.07×
Reported · license unknown · no source
2026-06-06

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
4
Databricks
102.3µs84% SOL
1.09×
Reported · license unknown · no source
2026-07-01

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
5
ac4k
105.0µs82% SOL
1.12×
Reported · license unknown · no source
2026-07-01

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
6
doubleAI
113.7µs80% SOL
1.21×
Reported · license unknown · no source
2026-05-20

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
7
Argus
118.6µs79% SOL
1.26×
Reported · license unknown · no source
2026-07-06

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
8
Amir M. Mir | SF Tensor
224.2µs63% SOL
2.39×
Reported · license unknown · no source
2026-06-02

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
9
Amir M. Mir | SF Tensor
269.4µs57% SOL
2.87×
Reported · license unknown · no source
2026-05-27

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
10
Amir M. Mir | SF Tensor
276.5µs56% SOL
2.95×
Reported · license unknown · no source
2026-04-25

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
11
Cursor
480.2µs40% SOL
5.12×
Reported · license unknown · no source
2026-04-14

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
12
Amir M. Mir | SF Tensor
576.4µs35% SOL
6.14×
Reported · license unknown · no source
2026-04-22

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →
13
Bragaia
2.58ms14% SOL
27.50×
Reported · license unknown · no source
2026-05-28

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

Implementations

Implementation
Runtime
Best latency
Evidence
Availability
Databricks
unknown
102.3µs
1.09×
Reported
license unknown · no source
Geometric
unknown
113.4µs
1.21×
Reported
license unknown · no source
Recursive
unknown
100.1µs
1.07×
Reported
license unknown · no source
ac4k
unknown
105.0µs
1.12×
Reported
license unknown · no source
Argus
unknown
118.6µs
1.26×
Reported
license unknown · no source
TH3CHARLie | TileGym
unknown
155.2µs
1.65×
Reported
license unknown · no source
Cursor
unknown
480.2µs
5.12×
Reported
license unknown · no source
Bragaia
unknown
2.58ms
27.50×
Reported
license unknown · no source
Show all 22 implementations ›
Amir M. Mir | SF Tensor
unknown
96.8µs
1.03×
Reported
license unknown · no source
Databricks
unknown
107.9µs
1.15×
Reported
license unknown · no source
Recursive
unknown
103.5µs
1.10×
Reported
license unknown · no source
doubleAI
unknown
103.5µs
1.10×
Reported
license unknown · no source
ac4k
unknown
109.9µs
1.17×
Reported
license unknown · no source
doubleAI
unknown
113.7µs
1.21×
Reported
license unknown · no source
Geometric
unknown
132.9µs
1.42×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
224.2µs
2.39×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
269.4µs
2.87×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
276.5µs
2.95×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
576.4µs
6.14×
Reported
license unknown · no source
Amir M. Mir | SF Tensor
unknown
93.9µs
1.00×
Reported
license unknown · no source
Hyra
unknown
98.8µs
1.05×
Reported
license unknown · no source
doubleAI
unknown
96.0µs
1.02×
Reported
license unknown · no source

Semantics

Inputs and outputs
hidden_statesbf16 [altup_num_inputs, batch_size, seq_len, hidden_size]
unembed_proj_1bf16 [hidden_size, hidden_size]
unembed_proj_2bf16 [hidden_size, hidden_size]
epsilonfp32 scalar
outputbf16 [batch_size, seq_len, hidden_size]
Axes and behavior
seq_lenvariable
batch_sizevariable
hidden_sizeconstant = 3072
altup_num_inputsconstant = 3
determinismunspecified
constraintsNo mutation or aliasing
Identity
alias052_altup_hidden_state_collapse_with_magnitude_normalizationmodelgemma-3n-e2bsha2569e11991cad12…
Sources: NVIDIA SOL-ExecBench (2026-08-13)last observed 2026-08-13How records are decidedJSON