FP8 multi latent attention QKV projection

31 eligible runs

quantization

FP8-quantized Multi-Latent Attention (MLA) QKV projection with low-rank compression. Implements DeepSeek-V3's novel MLA mechanism with Q path (q_a_proj -> RMSNorm -> q_b_proj) and KV path (kv_a_proj_with_mqa -> split -> RMSNorm -> kv_b_proj). All linear layers use FP8 quantization with blockwise scaling. The kv_a_proj output dimension is padded to 640 (multiple of 128) for FP8 compatibility.

Fastest reported

351.4µsmean · 89% of SOL

LaserLicense unknown · unknown

Reported evidence · last observed 2026-08-23. Reported by source; not independently reproduced. No public source. License unknown.

No public sourceRun detail →

Current records

Workloadsuite of 16 cases · mean latencynum_heads = 128 · block_size = 128 · q_head_dim = 192 · v_head_dim = 128 · hidden_size = 7168 · q_b_out_dim = 24576 · q_lora_rank = 1536 · kv_b_out_dim = 32768 · kv_lora_rank = 512 · qk_nope_head_dim = 128 · qk_rope_head_dim = 64 · kv_a_out_dim_padded = 64016 cases

batch_size

seq_len

dtype

1128bf16 1256bf16 1512bf16 11024bf16 14096bf16 2512bf16 22048bf16 4128bf16 8128bf16 8256bf16 81024bf16 82048bf16 16512bf16 161024bf16 32128bf16 32512bf16

scrolls · 16 cases total

Aggregatessuite of 16 cases · mean latency

HardwareBest knownImplementationRuns

NVIDIA B200 · env 1351.4 µsLaser24 NVIDIA B200 · env 2456.5 µsRecursive7

Not measured on H100 for this workload. Challenges →

Source-native comparison · GPU NVIDIA B200 · Workload suite of 16 cases · mean latency · Protocol SOL-ExecBench evaluation stack v1.1 · mean · 24 results · last observed 2026-08-23Record history →

#

Implementation

Latency

vs #1

Trust

Observed

1

351.4µs89% SOL

1.00×

Reported · license unknown · no source

2026-08-23

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

2

Amir M. Mir | SF Tensor

373.3µs88% SOL

1.06×

Reported · license unknown · no source

2026-08-14

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

3

387.7µs87% SOL

1.10×

Reported · license unknown · no source

2026-08-01

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

4

Amir M. Mir | SF Tensor

397.5µs87% SOL

1.13×

Reported · license unknown · no source

2026-07-22

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

5

427.1µs86% SOL

1.22×

Reported · license unknown · no source

2026-08-23

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

6

450.2µs85% SOL

1.28×

Reported · license unknown · no source

2026-08-22

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

7

475.6µs84% SOL

1.35×

Reported · license unknown · no source

2026-08-20

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

8

486.1µs84% SOL

1.38×

Reported · license unknown · no source

2026-07-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

9

486.2µs84% SOL

1.38×

Reported · license unknown · no source

2026-08-20

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

10

487.7µs84% SOL

1.39×

Reported · license unknown · no source

2026-08-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

11

504.0µs83% SOL

1.43×

Reported · license unknown · no source

2026-08-16

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

12

517.5µs83% SOL

1.47×

Reported · license unknown · no source

2026-07-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

13

535.1µs82% SOL

1.52×

Reported · license unknown · no source

2026-08-19

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

14

535.7µs82% SOL

1.52×

Reported · license unknown · no source

2026-08-17

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

15

549.4µs82% SOL

1.56×

Reported · license unknown · no source

2026-08-11

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

16

644.8µs78% SOL

1.83×

Reported · license unknown · no source

2026-08-09

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

17

651.9µs78% SOL

1.86×

Reported · license unknown · no source

2026-08-09

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

18

Amir M. Mir | SF Tensor

744.0µs76% SOL

2.12×

Reported · license unknown · no source

2026-07-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

19

746.3µs75% SOL

2.12×

Reported · license unknown · no source

2026-08-08

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

20

852.5µs72% SOL

2.43×

Reported · license unknown · no source

2026-08-07

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

21

944.6µs70% SOL

2.69×

Reported · license unknown · no source

2026-07-19

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

22

962.7µs70% SOL

2.74×

Reported · license unknown · no source

2026-08-05

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

23

1.82ms54% SOL

5.18×

Reported · license unknown · no source

2026-07-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

24

6.48ms24% SOL

18.45×

Reported · license unknown · no source

2026-07-10

Measured exactly what you asked. Reported by source; not independently reproduced.

no sourcelicense unknownno install recipeRun detail →

Implementations

Implementation

Runtime

Best latency

Evidence

Availability

Amir M. Mir | SF Tensor

unknown

373.3µs

1.06×

Reported

license unknown · no source

unknown

387.7µs

1.10×

Reported

license unknown · no source

Amir M. Mir | SF Tensor

unknown

397.5µs

1.13×

Reported

license unknown · no source

unknown

351.4µs

1.00×

Reported

license unknown · no source

unknown

427.1µs

1.22×

Reported

license unknown · no source

unknown

456.5µs

1.30×

Reported

license unknown · no source

unknown

482.7µs

1.37×

Reported

license unknown · no source

unknown

944.6µs

2.69×

Reported

license unknown · no source

Show all 31 implementations ›

unknown

1.50ms

4.26×

Reported

license unknown · no source

unknown

6.35ms

18.07×

Reported

license unknown · no source

unknown

450.2µs

1.28×

Reported

license unknown · no source

unknown

475.6µs

1.35×

Reported

license unknown · no source

unknown

486.1µs

1.38×

Reported

license unknown · no source

unknown

486.2µs

1.38×

Reported

license unknown · no source

unknown

487.7µs

1.39×

Reported

license unknown · no source

unknown

504.0µs

1.43×

Reported

license unknown · no source

unknown

517.5µs

1.47×

Reported

license unknown · no source

unknown

535.1µs

1.52×

Reported

license unknown · no source

unknown

535.7µs

1.52×

Reported

license unknown · no source

unknown

549.4µs

1.56×

Reported

license unknown · no source

unknown

640.3µs

1.82×

Reported

license unknown · no source

unknown

651.9µs

1.86×

Reported

license unknown · no source

unknown

644.8µs

1.83×

Reported

license unknown · no source

Amir M. Mir | SF Tensor

unknown

712.1µs

2.03×

Reported

license unknown · no source

Amir M. Mir | SF Tensor

unknown

744.0µs

2.12×

Reported

license unknown · no source

unknown

746.3µs

2.12×

Reported

license unknown · no source

unknown

852.5µs

2.43×

Reported

license unknown · no source

unknown

962.7µs

2.74×

Reported

license unknown · no source

unknown

1.82ms

5.18×

Reported

license unknown · no source

unknown

6.48ms

18.45×

Reported

license unknown · no source

Amir M. Mir | SF Tensor

unknown

10.2ms

29.16×

Reported

license unknown · no source

Semantics

Inputs and outputs

hidden_statesbf16 [batch_size, seq_len, hidden_size]

q_a_proj_weightbf16 [q_lora_rank, hidden_size]

q_a_layernorm_weightbf16 [q_lora_rank]

q_b_proj_weightbf16 [q_b_out_dim, q_lora_rank]

kv_a_proj_weightbf16 [kv_a_out_dim_padded, hidden_size]

kv_a_layernorm_weightbf16 [kv_lora_rank]

kv_b_proj_weightbf16 [kv_b_out_dim, kv_lora_rank]

rms_norm_epsfp32 scalar

q_nopebf16 [batch_size, seq_len, num_heads, qk_nope_head_dim]

q_pebf16 [batch_size, seq_len, num_heads, qk_rope_head_dim]

k_nopebf16 [batch_size, seq_len, num_heads, qk_nope_head_dim]

k_pebf16 [batch_size, seq_len, 1, qk_rope_head_dim]

value_statesbf16 [batch_size, seq_len, num_heads, v_head_dim]

Axes and behavior

seq_lenvariable

num_headsconstant = 128

batch_sizevariable

block_sizeconstant = 128

q_head_dimconstant = 192

v_head_dimconstant = 128

hidden_sizeconstant = 7168

q_b_out_dimq_b_out_dim = num_heads * q_head_dim

q_lora_rankconstant = 1536

kv_b_out_dimkv_b_out_dim = num_heads * (qk_nope_head_dim + v_head_dim)

kv_lora_rankconstant = 512

qk_nope_head_dimconstant = 128

qk_rope_head_dimconstant = 64

kv_a_out_dim_paddedconstant = 640

determinismunspecified

constraintsNo mutation or aliasing

Identity

alias016_fp8_multi_latent_attention_qkv_projectionmodeldeepseek-v3sha25674742d003246…

Sources: NVIDIA SOL-ExecBench (2026-08-23)last observed 2026-08-23How records are decided JSON