Skip to content
KernelIndex
Search⌘K
Sign in

Fused linear + CPO loss

16 eligible runs
cpo

CPO preference loss over batch×seq×hidden activations projected through a vocab×hidden LM head against integer targets, computed chunked without materializing logits.

Best usable
31.5msmedian · 2.17× vs fastest
LigerLiger-Kernel · BSD-2-Clause · triton

Reported evidence · last observed 2024-11-14. Reported by source; not independently reproduced.

pip install "liger-kernel==0.4.1"
View source →Run detail →
Source baseline · unbeaten · not usable as-is
14.5msmedian
TransformersHugging Face Transformers · Apache-2.0 · python

Reported evidence · last observed 2024-11-14. The source's designated baseline implementation. Reported by source; not independently reproduced.

No package · vendor the source →View source →Run detail →

Fastest by languagepython · Transformers · 14.5 ms · #1triton · Liger · 31.5 ms · 2.17×

Current records

Workloadseq = 1024 · batch = 2 · vocab = 128256 · hidden = 4096 · bf16seq = 1024 · batch = 4 · vocab = 128256 · hidden = 4096 · bf16seq = 1024 · batch = 8 · vocab = 128256 · hidden = 4096 · bf16seq = 1024 · batch = 16 · vocab = 128256 · hidden = 4096 · bf16
HardwareBest knownImplementationRuns
NVIDIA A100 · env 114.5 msTransformers2NVIDIA A100 · env 233.3 msLiger2

Not measured on H100, B200 for this workload. Challenges →

Source-native comparison · GPU NVIDIA A100 · Workload seq = 1024 · batch = 2 · vocab = 128256 · hidden = 4096 · bf16 · Protocol Liger-Kernel benchmark scripts · median · 2 results · last observed 2024-11-14Record history →
Estimated floor 523.5 µs · record 27.73× above itestimate, not evidence ›
DRAM 523.5 µs · bandwidth-bound on A100 80GB
every declared tensor crosses HBM exactly once (2,039 GB/s, A100 80GB datasheet)
no arithmetic formula for this family: bandwidth floor only
headroom-v1: a lower bound from declared tensors and datasheet peaks. A kernel can sit well above it for good reasons.
#
Implementation
Latency
vs #1
Trust
Observed
1
TransformersbaselineHugging Face Transformers
14.5ms
1.00×
Reported · Apache-2.0 · source
2024-11-14stale

Measured exactly what you asked. The source's designated baseline implementation. Reported by source; not independently reproduced.

source mirroredApache-2.0no install recipeView source →Run detail →
2
LigerLiger-Kernel
31.5ms
2.17×
Reported · BSD-2-Clause · source
2024-11-14stale

2.17× slower than the baseline. Measured exactly what you asked. Reported by source; not independently reproduced.

source mirroredBSD-2-ClauseinstallableView source →Run detail →

Scaling by batch

100.0 ms200.0 ms24816batch →14.5 ms26.1 ms51.9 ms105.0 msTransformers31.5 ms62.4 ms123.6 ms245.7 msLiger
TransformersLigerbest per workload · NVIDIA A100 · Liger-Kernel benchmark scripts held constant

Implementations

Implementation
Runtime
Best latency
Evidence
Availability
TransformersHugging Face Transformers
python
14.5ms
1.00×
Reported
Apache-2.0 · source
LigerLiger-Kernel
triton
31.5ms
2.17×
Reported
BSD-2-Clause · source

Semantics

Inputs and outputs
xfloat [batch, seq, hidden]
weightfloat [vocab, hidden]
targetint64 [batch, seq]
lossfloat [1]
Axes and behavior
seqvariable
batchvariable
vocabvariable
hiddenvariable
determinismunspecified
constraintsNo mutation or aliasing
Identity
sha2569f18d831dd7f…
Sources: Liger-Kernel benchmarks (2024-11-14) · BSD-2-Clauselast observed 2024-11-14How records are decidedJSON