KLM 1.6: a 99M hybrid base model, counted from its header, and the last-digit test behind its lead

David Hendrickson's post on X opens with "How much intelligence can you squeeze into just 99M parameters to run on your Phone or IoT?" and answers with Minima Labs' KLM 1.6: an open-weight base model that "ranks #1 among models under 250M parameters on the Open SLM Leaderboard", scores 55.9% on the ArithMark-3 arithmetic benchmark against 50.9% for GPT-X3 (149M), 39.2% for SmolLM2 (135M) and 36.6% for Liquid AI's LFM2.5 (230M), and got there by one change: "It extended the final learning-rate decay phase from 20B to 100B training tokens." The rest of the post is a spec sheet: 2K context, "Hybrid Gated DeltaNet-2 + attention", a 396MB FP32 download, Apache 2.0, and estimated sizes of about 198MB in FP16, 99MB at 8 bits and 50MB at 4 bits, which it labels "theoretical, not released quantizations."

The link in the post's reply was not in what the fxtwitter mirror returned, so I found the repository by searching Hugging Face: MinimaLabs/KLM-1.6, created on 2026-11-09, next to KLM-1.5 and KLM-1.1.

expand

This site already looked at KLM 1.6 once, as a side note in the Raon-1 nano teardown, which recomputed the leaderboard's index and pointed out who runs it. This piece is about the model itself: what the 99M parameters are, what a Gated DeltaNet-2 layer does with its memory, what "extended decay" means in steps and tokens, and what ArithMark-3 actually asks. I did not download the weights or run the model. Everything below comes from the repository's files, byte-range reads of the weight headers, the papers it cites, the benchmark's data, and my own arithmetic.

Ninety-nine million, counted

A safetensors file starts with an 8-byte length and then a JSON header that names every tensor, its dtype and its shape. For KLM-1.6 the header is 51,160 bytes, so two small range requests give the whole parameter inventory without touching the 396,104,400-byte file behind it. The tensors sum to 98,975,308 elements, exactly the card's figure. Of those, 98,812,308 are FP32 weights and 163,000 are integers: 125,000 int32 bigram keys and 38,000 int64 trigram keys that the n-gram memory searches. The FP32 weights take 395,249,232 bytes and the keys 804,000, which plus the header and its length prefix is the file size to the byte.

Grouped by what they do:

The embedding is the surprise. It is 4,160 × 512. SmolLM2-135M's tied embedding, from its config.json, is 49,152 × 576 = 28,311,552 parameters, about a fifth of that model; KLM spends 2.2%. The tokenizer is a byte-level BPE with 4,096 entries (256 bytes plus 3,837 merges plus <pad>, <bos> and <eos> at ids 0 to 2) and four more special tokens at ids 4,096 to 4,099: <|im_start|>, <|im_end|>, <think> and </think>. That makes the card's 4,100 tokens. The matrix has 4,160 rows because 4,160 is a multiple of 64, and generation_config.json suppresses ids 4,100 to 4,159 so the padding rows are never sampled. The pre-tokenizer regex matches \p{N} one character at a time, so every number is spelled out in single digits: "1408" is four tokens. The KLM 1.5 card says this directly ("splits every number into single digits"), and it matters later.

The integer keys explain one more detail. A trigram key is (prev2 × V + prev1) × V + token with V = 4,160, so it can reach about 7.2 × 10¹⁰, which does not fit in 32 bits. A bigram key tops out near 1.73 × 10⁷, which does. Hence one int32 table and one int64 table, and hence 804,000 bytes that no quantizer may touch.

The post's size arithmetic holds. At 2 bytes per weight the floats are 197.6 MB, and with the integer keys the total is 198.4 MB. At 8 bits it is 99.6 MB and at 4 bits 50.2 MB, before the scales any real 4-bit format adds. None of those files exist; the release is FP32 only.

The widget's lower half is the part a phone cares about once the weights are loaded. The six attention layers cache keys and values: 4 KV heads × 64 dimensions × 2 = 512 numbers per token per layer. The eighteen Gated DeltaNet-2 layers keep a fixed state of 4 heads × 128 × 128 = 65,536 numbers each, whatever the length. One recurrent layer and one attention layer hold the same amount at 128 tokens. The reference code always runs the recurrent state in FP32; the cache follows the dtype the model is loaded in, which for the released weights is FP32 too. At the full 2,048 tokens the attention cache is 25.2 MB and the recurrent state, with its convolution windows, about 5.1 MB. If all 24 layers were attention, the cache would be 101 MB, a quarter of the weights again.

One more read of a header settles the "only the decay changed" story in one direction. KLM-1.5's header is also 51,160 bytes, and it is byte-identical to KLM-1.6's: same tensor names, same shapes, same dtypes, same offsets. The two config.json files are identical, so are the two tokenizer.json files, and the two modeling_klm.py files differ only in the version number in the first line. Whatever moved between the versions, it was not the architecture.

Three recurrent layers, then one that looks back

configuration_klm.py decides the layer types in one line: a layer is attention when layer % 4 == 3. With 24 layers that puts attention at layers 3, 7, 11, 15, 19 and 23 (counting from zero) and Gated DeltaNet-2 everywhere else, the 3:1 ratio the card states. Every layer has the same shape around its mixer: an RMSNorm, the mixer, an RMSNorm, a SwiGLU with hidden size 1,408.

The attention layers are small and modern. Eight query heads share four key/value heads (grouped-query attention), each 64 wide. Queries and keys get a per-head RMSNorm before rotary embeddings with base 10,000. The output passes through a sigmoid gate computed from the layer's input, o_proj(out * sigmoid(gate_proj(x))). And every attention layer mixes its values with the values of the first attention layer through two learned weights, value_mix, a value-residual trick that gives later attention layers direct access to early token content.

Attention is also masked at document boundaries. A training sequence packs several documents separated by <eos>, and the mask only lets a token attend to earlier tokens of its own document. The recurrent state, the convolution windows and the n-gram history all reset at the same points. That is unusual for a base model this small, and it means the 2,048-token window is really "up to 2,048 tokens of one document".

What a Gated DeltaNet-2 layer does with its memory

The other 18 layers are where the "hybrid" lives. A linear-attention layer keeps a matrix $S$ instead of a cache. For a head with key width $d_k$ and value width $d_v$, $S$ is $d_k \times d_v$, and a query reads it with one product, $o_t = S_t^\top q_t$. Plain linear attention writes by adding $k_t v_t^\top$ every token. That is cheap, but nothing ever leaves: write two values at overlapping keys and every later read returns a blend.

The delta rule fixes that by reading before writing. It looks up what $S$ already returns for $k_t$, subtracts it, and writes the difference, so a repeated key replaces its value instead of piling on top. Gated DeltaNet adds a decay that shrinks the whole state a little each token, and Kimi Delta Attention made the decay per channel. The RWKV-7 G1k article derives that family with its own stepper, and the linear-attention roundup reads it as online regression.

All of those use one scalar, $\beta_t$, for two jobs: how much old content to erase, and how much new content to write. Gated DeltaNet-2 (Hatamizadeh, Choi and Kautz, NeurIPS 2026) splits it. Its Eq. 10, the "Gated Delta Rule-2", is

with $D_t = \mathrm{Diag}(\alpha_t)$ the per-channel decay, $b_t \in [0,1]^{d_k}$ an erase gate over key channels and $w_t \in [0,1]^{d_v}$ a write gate over value channels, each a sigmoid of its own projection of the token. Set both gates to the same scalar and it is Kimi Delta Attention; make the decay a scalar too and it is Gated DeltaNet.

KLM's recurrence() in modeling_klm.py is that equation written out one token at a time, in FP32, with the docstring

S <- (I - k (b*k)^T) diag(exp g) S + k (w*v)^T and o = q^T S / sqrt(K)

and the loop body does it in three moves: multiply the state by the decay, read erase = (b * k)^T S, then add k (w * v - erase)^T. Keys and queries are L2-normalized first, so a key has unit length. The toy below runs exactly those three moves on a 4 × 4 head.

The setup has two keys that overlap: $k_1 = (0.8, 0.6, 0, 0)$ and $k_2 = (0, 0.6, 0.8, 0)$, which share channel 2 and have a dot product of 0.36. Token 1 writes $v_1$ at $k_1$, token 2 writes $v_2$ at $k_2$, both with the gates fully open. After those two, reading at $k_1$ gives 0.87 on $v_1$'s channel and 0.36 on $v_2$'s: the second write leaked through the shared channel. That leak is the cost of storing many keys in 16 numbers, and in KLM's case in 16,384 per head.

Token 3 rewrites $k_1$ with $v_3$, and the sliders set its gates.

• With the gates tied at 1, the classic delta rule, $k_1$ reads back $v_3$ exactly. The damage lands on $k_2$: its own value drops from 1.00 to 0.87 and it picks up 0.36 of $v_3$.
• Tied at 0.5, the read at $k_1$ is 0.44 of the old value and 0.5 of the new. A tied gate can only interpolate between keeping the old association and replacing it.
• Untied, with $b = 1$ and $w = 0.5$, the old value is gone completely and the new one is written at half strength. No tied setting produces that. With $b = 1$ and $w = 0$ the layer forgets $k_1$ without writing anything.
• Channel-wise erase, $b = 0$ on the shared channel only, protects $k_2$: it keeps 1.00 on its own value. The price is that $k_1$ keeps 0.23 of its old value. The real gates are 128 wide and set per token by the model, so it can make that trade differently for every token and channel.

That is the whole idea of the paper, and its 1.3B-parameter, 100B-token comparison against Mamba-2, Gated DeltaNet, KDA and Mamba-3 is what the authors offer as evidence that it helps. KLM is a 99M model and does not test it. Nobody has published a KLM with Gated DeltaNet instead, so the architecture is a sensible choice here, not a demonstrated one.

expand

KLM's layer matches the right half of that figure: projections for $q$, $k$ and $v$, a width-4 causal convolution on each followed by SiLU, L2 norm on $q$ and $k$, separate projections for $b$ and $w$, and an RMSNorm and a gate on the output. Three details differ from the paper or are KLM's own choices.

1. The decay projection goes through a 128-wide bottleneck, 512 → 128 → 512 (the paper's text does not give a width for it), with the paper's $-\exp(a)\,\mathrm{softplus}(\cdot)$ form and a floor of −20 on the log-decay, so one token can shrink a channel by at most a factor of $e^{20}$.
2. The erase gate is a plain sigmoid in $[0, 1]$. The paper also allows scaling it to $[0, 2]$ so the state transition can have negative eigenvalues; KLM does not use that.
3. The hybrid ratio. The paper's figure alternates one recurrent and one sliding-window attention block. KLM uses three recurrent layers per attention layer, and its attention sees the whole document, not a window.

The full-attention census counts how common that 3:1 shape has become in large open models. KLM is the same recipe at 512 wide.

Two more inputs next to the embedding

Two pieces of KLM sit outside the layer stack, and both change what a layer sees as its input.

The n-gram memory is the table counted above. For every position, NgramMemory forms the bigram ending at the current token and the trigram ending there, looks each up by exact match in its sorted key list, and fetches a learned 32-wide row (row 0 for anything not in the list). Two small projections lift the rows to 512 wide and their sum becomes an input. Per token, that is two table rows read, not a matrix multiplied. The KLM 1.5 card describes these as "the 125,000 most frequent bigrams and 38,000 most frequent trigrams in the training data".

The residual stream is not a running sum. KLM cites Attention Residuals from the Kimi team, which replaces "add every layer's output into one vector" with a softmax over earlier outputs, weighted by a learned query per layer. The paper's Block variant sums the layers inside a block and attends over block summaries to save memory at scale. In KLM's code (AttnResSources), the sources are the token embedding, the n-gram vector, and one entry per layer holding that layer's mixer output plus its feed-forward output. Each mixer and each feed-forward reads a softmax mix of everything so far, and a feed-forward also sees its own mixer's fresh output as one more source. In the paper's pseudocode, whose block_size counts an attention and an MLP sublayer as two, that is Block AttnRes with block_size 2: one layer per block, 24 blocks, and a 49th query for the final readout: the attn_res_query tensor is 49 × 512. At 24 layers the memory saving that motivates blocks does not matter, so the finest granularity costs almost nothing.

What "extended decay" changed, and what else changed with it

The training section of the card reads like a three-stage recipe:

1. Superposition warm-up, 70,000 steps of Token-Superposition Training: each input position is the average of 7 consecutive tokens, and the model predicts the next 7 as an unordered bag. At 128 sequences of 2,048 positions per step that is 18.35B positions, and times 7 the 128B tokens of ClimbMix the card mentions "for the compute of about 18B".
2. Stable training on 80B tokens of ordinary next-token prediction at constant learning rate. Weight matrices use MuonEq-R at 0.01, and embeddings, norms and gates use AdamW at 3e-3, in bf16.
3. Decay while the learning rate falls to zero along a $1-\sqrt{t}$ curve.

KLM 1.6 "continues training from a checkpoint of our KLM 1.5 taken before its decay stage", so stages 1 and 2 are shared. Stage 3 is what differs. Concretely, from the two cards:

expand

A $1-\sqrt{t}$ decay drops fast at first and then crawls. Half the peak rate arrives a quarter of the way in: 5B tokens into KLM 1.5's decay, 25B into KLM 1.6's. The last tenth of each decay runs below 5.1% of peak, which is 2B tokens for 1.5 and 10B for 1.6. So "extended" means the model spends five times as many tokens in the low-rate regime where the loss settles, and 80B more tokens overall.

That is why I would not credit the 1.5 → 1.6 gains to the schedule alone, even though the card calls 1.6 "an incremental update". Three things changed at once: the decay length (and so the total token count went from 100B to 180B), the batch (128 to 192 sequences), and the data, at least to the extent of two new math sources. KLM 1.5's card shows how much the decay mix leans on math and Q&A: Math rises from 11% of the stable mix to 20% of the decay mix, Q&A from 13% to 23%, and web text falls from 56% to 26%. KLM 1.6's card lists the sources but drops the percentages, so I cannot say how much more math it saw.

The card's own table shows the pattern you would expect from that. All five scores went up: ARC-Easy by 1.39 points, ARC-Challenge by 0.85, HellaSwag by 1.54, PIQA by 1.63, and ArithMark-3 by 7.7, from 48.20% to 55.90%. The general benchmarks moved by one to two points each. The arithmetic score moved five times as much, on a stage where the data leans hardest on grade-school math.

ArithMark-3 is a last-digit test

ArithMark 3.0 is a dataset from AxiomicLabs, the lab that also makes GPT-X3 and runs the leaderboard. Its card describes 1,000 synthetic English word problems, 47 templates across 17 topics, grades 1 to 5, with "Random-choice accuracy is 25%". Each item is an unfinished sentence and four endings, scored by a base model's log-likelihood. The card's example:

Sam had 43 rubber bands. A friend gave Sam 19 more. Afterward, Sam had
   63 rubber bands. | 62 rubber bands. | 61 rubber bands. | 60 rubber bands.

I read all 1,000 lines of arithmark-3.jsonl. The four endings are always four consecutive integers. Every item has distractors at the answer plus 1 and minus 1, and one more at plus 2 (508 items) or minus 2 (492). Two things follow.

First, the correct answer is never the smallest or the largest of the four. It is one of the two middle values in all 1,000 items. A language model scoring each ending on its own cannot exploit that, since it never sees the four side by side, but a program that could would score 50% without reading the question.

Second, in 688 of the 1,000 items the four endings differ only in their last digit. The largest answer in the set is 191. So for most items the question the model actually answers is "which last digit?", given that the leading digits are already common to every option. That is a real skill (the carry has to be right) but a narrow one. With KLM's tokenizer every digit is its own token, so the leading digit tokens score the same for all four endings, and what separates them is the probability of the final digit (plus whatever that digit does to the unit words after it).

The scoring script, bencharithmark-3.py, computes both raw and length-normalized accuracy, where the normalized score divides the summed log-probability of an ending by its token count. The card says its scores are length-normalized, which is also the script's default. That can only matter where the endings tokenize to different lengths, and with a digit-level tokenizer like KLM's those are the 102 items whose endings differ in length, such as 99 against 100.

The test also has a size. With 1,000 items, a score near 56% has a binomial standard error of about 1.6 points, so the 7.7-point gain from 1.5 to 1.6 is real, and the 5-point lead over GPT-X3-150M probably is. Neither is huge.

expand

Put the post's comparison next to the whole board and it changes shape. The post picks four peers, and KLM 1.6 beats all four. Of the 257 leaderboard models with an ArithMark-3 score, 6 score higher than 55.9, and two of them are under 250M: AlgoDriveAI's Inferno-Exp-1.5, a 76M model at 71.0, and Meta's MobileLLM-R1-140M-base at 65.7. Neither is on the card's chart. So on arithmetic, KLM 1.6 is third under 250M, behind a model smaller than it.

Inferno-Exp-1.5 also shows how narrow the skill is. Its HellaSwag is 27.85 against a chance level of 25, and its PIQA 58.49 against 50: it can pick the right last digit and very little else. MobileLLM-R1-140M-base scores 33.84 on HellaSwag and 63.22 on PIQA, both well below KLM 1.6's 45.28 and 70.18. KLM 1.6 is the better general model of the three, and the post's framing ("A 99M model outperforming models more than 2x its size on arithmetic!") is true of the rows it shows and not of the board.

First under 250M, by 0.9 points

The Open SLM Leaderboard ranks by its Intelligence Index, and the formula is in the Space's index.html: rescale each benchmark so chance is 0 and perfect is 100, average HellaSwag, the mean of the two ARC sets, and PIQA at weight 1 each, add ArithMark-3 at weight 0.65, and divide by the total weight. ArithMark-3 is 17.8% of the index. Recomputing it from the page's 265 model rows gives KLM 1.6 33.95, the card's number, and first under 100M, under 150M and under 250M. GPT-X3-150M is second at 33.05. Two models score higher at any size, SmolLM2-360M at 41.86 and MobileLLM-R1-360M-base at 38.65, which matches the card's "only two models of any size score higher".

The Raon-1 nano article did this recount first and found the same thing I do: drop ArithMark-3 and GPT-X3-150M edges ahead, 32.72 to 32.38. The lead is the arithmetic score. The gap is also small against the noise. ArithMark-3's 1.6-point standard error alone moves KLM's index by about 0.37 per standard error, and the four other benchmarks add their own sampling noise on top. A 0.9-point lead is a lead, not a margin.

On provenance, the leaderboard's rules say submitters "Provide zero-shot results for the listed benchmarks using LM-eval harness or the linked custom ArithMark script", and that "Submitted results are independently checked by the leaderboard team before they are merged." KLM's card says it scored ARC, HellaSwag and PIQA with lm-eval 0.4.12 and ArithMark-3 with the official script, and took the peers' scores from the board. So KLM's own numbers are self-reported and checked by AxiomicLabs, and the comparison rows are whatever their submitters reported. I found nothing that says which precision or batch size each row used. The ArithMark script falls back to FP32 on CPU and warns that reduced-precision scores can depend on batch shape.

Running it on a CPU

The card's claim that "The model code is plain PyTorch" is accurate, and it has a consequence. There is no flash-linear-attention kernel: recurrence() is a Python for loop over time steps, run once per Gated DeltaNet-2 layer. Scoring or prefilling a full 2,048-token window is 18 × 2,048 = 36,864 iterations of that loop body, each a handful of small einsums. Generation with KLMCache is one step per token, which is where a recurrent layer is cheap anyway. So expect prefill on a laptop to be slow relative to the model's size, and generation to be fine. I did not time it.

Loading needs trust_remote_code=True, and the cache cannot be cropped (_is_stateful = True), so assisted decoding is off. Batched generation needs left padding. The KLM-1.6 card is direct about the rest: a base model with no instruction tuning and no safety alignment, English only, "The model's knowledge is thin, and the facts it states are often wrong." It ships the chat and thinking tokens already in the vocabulary so a fine-tune does not have to resize the embedding.

What would I use it for? Exactly what the post suggests and the card implies: a starting point for a narrow fine-tune (a classifier, a router, a tagger) that has to run on a CPU or a phone. 99.6 MB at 8 bits plus a 25 MB cache at full context is small enough to sit next to an app. For that job, what matters is how quickly it adapts to your task, and none of the published numbers measure that.

What I think

KLM 1.6 is a good small model with an honest card. The architecture is current (Gated DeltaNet-2, full attention in one layer of four, attention residuals) and the parameter budget is spent carefully: a 4,100-token vocabulary that frees almost all of the 99M for layers, and a 5.25M n-gram table that costs two lookups per token. The header and the card agree to the byte.

The two claims in the post are each narrower than they sound. "Only the decay changed" is true of the architecture and not of the training: the decay brought 80B more tokens, a bigger batch and more math. And "#1 under 250M" is true of an index where a 1,000-item last-digit test carries 17.8% of the weight and the lead is 0.9 points. On that test alone, a 76M math model and Meta's MobileLLM-R1-140M are ahead. If MinimaLabs wants the schedule claim to stand, the experiment is cheap at this size: run the 20B decay on the 1.6 data mix and batch, and publish both.

What I read and what I computed

Read: the KLM-1.6 and KLM-1.5 model cards, config.json, configuration_klm.py, modeling_klm.py, the tokenizer files and generation_config.json; the safetensors headers of both versions by byte range; the ArithMark 3.0 card, script and all 1,000 items; the Open SLM Leaderboard's index.html (formula, rules, model rows); the Gated DeltaNet-2 paper's method section (arXiv HTML v2); the Attention Residuals method section (arXiv source); the Token-Superposition Training abstract. Computed: the parameter groups and byte totals, the cache and state sizes, the decay steps, the toy's readouts, the distractor structure, the binomial error, and the index for every listed model. Not run: the model, the benchmark, or any of their code. Every benchmark score here is the publisher's or the leaderboard's.