Gated DeltaNet hybrid (3 linear : 1 full) · 2024 · SSM / RNN · 12 min
- linear-attention
- delta-rule
- hybrid
- fixed-state
- hybrid-attention
- explainer
A 2:13 narrated explainer, drawn in code. Every number and picture in it is this page's own; the sources are below.
› transcript
A memory the same size at token ten and at token one million. What does it have to forget? Hi, I'm Ripley! This is the layer that fills three slots in four of Qwen's recent models. Sixty-four layers. Forty-eight keep a fixed memory. Sixteen are ordinary attention, and their cache grows with every token. Each head keeps one matrix, the state. Every token edits it. Alpha decays the whole state. Near zero, it wipes the memory. Then it reads what the state holds for this key, and takes it out. And writes the new value in its place. Beta sets how hard. Follow one token through one head. It brings a query, a key, a value and two gates. First the decay. Then the erase. A repeated key replaces its value instead of blending with it. Then the write. The query reads the result with one product. Same cost at any length. The paper's block. Short convolutions on the query, key and value, a norm on the keys, and a gate on the output. Its hybrids paired the layer with windowed attention. Production models chose full attention instead. So a block is three recurrent layers and one attention layer, sixteen times over. Only the attention layer keeps a cache that grows. It does the exact lookups a summary cannot. One layer in four stays global, because a finite state cannot reproduce exact token-level retrieval no matter how well it is gated. At a quarter of a million tokens, the sixteen attention layers hold seventeen gigabytes. All sixty-four would hold almost seventy. The fixed part is small, and fragile. Store it in sixteen-bit floats and the score falls fifteen points. Three layers keep a cheap memory they can edit. The fourth keeps everything, for exact recall. Decay, erase, write, and one attention layer in four. Thanks for watching! Every source is in the full article. I'm Ripley. Bye!
A Gated DeltaNet layer keeps a matrix of fixed size per head instead of a cache. Each token first shrinks it by a decay α, then reads what the matrix already stores for its key and replaces it with the new value, at a strength β the token picks. Three such layers share each block with one full-attention layer, whose cache still grows and does the exact retrieval a summary cannot.
Open a recent Qwen config.json and the layer_types list reads like a drum pattern: three linear_attention, one full_attention, repeated. Qwen3.8-27B has 64 layers, 48 of them Gated DeltaNet and 16 full attention. KLM 1.6, a 99M-parameter base model, puts attention at layers 3, 7, 11, 15, 19 and 23 and Gated DeltaNet-2 everywhere else. The full-attention census found that most DeltaNet and KDA designs sit at exactly that 3 to 1 ratio.
This page builds the layer that fills three slots in four, from plain linear attention up, then explains why the fourth slot is still softmax attention. The source is the Gated DeltaNet paper (Yang, Kautz and Hatamizadeh, from MIT and NVIDIA); the numbers on real models come from this site's pieces on KLM 1.6, the census and the linear-attention roundup.
A cache that grows, and a state that doesn't
Softmax attention keeps every key and value it has seen. For a query it computes
and the softmax couples every term to every other, so nothing can be summarised in advance. Decoding token reads cached entries per layer. At long context that read, and the memory holding it, is the cost.
Drop the softmax and the sum factorises. Following the paper's convention, with the state a matrix:
That is linear attention. The state is the same size at token 10 and at token 1,000,000: one matrix per head. Reading it is one matrix-vector product. In the paper's words, it "underperforms Transformers in language modeling by a large margin", and the reason is visible in the update. Every token adds an outer product and nothing ever leaves. Write two values at overlapping keys and every later read returns a blend of both.
Two repairs: forget, and overwrite
The first repair is a decay. Mamba-2, written as linear attention, multiplies the state by a data-dependent scalar before each write:
The model can now clear its memory when the topic changes. It cannot clear one association without fading all of them; the decay is global.
The second repair is the delta rule. Treat as a memory that maps keys to values. Before writing, read what it currently returns for , call that , take it out, and put a blend of the new and old value back:
is the writing strength. With unit-length keys and , the slot along is replaced exactly. Associations along orthogonal keys are untouched. The transition is a generalised Householder matrix: it shrinks the state along one direction and leaves the rest alone.
The paper reads this as optimisation. The delta rule is one step of gradient descent on the regression loss , with learning rate :
So a delta-rule layer is an online regressor fitted to the key-value pairs it has seen, and its state is the regressor's weights. The linear-attention roundup builds its whole comparison on that reading.
The gated delta rule
Gated DeltaNet puts the two repairs together. Its Eq. 10 is
Both gates are computed from the token. When the memory is wiped in one step. When the layer is a pure delta rule and edits one association without touching the others. In optimisation terms, is an adaptive weight decay on the online regressor.
Per head and per token, the work is three moves on a matrix:
- Decay: multiply the state by .
- Erase: read and subtract .
- Write: add .
Then the query reads it, . The diagram above draws those three moves on a 4 by 4 head. KLM's reference code is the same loop written out one token at a time in FP32: multiply the state by the decay, read the erase term, add the corrected write.
The paper's single-needle results (its Table 2, 1.3B models trained on 100B tokens) show why both repairs matter. On S-NIAH-1, a pass-key buried in repeated filler, Mamba-2 falls from 99.2 at 1K tokens to 30.4 at 8K because its decay forgets too fast, while DeltaNet holds 98.8. On S-NIAH-2, a number hidden in real essays, DeltaNet drops to 14.4 at 8K because it never clears anything and the needle collides with everything else, while Gated DeltaNet keeps 92.2 at 4K. The authors summarise it as three observations: decay hurts retention, gating helps filtering, and the delta rule helps memorisation.
Inside the layer
The block around the recurrence is close to a Llama layer with the attention swapped out. Queries, keys and values each come from a linear projection, a short causal convolution and SiLU; queries and keys are then L2-normalised, which keeps the Householder transition stable. and come from their own linear projections, in Mamba-2's parameterisation. The output is normalised and multiplied by a gate before the output projection. The paper's figure shows that block next to its two hybrid layouts:

Production models kept the layer and changed the hybrid. The paper's H1 alternates one Gated DeltaNet layer with one sliding-window attention layer. Qwen and KLM put three recurrent layers under one full-attention layer that sees the whole context.
Training it in parallel
Written as a recurrence, the layer is a loop over tokens, which a GPU hates. Written fully in parallel, as a masked matrix, it is quadratic again. The paper trains it chunkwise: split the sequence into chunks of tokens, run the recurrence only across chunk boundaries, and do everything inside a chunk with matrix multiplies.
The hard part is the product of Householder transitions inside a chunk. The paper extends the WY representation that DeltaNet's parallel algorithm used (Yang et al., 2024) to carry the decay. A product of transitions collapses to , and the vectors and their value-side twins come from one triangular solve over the chunk:
where holds the ratios of cumulative decays inside the chunk. After that, the chunk's output and the state handed to the next chunk are a few dense matmuls. The paper's throughput comparison puts Gated DeltaNet at essentially DeltaNet's speed and slightly behind Mamba-2, whose scalar transition is cheaper than a Householder one. The same chunkwise shape is what Kimi K3's KDA layers and Mamba-2's SSD use.
Why one layer in four stays attention
A fixed state is a lossy summary. The roundup's toy recall task puts a number on it: the delta rule recalls 96.9% of 8 stored pairs and 49.2% of 32. A softmax layer recalls whatever it was given, because it kept all of it. The team behind Qwen3.8-Flash-Next gives the design rule plainly: one layer in four stays global "because a finite state cannot reproduce exact token-level retrieval no matter how well it is gated".
The paper says the same about linear models in general. They "have limitations in modeling local shifts and comparisons, and their fixed state size makes it hard for retrieval tasks". On its recall-heavy benchmarks, the hybrids beat both the pure recurrent models and the pure Transformer.
So the hybrid splits the work. The recurrent layers carry recency and gist at a constant cost per token. The attention layers do exact lookup over the whole context. Where the attention layer sits is a choice. KLM puts it at the top of each group of four. The ratio is a choice too: across the census, hybrids keep between one layer in ten and one in two as full attention, and the DeltaNet and KDA designs cluster at one in four.
What it saves, in bytes
Qwen3.8-27B. The census works it from the config. Each of the 16 full-attention layers has 4 KV heads of width 256, so it caches values, 4,096 bytes per token in BF16. The 16 layers together hold 64 KiB per token, and at the model's 262,144-token context that is 17.2 GB for one sequence. With all 64 layers full it would be 68.7 GB. Each Gated DeltaNet layer holds 48 value heads of a 128 by 128 state, 786,432 values, kept in FP32: 3 MiB per layer and 144 MiB for all 48, at any length. One recurrent layer's state equals one attention layer's cache at 768 tokens, and past that every token is savings.
KLM 1.6. At 99M parameters the arithmetic is the same and the crossover is shorter. Each attention layer caches 4 KV heads of 64 dimensions, keys and values, so 512 numbers per token. Each Gated DeltaNet-2 layer keeps 4 heads of 128 by 128, 65,536 numbers. One state equals one layer's cache at 128 tokens. At the full 2,048-token context the six attention layers hold 25.2 MB and the 18 recurrent states, with their convolution windows, about 5.1 MB. All 24 layers as attention would be 101 MB.
The generic version, for a model with layers, a fraction of them full attention with heads of width , and recurrent heads of :
with and the bytes per cached value and per state value. At the growing term is a quarter of an all-attention model's.
The state is fragile in precision
The fixed part is small, and it does not like being compressed. LeapQuant measured what happens when a Gated DeltaNet state is stored in BF16 instead of FP32: Qwen3.5-9B's AIME 2026 score falls from 87.9 to 72.1, and in INT8 to 3.9. That is why Qwen's config asks for an FP32 state and why KLM's reference code runs the recurrence in FP32 whatever the weights are in. The state is not written once and read; every token multiplies it and adds to it, so its rounding error compounds over thousands of steps.
Serving has its own costs. A server now manages two kinds of memory with different lifetimes, a paged KV cache and a per-sequence state, and a cached prefix can only be reused if a state snapshot was saved at that exact boundary. Prefix caching gets harder, not impossible.
Gated DeltaNet-2: splitting the gate in two
The gated delta rule uses one scalar, , for two jobs: how much old content to erase and how much new content to write. Kimi Delta Attention made the decay per channel. Gated DeltaNet-2 (Hatamizadeh, Choi and Kautz, NeurIPS 2026) splits as well. In the convention of the KLM article, with as , its Eq. 10 is
with a per-channel decay, an erase gate over key channels and a write gate over value channels. Tie both gates to the same scalar and it is KDA; make the decay a scalar too and it is Gated DeltaNet. Untied, the layer can erase an old value completely and write the new one at half strength, or forget a key without writing anything, neither of which a single can express. KLM 1.6 uses this layer; its article has a widget that runs the three moves on a 4 by 4 head with the gates on sliders, and its figure shows the block.

Where I land
The Gated DeltaNet layer earns its place because the math is honest about what a fixed memory can do. The delta rule gives it a way to edit one association, the decay gives it a way to drop many, and the chunkwise algorithm makes both trainable at matmul speed. It still cannot do exact recall over a long context, and nobody shipping it pretends otherwise: every production model built on it keeps full attention in about one layer in four, and the census found no large open model whose only long-range memory is a recurrent state.
If I were sizing a long-context deployment, the line I would watch is the one that grows: the attention quarter. The recurrent three quarters are cheap and constant, as long as their state stays in FP32. The Mamba entry covers the selective-scan side of the same family, and Kimi K3 shows the per-channel decay at 2.78T parameters.