~/satyajit

architectures / ssm / rnn

Gated DeltaNet hybrids: a fixed memory on three layers in four

mdjsonmcp

Gated DeltaNet hybrid (3 linear : 1 full) · 2024 · SSM / RNN · 12 min

  • linear-attention
  • delta-rule
  • hybrid
  • fixed-state
  • hybrid-attention
  • explainer

A 2:13 narrated explainer, drawn in code. Every number and picture in it is this page's own; the sources are below.

› transcript

A memory the same size at token ten and at token one million. What does it have to forget? Hi, I'm Ripley! This is the layer that fills three slots in four of Qwen's recent models. Sixty-four layers. Forty-eight keep a fixed memory. Sixteen are ordinary attention, and their cache grows with every token. Each head keeps one matrix, the state. Every token edits it. Alpha decays the whole state. Near zero, it wipes the memory. Then it reads what the state holds for this key, and takes it out. And writes the new value in its place. Beta sets how hard. Follow one token through one head. It brings a query, a key, a value and two gates. First the decay. Then the erase. A repeated key replaces its value instead of blending with it. Then the write. The query reads the result with one product. Same cost at any length. The paper's block. Short convolutions on the query, key and value, a norm on the keys, and a gate on the output. Its hybrids paired the layer with windowed attention. Production models chose full attention instead. So a block is three recurrent layers and one attention layer, sixteen times over. Only the attention layer keeps a cache that grows. It does the exact lookups a summary cannot. One layer in four stays global, because a finite state cannot reproduce exact token-level retrieval no matter how well it is gated. At a quarter of a million tokens, the sixteen attention layers hold seventeen gigabytes. All sixty-four would hold almost seventy. The fixed part is small, and fragile. Store it in sixteen-bit floats and the score falls fifteen points. Three layers keep a cheap memory they can edit. The fourth keeps everything, for exact recall. Decay, erase, write, and one attention layer in four. Thanks for watching! Every source is in the full article. I'm Ripley. Bye!

gated deltanet hybrid · three recurrent layers, then one that attends in full
one block, read bottom to topfull attentionKV cache growsGated DeltaNetfixed stateGated DeltaNetfixed stateGated DeltaNetfixed statehidden state from the block belowto the next block×None Gated DeltaNet head, one tokenSₜ = αₜ (I − βₜ kₜkₜᵀ) Sₜ₋₁ + βₜ kₜvₜᵀSₜ₋₁dₖ × dᵥ per headdecay× αₜ (0 to 1)erase along kₜ− βₜ kₜ (kₜᵀ S)write along kₜ+ βₜ kₜ vₜᵀSₜreadSₜᵀqₜthe row along kₜ now holds vₜtoken xₜ→ qₜ, kₜ, vₜ, αₜ, βₜfixed memory on three layers in four · exact lookup left to the fourth

A Gated DeltaNet layer keeps a matrix of fixed size per head instead of a cache. Each token first shrinks it by a decay α, then reads what the matrix already stores for its key and replaces it with the new value, at a strength β the token picks. Three such layers share each block with one full-attention layer, whose cache still grows and does the exact retrieval a summary cannot.

Open a recent Qwen config.json and the layer_types list reads like a drum pattern: three linear_attention, one full_attention, repeated. Qwen3.8-27B has 64 layers, 48 of them Gated DeltaNet and 16 full attention. KLM 1.6, a 99M-parameter base model, puts attention at layers 3, 7, 11, 15, 19 and 23 and Gated DeltaNet-2 everywhere else. The full-attention census found that most DeltaNet and KDA designs sit at exactly that 3 to 1 ratio.

This page builds the layer that fills three slots in four, from plain linear attention up, then explains why the fourth slot is still softmax attention. The source is the Gated DeltaNet paper (Yang, Kautz and Hatamizadeh, from MIT and NVIDIA); the numbers on real models come from this site's pieces on KLM 1.6, the census and the linear-attention roundup.

A cache that grows, and a state that doesn't

Softmax attention keeps every key and value it has seen. For a query qtq_t it computes

ot=∑i≤texp⁡(qt⊤ki)∑j≤texp⁡(qt⊤kj) vio_t = \sum_{i \le t} \frac{\exp(q_t^\top k_i)}{\sum_{j \le t} \exp(q_t^\top k_j)}\, v_i

and the softmax couples every term to every other, so nothing can be summarised in advance. Decoding token tt reads tt cached entries per layer. At long context that read, and the memory holding it, is the cost.

Drop the softmax and the sum factorises. Following the paper's convention, with the state SS a dv×dkd_v \times d_k matrix:

St=St−1+vtkt⊤,ot=St qt=∑i≤tvi (ki⊤qt)S_t = S_{t-1} + v_t k_t^\top, \qquad o_t = S_t\, q_t = \sum_{i \le t} v_i\, (k_i^\top q_t)

That is linear attention. The state is the same size at token 10 and at token 1,000,000: one dv×dkd_v \times d_k matrix per head. Reading it is one matrix-vector product. In the paper's words, it "underperforms Transformers in language modeling by a large margin", and the reason is visible in the update. Every token adds an outer product and nothing ever leaves. Write two values at overlapping keys and every later read returns a blend of both.

Two repairs: forget, and overwrite

The first repair is a decay. Mamba-2, written as linear attention, multiplies the state by a data-dependent scalar before each write:

St=αtSt−1+vtkt⊤,αt∈(0,1)S_t = \alpha_t S_{t-1} + v_t k_t^\top, \qquad \alpha_t \in (0, 1)

The model can now clear its memory when the topic changes. It cannot clear one association without fading all of them; the decay is global.

The second repair is the delta rule. Treat SS as a memory that maps keys to values. Before writing, read what it currently returns for ktk_t, call that vtold=St−1ktv_t^{\text{old}} = S_{t-1} k_t, take it out, and put a blend of the new and old value back:

St=St−1−(St−1kt) kt⊤+(βtvt+(1−βt)St−1kt) kt⊤=St−1(I−βtktkt⊤)+βtvtkt⊤S_t = S_{t-1} - (S_{t-1} k_t)\, k_t^\top + \big(\beta_t v_t + (1 - \beta_t) S_{t-1} k_t\big)\, k_t^\top = S_{t-1}\big(I - \beta_t k_t k_t^\top\big) + \beta_t v_t k_t^\top

βt∈(0,1)\beta_t \in (0, 1) is the writing strength. With unit-length keys and βt=1\beta_t = 1, the slot along ktk_t is replaced exactly. Associations along orthogonal keys are untouched. The transition I−βtktkt⊤I - \beta_t k_t k_t^\top is a generalised Householder matrix: it shrinks the state along one direction and leaves the rest alone.

The paper reads this as optimisation. The delta rule is one step of gradient descent on the regression loss 12∥Skt−vt∥2\tfrac{1}{2}\lVert S k_t - v_t \rVert^2, with learning rate βt\beta_t:

St=St−1−βt(St−1kt−vt)kt⊤S_{t} = S_{t-1} - \beta_t (S_{t-1} k_t - v_t) k_t^\top

So a delta-rule layer is an online regressor fitted to the key-value pairs it has seen, and its state is the regressor's weights. The linear-attention roundup builds its whole comparison on that reading.

The gated delta rule

Gated DeltaNet puts the two repairs together. Its Eq. 10 is

St=St−1(αt(I−βtktkt⊤))+βtvtkt⊤S_t = S_{t-1}\Big(\alpha_t \big(I - \beta_t k_t k_t^\top\big)\Big) + \beta_t v_t k_t^\top

Both gates are computed from the token. When αt→0\alpha_t \to 0 the memory is wiped in one step. When αt→1\alpha_t \to 1 the layer is a pure delta rule and edits one association without touching the others. In optimisation terms, αt\alpha_t is an adaptive weight decay on the online regressor.

Per head and per token, the work is three moves on a dv×dkd_v \times d_k matrix:

  1. Decay: multiply the state by αt\alpha_t.
  2. Erase: read SktS k_t and subtract βt(Skt)kt⊤\beta_t (S k_t) k_t^\top.
  3. Write: add βtvtkt⊤\beta_t v_t k_t^\top.

Then the query reads it, ot=Stqto_t = S_t q_t. The diagram above draws those three moves on a 4 by 4 head. KLM's reference code is the same loop written out one token at a time in FP32: multiply the state by the decay, read the erase term, add the corrected write.

The paper's single-needle results (its Table 2, 1.3B models trained on 100B tokens) show why both repairs matter. On S-NIAH-1, a pass-key buried in repeated filler, Mamba-2 falls from 99.2 at 1K tokens to 30.4 at 8K because its decay forgets too fast, while DeltaNet holds 98.8. On S-NIAH-2, a number hidden in real essays, DeltaNet drops to 14.4 at 8K because it never clears anything and the needle collides with everything else, while Gated DeltaNet keeps 92.2 at 4K. The authors summarise it as three observations: decay hurts retention, gating helps filtering, and the delta rule helps memorisation.

Inside the layer

The block around the recurrence is close to a Llama layer with the attention swapped out. Queries, keys and values each come from a linear projection, a short causal convolution and SiLU; queries and keys are then L2-normalised, which keeps the Householder transition stable. α\alpha and β\beta come from their own linear projections, α\alpha in Mamba-2's parameterisation. The output is normalised and multiplied by a gate before the output projection. The paper's figure shows that block next to its two hybrid layouts:

Three panels. Gated DeltaNet-H1: a block repeated N times with Gated DeltaNet then MLP, then sliding-window attention then MLP, each with a residual add. Gated DeltaNet-H2: Mamba2 then MLP, Gated DeltaNet then MLP, sliding-window attention then MLP. Block design: q and k pass through a linear projection, a short convolution, SiLU and L2 norm; v through a linear projection, convolution and SiLU; alpha and beta through linear projections; all feed the Gated Delta Rule, whose output is normalised, multiplied by a gate from its own linear projection, and projected.
The paper's two hybrids and its layer: H1 alternates Gated DeltaNet with sliding-window attention, H2 stacks Mamba2, Gated DeltaNet and sliding-window attention (Gated DeltaNet paper, Figure 1).

Production models kept the layer and changed the hybrid. The paper's H1 alternates one Gated DeltaNet layer with one sliding-window attention layer. Qwen and KLM put three recurrent layers under one full-attention layer that sees the whole context.

Training it in parallel

Written as a recurrence, the layer is a loop over tokens, which a GPU hates. Written fully in parallel, as a masked L×LL \times L matrix, it is quadratic again. The paper trains it chunkwise: split the sequence into chunks of CC tokens, run the recurrence only across chunk boundaries, and do everything inside a chunk with matrix multiplies.

The hard part is the product of Householder transitions inside a chunk. The paper extends the WY representation that DeltaNet's parallel algorithm used (Yang et al., 2024) to carry the decay. A product of rr transitions collapses to I−∑iwiki⊤I - \sum_i w_i k_i^\top, and the vectors ww and their value-side twins come from one triangular solve over the chunk:

U~[t]=[I+strictLower⁡(diag⁡(β[t]) (Γ[t]⊙K[t]K[t]⊤))]−1diag⁡(β[t]) V[t]\widetilde{U}_{[t]} = \Big[I + \operatorname{strictLower}\big(\operatorname{diag}(\beta_{[t]})\,(\Gamma_{[t]} \odot K_{[t]} K_{[t]}^\top)\big)\Big]^{-1} \operatorname{diag}(\beta_{[t]})\, V_{[t]}

where Γ[t]\Gamma_{[t]} holds the ratios of cumulative decays inside the chunk. After that, the chunk's output and the state handed to the next chunk are a few dense matmuls. The paper's throughput comparison puts Gated DeltaNet at essentially DeltaNet's speed and slightly behind Mamba-2, whose scalar transition is cheaper than a Householder one. The same chunkwise shape is what Kimi K3's KDA layers and Mamba-2's SSD use.

Why one layer in four stays attention

A fixed state is a lossy summary. The roundup's toy recall task puts a number on it: the delta rule recalls 96.9% of 8 stored pairs and 49.2% of 32. A softmax layer recalls whatever it was given, because it kept all of it. The team behind Qwen3.8-Flash-Next gives the design rule plainly: one layer in four stays global "because a finite state cannot reproduce exact token-level retrieval no matter how well it is gated".

The paper says the same about linear models in general. They "have limitations in modeling local shifts and comparisons, and their fixed state size makes it hard for retrieval tasks". On its recall-heavy benchmarks, the hybrids beat both the pure recurrent models and the pure Transformer.

So the hybrid splits the work. The recurrent layers carry recency and gist at a constant cost per token. The attention layers do exact lookup over the whole context. Where the attention layer sits is a choice. KLM puts it at the top of each group of four. The ratio is a choice too: across the census, hybrids keep between one layer in ten and one in two as full attention, and the DeltaNet and KDA designs cluster at one in four.

What it saves, in bytes

Qwen3.8-27B. The census works it from the config. Each of the 16 full-attention layers has 4 KV heads of width 256, so it caches 2×4×256=2,0482 \times 4 \times 256 = 2{,}048 values, 4,096 bytes per token in BF16. The 16 layers together hold 64 KiB per token, and at the model's 262,144-token context that is 17.2 GB for one sequence. With all 64 layers full it would be 68.7 GB. Each Gated DeltaNet layer holds 48 value heads of a 128 by 128 state, 786,432 values, kept in FP32: 3 MiB per layer and 144 MiB for all 48, at any length. One recurrent layer's state equals one attention layer's cache at 768 tokens, and past that every token is savings.

KLM 1.6. At 99M parameters the arithmetic is the same and the crossover is shorter. Each attention layer caches 4 KV heads of 64 dimensions, keys and values, so 512 numbers per token. Each Gated DeltaNet-2 layer keeps 4 heads of 128 by 128, 65,536 numbers. One state equals one layer's cache at 128 tokens. At the full 2,048-token context the six attention layers hold 25.2 MB and the 18 recurrent states, with their convolution windows, about 5.1 MB. All 24 layers as attention would be 101 MB.

The generic version, for a model with LL layers, a fraction ff of them full attention with hkvh_{kv} heads of width dd, and recurrent heads of dk×dvd_k \times d_v:

bytes≈fL⋅2 hkv d b⋅N⏟grows with context N  +  (1−f)L⋅h dk dv bs⏟fixed\text{bytes} \approx \underbrace{f L \cdot 2\, h_{kv}\, d\, b \cdot N}_{\text{grows with context } N} \;+\; \underbrace{(1 - f) L \cdot h\, d_k\, d_v\, b_s}_{\text{fixed}}

with bb and bsb_s the bytes per cached value and per state value. At f=1/4f = 1/4 the growing term is a quarter of an all-attention model's.

The state is fragile in precision

The fixed part is small, and it does not like being compressed. LeapQuant measured what happens when a Gated DeltaNet state is stored in BF16 instead of FP32: Qwen3.5-9B's AIME 2026 score falls from 87.9 to 72.1, and in INT8 to 3.9. That is why Qwen's config asks for an FP32 state and why KLM's reference code runs the recurrence in FP32 whatever the weights are in. The state is not written once and read; every token multiplies it and adds to it, so its rounding error compounds over thousands of steps.

Serving has its own costs. A server now manages two kinds of memory with different lifetimes, a paged KV cache and a per-sequence state, and a cached prefix can only be reused if a state snapshot was saved at that exact boundary. Prefix caching gets harder, not impossible.

Gated DeltaNet-2: splitting the gate in two

The gated delta rule uses one scalar, βt\beta_t, for two jobs: how much old content to erase and how much new content to write. Kimi Delta Attention made the decay per channel. Gated DeltaNet-2 (Hatamizadeh, Choi and Kautz, NeurIPS 2026) splits β\beta as well. In the convention of the KLM article, with SS as dk×dvd_k \times d_v, its Eq. 10 is

St=(I−kt(bt⊙kt)⊤) Dt St−1+kt(wt⊙vt)⊤S_t = \big(I - k_t (b_t \odot k_t)^\top\big)\, D_t\, S_{t-1} + k_t (w_t \odot v_t)^\top

with Dt=Diag(αt)D_t = \mathrm{Diag}(\alpha_t) a per-channel decay, btb_t an erase gate over key channels and wtw_t a write gate over value channels. Tie both gates to the same scalar and it is KDA; make the decay a scalar too and it is Gated DeltaNet. Untied, the layer can erase an old value completely and write the new one at half strength, or forget a key without writing anything, neither of which a single β\beta can express. KLM 1.6 uses this layer; its article has a widget that runs the three moves on a 4 by 4 head with the gates on sliders, and its figure shows the block.

Left: the Gated DeltaNet-2 paper's hybrid block, repeated N times: Gated DeltaNet-2 then MLP, then sliding-window attention then MLP, with residual additions. Right: the block design, where q and k go through a linear projection, short convolution, SiLU and L2 norm, v through a projection, convolution and SiLU, and alpha, b and w come from separate linear projections, b and w through sigmoids, all feeding the Gated Delta Rule-2 operator, whose output is normalised, multiplied by a gate and projected.
The same block one generation on: separate erase and write gates, b and w, feed the Gated Delta Rule-2 operator (Gated DeltaNet-2 paper, Figure 1).

Where I land

The Gated DeltaNet layer earns its place because the math is honest about what a fixed memory can do. The delta rule gives it a way to edit one association, the decay gives it a way to drop many, and the chunkwise algorithm makes both trainable at matmul speed. It still cannot do exact recall over a long context, and nobody shipping it pretends otherwise: every production model built on it keeps full attention in about one layer in four, and the census found no large open model whose only long-range memory is a recurrent state.

If I were sizing a long-context deployment, the line I would watch is the one that grows: the attention quarter. The recurrent three quarters are cheap and constant, as long as their state stays in FP32. The Mamba entry covers the selective-scan side of the same family, and Kimi K3 shows the per-channel decay at 2.78T parameters.

share