Sliding-window attention (local + global) · 2020 · Attention · 10 min
- local-attention
- kv-cache
- attention-sink
- hybrid
- attention
- long-context
- explainer
A 1:53 narrated explainer, drawn in code. Every number and picture in it is this page's own; the sources are below.
› transcript
Hi, I'm Quenby! The cheapest way to stop attention growing is to stop it looking far back. A normal layer lets each query see every earlier key. A growing triangle. A local layer keeps only a band, the last few keys. Every query reads the same amount. Many models also keep the first token visible to everyone. More on that soon. One window layer sees a window back. Stack another, and each position already summarises the window before it. Mistral's paper puts it at about one hundred and thirty-one thousand tokens, with a window of four thousand and ninety-six. In theory. Longformer drew the band in twenty twenty. And added a few global positions. Today's models add whole global layers instead. Window attention fails when the text length surpasses the cache size. A softmax has to put its weight somewhere, even when nothing matters. Trained models park it on the first tokens, which every query can see. Slide them out of the window, and that weight lands anywhere. So keep the first four tokens forever. StreamingLLM ran four million tokens that way. Modern models interleave. MiMo stacks five window layers of just a hundred and twenty-eight tokens under one global layer, eight times. The global layers carry every long-range lookup. The windows carry everything local, at a fixed cost. Gemma 4 does it one in six. Fifty window layers hold under a gigabyte. Made full, the same model would need two hundred and twenty-five. A window caps memory by position. A global layer every few reads the whole past. A band, a sink, and a global layer every few. Thanks for watching! Every source is in the full article. I'm Quenby. Bye!
A local layer attends over a fixed window of recent keys, so its cache stops growing once the context passes W tokens. Stacking windows widens what the top of the stack can reach, and a global layer every few layers reads the whole past. Some models also keep the first tokens visible to every query as an attention sink.
The cheapest way to stop an attention layer growing with the context is to forbid it from looking far back. Give each query a window of the last keys and the layer's cost per token is fixed, its cache tops out at entries, and a rolling buffer can hold it. That idea is older than most of the models using it: Longformer built a long-document Transformer on it in 2020. What changed since is the recipe around the window. Nobody ships a model that is windows all the way up. They interleave window layers with layers that see everything, and most add a sink.
The full-attention census lists sliding windows as one of the six ways current open models mix tokens, in Gemma, gpt-oss and MiMo. This page derives what a window buys, where it breaks, and how the hybrids are put together.
A band instead of a triangle
A causal attention layer lets query score every key : a lower triangle. A sliding window keeps only the band near the diagonal,
so each query reads at most keys. Longformer states the cost: for a sequence of tokens and window , linear in length. Its window was bidirectional, tokens on each side, because it was an encoder; decoders use the causal half.
The cache follows the band. Mistral 7B stores the keys and values for position at slot of a fixed buffer, so once passes the oldest entry is overwritten and "the size of the cache stops increasing". On a 32K-token sequence, the paper says, that cut cache memory 8 times without hurting quality.

Reach compounds with depth
One window layer sees tokens back. Two stacked layers see : position in layer 2 reads positions down to in layer 1, each of which already holds a summary of the before it. Mistral's paper writes it as a recursion: the hidden state at position of layer "can access tokens from the input layer at a distance of up to tokens". With that is a theoretical span of about 131K tokens at the last layer. Longformer gives the same rule, for layers, and a dilated window with gaps of stretches it to .
"Theoretical" is doing work in that sentence. Information from 100K tokens back reaches the top only after being folded into a fixed-width hidden state about 25 times, once per hop. It is the same compression a recurrent layer applies, done in a deep stack rather than in time. A fact that needs to be read verbatim from far back does not survive that reliably, which is why every modern recipe adds global layers.
Why pure window attention collapses, and the sink
The sharper failure was found in 2023. StreamingLLM took ordinary pretrained models and served them with a rolling window cache, evicting the oldest keys. Its abstract: window attention "fails when the text length surpasses the cache size". Perplexity explodes the moment the first tokens leave the cache.
The reason is the softmax. Its weights must sum to one, and when a query has nothing it particularly needs, it still has to put the mass somewhere. Trained models learn to dump that mass on the first few tokens, whatever they contain, because those tokens are visible to every query. StreamingLLM calls them attention sinks. Evict them and the leftover mass lands on arbitrary recent tokens, and the layer's outputs shift.
The fix is to never evict them. Keep the first four tokens' keys and values plus the recent window, and the paper reports stable language modelling on Llama-2, MPT, Falcon and Pythia "with up to 4 million tokens and more", with up to a 22.2 times speedup over recomputing a sliding window. One or two initial tokens were not enough for those models; four were. It also found that a model pretrained with one dedicated, learnable sink token at the start needs only that one.
Models designed for windows now build the sink in. MiMo-V2-Flash adds a learnable attention-sink bias, which its report credits with much of the hybrid's stability. Naive-N0.5-Flash, built on MiMo, keeps a sink column in its window layers only (add_swa_attention_sink_bias: true). The diagram above draws it as the first column, visible to every query in the local layer.
Interleaving local and global
The modern pattern is a block of window layers capped by one layer with full causal attention. The global layers carry long-range retrieval; the local layers carry everything that does not need it, at a fixed cost.
Gemma 2 (paper) alternates the two "in every other layer", with a 4,096-token window on the local layers and an 8,192-token span on the global ones. At Gemma 2's context length the window is half the context, so the saving is modest. The pattern was the point.
gpt-oss also uses one full layer in two, which the census found to be the most attention-heavy end of the hybrid range.
Gemma 4 pushed the ratio to one global layer in six. The Gemma 4 article and the census work through the 31B model: 10 global layers whose keys double as values, 4 heads of 512, and 50 window layers with 16 KV heads of 256 and a 1,024-token window.
MiMo-V2-Flash went furthest on the window. Its 48 layers are 8 hybrid blocks of 5 sliding-window layers and 1 global layer, and each window layer sees only the previous 128 tokens. The report frames the small window as an inductive bias: "smaller windows force the model to focus on local context". The window layers use 64 query and 8 KV heads, the global layers 64 and 4.

Llama 4 took a related route with chunked attention, where a query sees only its own fixed 8,192-token chunk rather than a sliding window; the census files both under the same kind.
The cache, worked out
For a model with layers, of them global, and the rest windowed at , the per-sequence KV cache at context is
with bytes per value (the factor 2 is keys and values; Gemma 4's global layers share them, so it drops there). Past the second term is a constant and the first is the only one that grows. That is the whole economics: the global fraction sets the slope.
The census puts numbers on Gemma 4 31B at 262,144 tokens. The 10 global layers keep 10.7 GB and the 50 window layers 0.84 GB. With every layer full at its own width the same sequence would need 225 GB, because the window layers are the wide ones: 16 KV heads against 4. A window does the most good on the layers with the most heads.
MiMo's report claims "nearly a 6× reduction in KV-cache storage and attention computation for long contexts", which is the order the formula gives when one layer in six is global and the window is tiny next to the context.
Prefill, one window at a time
Decoding is where the ring buffer shines, but a window also changes how a long prompt is processed. Mistral's paper describes it: split the prompt into chunks, use the window size as the chunk size, and fill the cache chunk by chunk. Each new chunk attends to itself with a causal mask and to the cached chunk before it through the window, and never further back, because anything older is outside every query's reach. So prefill memory is bounded by two windows of keys, whatever the prompt length, and the work is a sequence of fixed-size blocks.
The paper also reports what the mask is worth in practice: on a 16K-token sequence with , its changes to FlashAttention and xFormers gave a 2 times speedup over a vanilla attention baseline. In a hybrid, the global layers still prefill quadratically, so the speedup applies only to the window share of the stack. A model with one global layer in six has five sixths of its attention work bounded and one sixth not, and at very long prompts that sixth is most of the time.
What a window forgets
Every kind of efficient layer gives up exact access to some of the past. A window gives it up by position. The census says it of MiMo directly: five layers of six "cannot see anything older than a paragraph; the global layers carry all long-range retrieval". A recurrent state forgets by capacity instead, and an index-selected layer forgets by ranking. Those are different failure modes, and they show up on different tasks: a window layer is perfect inside and blind outside it, where a recurrent layer degrades everywhere a little.
That predictability is a real advantage. MiMo-V2-Flash, with five layers in six blind beyond 128 tokens, reports 96.7% needle retrieval at 256K tokens, because its 8 global layers are full attention and can find the needle wherever it is. The risk sits in tasks that need many far-apart facts combined, where only one layer in six can gather them.
The newest designs keep the windows and replace the global layer. Naive-N0.5-Flash takes MiMo's skeleton and swaps each global layer for index-selected sparse attention: 39 window layers and 9 DSA layers, no dense layer at all. At 1M tokens a decoded token reads 23,424 backbone keys per step, a number that stops growing once the context passes 2,048.
Where I land
Sliding windows are the least clever efficient attention and the most predictable. The cost is exactly bounded, the kernel is ordinary attention with a mask, the cache is a ring buffer, and the failure mode is easy to state: beyond , a local layer sees nothing. The sink was the one surprise, and it is now a line in the config.
In a new model I would look at the global fraction and at what fills the global slot before I looked at the window size. One in two is cautious, and one in six is where Gemma 4 and MiMo landed. The move I find most promising is the one Naive made: keep the windows, make the global layer sparse, and the whole stack reads a fixed number of keys per token at any length. Gated DeltaNet hybrids make the opposite trade, a recurrent state where the window would be, and the KV-cache entry covers the head-sharing that every one of these designs layers on top.