Full attention in open LLMs: a 180-config recount of the chart that says it is gone

On 12 November @crazyMLguy posted a chart under the headline "Full Attention Is Disappearing From Open LLMs!" The numbers in the post:

2025: 59% of models used full attention throughout. Jan–Aug 2026: Only 27% did. Aug–Nov 2026: 0% used full attention throughout; 48% used no full-attention layers.

The post names three examples, Mistral Small 4, Kimi K3 and MiniMax M3.1 Flash, and gives the reason in one line: "Less attention-related memory traffic and better efficiency for long-context inference."

expand

The post gives no dataset. The picture gives it away. The three diagrams are watermarked "Sebastian Raschka", and the bar labels say 32, 60 and 21 models. Raschka's LLM Architecture Gallery lists 122 models, each card with a release date, a layer mix and a link to its config.json. Counting its cards by date gives exactly 32 for 2025 and 60 for 1 January to 25 August 2026. The third window holds 22 cards to 5 November, and the gallery's changelog shows the 22nd, ERNIE 5.5 Flash Base, added on 5 November itself. So the chart is the gallery, drawn before ERNIE went in. I could not find where Raschka first published it: the gallery page, its changelog and the public pages of its six source articles do not contain the chart's labels.

A count like this is only as good as two choices: which models are on the list, and what counts as "full attention". I checked both by going below the gallery cards to the configs.

What a layer has to do to count as full

Every decoder layer mixes information across positions somehow. The census sorts each of those token-mixing layers into one of six kinds.

• Full attention. Each new token's query scores every earlier key with a softmax: MHA, GQA, MQA, and MLA, which compresses keys and values into a latent but still keeps one per token and reads all of them.
• Sliding window or chunked. The query sees only the last $W$ tokens (Gemma, gpt-oss, MiMo) or its own fixed chunk (Llama 4's 8,192-token chunks).
• Linear or recurrent. The past is folded into a fixed-size state: Gated DeltaNet, Kimi Delta Attention, Mamba-2, lightning attention, mLSTM.
• Short convolution. Liquid's LFM blocks mix a few neighbouring tokens and nothing else.
• Index-selected sparse. A small indexer scores the past and the real attention reads only the top $k$ tokens or blocks: DeepSeek Sparse Attention (DSA), MiniMax Sparse Attention (MSA), Qwen Sparse Attention, LongCat's LSA, DeepSeek's CSA.
• Compressed. DeepSeek V4's heavily compressed attention (HCA) attends densely, but over a sequence squeezed 128 to 1.

What each costs is easiest to see per layer. A full layer with $h_{kv}$ key-value heads of width $d$ stores, for every token in the context,

with $b$ bytes per value, and each decoded token reads all $N$ of those entries. A window layer stores and reads at most $W$ of them. A recurrent layer stores a state whose size has no $N$ in it at all. An index-selected layer stores all $N$ entries plus a small index key per token, reads the index keys of all $N$ and the full entries of only $k$. So "sparse" and "linear" are different bargains: one keeps everything and reads little, the other keeps little.

The chart's three classes follow from the per-layer kinds. A model is all-full if every token-mixing layer is full attention, hybrid if some are, no-full if none are. The open question is the sparse kinds. A DSA layer can attend to any earlier token, and below 2,048 tokens of context it attends to all of them. It is full attention with a gate in front. The chart counts it as not full: its 2025 bar has 2 "no full attention" models out of 32, and the gallery has exactly two 2025 cards without a dense full layer, xLSTM and DeepSeek V3.2, whose 61 layers are all DSA. I keep that convention so the numbers compare, and report the other one next to it.

Reading 180 configs

The gallery's layer mixes are hand-written summaries. I wanted the configs. For each card dated 2025 or later I fetched the linked config.json from Hugging Face (no weights), and for every model I classified each layer from whatever the config uses to describe the stack. There is no common field. Gemma, gpt-oss, OLMo and Qwen3.5 write a layer_types list; Nemotron-H writes a pattern string where M is Mamba-2 and * is attention; Kimi lists kda_layers and full_attn_layers, 1-based; MiniMax M3 has a sparse_attention_freq list; DeepSeek V4 encodes each layer as a compress_ratios entry, 0 for window-only, 4 for CSA and 128 for HCA. The core of the classifier, condensed:

# condensed from the census script; one branch per config dialect
if t.get("layer_types"):                       # Gemma, gpt-oss, OLMo, Qwen3.5, Mellum, ...
    codes = [{"full_attention": "F", "sliding_attention": "S",
              "linear_attention": "L", "conv": "C",
              "deepseek_sparse_attention": "P"}[x] for x in t["layer_types"]]
elif t.get("hybrid_override_pattern"):         # Nemotron-H: M = Mamba-2, * = attention
    codes = ["L" if c == "M" else "F" for c in t["hybrid_override_pattern"] if c in "M*"]
elif "sparse_attention_config" in t:           # MiniMax M3 / M3.1 Flash
    codes = ["P" if x else "F" for x in t["sparse_attention_config"]["sparse_attention_freq"]]
elif "compress_ratios" in t:                   # DeepSeek V4: 0 window, 4 CSA, 128 HCA
    codes = [{0: "S", 4: "P", 128: "H"}[x] for x in t["compress_ratios"][:n_layers]]

Where the config gives only a rule, I read the modeling code. Motif 3's modeling_motif.py makes a layer full when layer_idx % period != 0 is false, so with period 4 it is 14 full and 39 window layers of 53. Phi-4-mini-flash-reasoning's modeling_phi4flash.py builds a SambaY stack: Mamba and 512-token window layers, one global layer at index 17, and seven cross-attention layers that reuse that one layer's keys and values. Llama-3.1-Nemotron-Ultra-253B is a pruned stack in which 98 of 162 blocks have no attention at all; its other 64 are full. Falcon-H1 runs attention and Mamba-2 side by side in every layer, so it counts as all-full. Draft and MTP layers, vision towers and FFN-only blocks are not counted.

Two gallery models have gated configs that I could not read, Soofi S and Antares 1B, and for those the census uses the gallery's own layer mix and says so in the table. For the Ling family the config gives a group size (one MLA layer per group of 4, 6 or 8) but not where in the group it sits, so those rows carry counts and no layer order.

On the 115 gallery cards the configs and the cards agree on every model's class; I parsed each card's layer mix and attention line and compared. Comparing each card's layer numbers with the config's turned up two real disagreements, and neither moves a model between classes. MiMo-V2-Flash's hybrid_layer_pattern has nine global layers of 48, where the card says eight. Step 3.5 Flash has 12 full and 33 window layers in its 45 num_hidden_layers; the card's 36 window layers include three MTP entries from the end of layer_types. The DeepSeek V4 cards list every layer as CSA/HCA, where compress_ratios makes V4-Flash's first two layers window-only; that is a summary, not an error.

The chart reproduces exactly

Classified from configs, the gallery's cards in the chart's three windows give:

Those are the chart's bars to the rounding: in the gallery, 59.4% of the 2025 models, 26.7% of those from January to August 2026 and none of the last 21 use full attention in every layer. The chart is an honest picture of the gallery. The question is whether the gallery is a picture of open LLMs.

A wider list, and the 0% goes

The gallery is curated for architectures worth drawing. That is its job, and it skews the count in two ways. It counts sizes of one family as separate models: 2025 has six Qwen3 cards, all full attention, and they are nearly a third of that year's 19 all-full models. And its late-2026 window is mostly large releases with new attention, because those are the ones worth a new diagram.

So I built a second list. The spine is the gallery's 115 cards from 2025 on. To it I added 65 open-weight releases from this site's own model table and coverage that the gallery does not have, each with a public config and a text decoder. I usually took one size per family, but kept sizes that were announced as separate models (K2 Horizon's three) or that the site covers on their own (Llama 4 Scout beside the gallery's Maverick). That brings in, among others, MiniMax-M1, Nemotron-H 56B, Nemotron Nano 9B v2, Granite 4.0 H Small, Command A, EXAONE 4.0 and 4.5, K-EXAONE, GLM-5.3, DeepSeek V3.1, Kimi K2 Thinking, Qwen3.8's 2.4T model, LongCat 2.0, A.X-K2, Inkling Small, IQuest-Q1, the K2 Horizon models, Nemotron 3.5 Bolt and Raon-1 nano. Then I took out 13 rows whose architecture is another release's, unchanged: third-party fine-tunes where the card's base_model or a site article says so (INTELLECT-3, BTL-3, Arete, Ember-1, Atria Dawn, Nex-N2.6 and others), and preview-to-final or dated re-uploads whose configs are byte-identical (Hy3, Motif 3, DeepSeek V4 Pro 0813, DeepSeek V4.1 Flash 1017). AI21's Jamba 1.7 is missing because both its repos are gated. Release dates are the gallery's for its cards and the Hugging Face repo creation date for the rest, which can run a little ahead of the public release.

That leaves 167 releases. Same classes, same windows, except that the last one runs to today:

The first two rows barely move. The third does. Five releases since 26 August use plain global attention in every layer:

• K2 Horizon 375B-A23B, 7B and MoVA 36B-A4B from MBZUAI's IFM. All three configs set sliding_window to null, and the 375B repo's modeling_k2_horizon.py builds the mask with create_causal_mask in that case, so every layer is ordinary causal GQA. The MoVA variant's novelty is routed value experts inside attention, not a cheaper mask.
• Nanbeige4.3-33B, a 52-layer LlamaForCausalLM.
• MiniCPM5-2B, a 42-layer LlamaForCausalLM.

None of them is on the gallery. Neither are most of my other additions to that window: IQuest-Q1 (25 full layers of 88), Inkling 2 Small, Ripple-65B, Nemotron 3.5 Bolt, Raon-1 nano, LFM2.5-24B-A2B, Mellum2.1, Spark-X2.6 and Intern-S2, all hybrids. The gallery's 0% is real for the gallery and false for open LLMs.

The direction survives every way I cut it. All-full falls from 60.3% to 24.6% to 14.3%, and the average model's share of full layers falls from 69.0% to 26.0%. Collapsing each lab to one row per class per window, so that six Qwen3 sizes or three K2 Horizons count once, gives 54.3%, 29.4% and 10.3% all-full over 98 rows. Weighting by total parameters makes the last window look like the chart again: 3.2% all-full and 62.9% no-full, because the all-full survivors are small except K2 Horizon 375B and the no-full models are the giants (Kimi K3.1 Pro at 2.8T, LongCat-2.5 at 1.6T, Mistral Large 4 at 1T). Weighting by last month's downloads gives 88.0% no-full in that window, which mostly says that GLM-5.3-Flash and Qwen3.8-Flash-Next are popular; three repos carry most of the weight, so I would not lean on that cut.

One boundary case: GLM-5.3's repo was created on 25 August, the last day of the middle window, and Z.ai had promised the weights two weeks after its 14 August launch, so they may have gone public inside the last window. Moved into the last window, it makes no-full 12 of 36, 33.3%. G9v3-39A5B is the other row I could not pin: its repo dates from 21 July and Artificial Analysis lists it for August. If it belongs in the last window, all-full there becomes 6 of 36.

The widget recomputes the bars from any slice. Pick the chart's list and the bars match the image; pick the census and filter by size, and the "under 40B" slice of the last window has four all-full models out of 11, while "over 300B" has one out of 18. Full attention is not gone. It has retreated to small models and to one lab's large one.

"No full attention" mostly means "full attention with an indexer"

The chart's last bar says 48% of models use no full-attention layers. Look at what is in it. In my census the window from 26 August holds eleven no-full models, and every one of them keeps layers that can reach any earlier token:

• Index-selected sparse in every softmax layer: Hy4-preview (78 DSA layers), MiniMax M3.1 Flash (60 MSA), Mistral Large 4 (72 indexed), LongCat-2.5 (76 LSA sublayers).
• Linear layers plus index-selected sparse: GLM-5.3-Flash (34 KDA, 11 DSA), Qwen3.8-Flash-Next (36 Gated DeltaNet, 12 QSA), Kimi K3.1 Pro (69 KDA, 24 DSA).
• Windows plus index-selected sparse: Step-5 Preview (69 window, 23 sparse), Naive-N0.5-Flash (39 window, 9 DSA), DeepSeek V4.1 Flash (2 window-only, 38 CSA2).
• Compressed plus sparse: ERNIE 5.5 Flash Base (37 HCA, 6 DSA).

The selection budget is 2,048 in most of these configs and 512 in two, DeepSeek V4.1 Flash (index_topk) and Step-5 Preview (topk, over 8-token regions).

Count any layer that can read every earlier token (full, index-selected or compressed) as "global", and no 2026 release in the census lacks one. In 2025 exactly one model did: xLSTM 7B, all mLSTM. Nobody has shipped a large open model whose only memory of the far past is a recurrent state or a window, and the eleven models above did not drop global attention. They put an indexer in front of it.

Two releases in the chart's own examples show the move in a single step. Kimi K3 has 69 KDA layers and 24 Gated MLA layers, a hybrid. Kimi K3.1 Pro keeps every tensor of K3 and adds a DeepSeek-style indexer to those 24 MLA layers, with index_topk 2,048; in the census that flips it from hybrid to no-full. MiniMax M3 shipped MSA in 57 of 60 layers with three full ones at the bottom; M3.1 Flash sets all 60 entries of sparse_attention_freq to 1. Same backbone, new class.

expand

expand

The sparse convention changes the story's middle, too. Under "any global layer counts", 2026's first window has 25 of 69 models global in every layer (36.2%), and the last has 10 of 35 (28.6%): the five all-full models plus five all-sparse or sparse-and-compressed ones. That is a slow drift, not a collapse.

Why labs are doing it: the memory arithmetic

The post's reason, memory traffic at long context, is right in general and wrong in one specific way. Work it through on real configs.

Qwen3.8-27B has 64 layers. Its layer_types marks 16 as full_attention, with 4 KV heads of width 256. From the formula above, one full layer keeps

and 16 of them keep 64 KiB per token. At the model's 262,144-token context that is 17.2 GB for one sequence. Make all 64 layers full with the same heads and it is 68.7 GB. The other 48 layers are Gated DeltaNet, each holding 48 value heads of a 128 by 128 state, 786,432 values. In the FP32 that the config's mamba_ssm_dtype asks for, that is 3 MiB a layer and 144 MiB for all 48, at any context length. A full layer costs 4,096 bytes per token, so a DeltaNet layer's state equals one full layer's cache at 768 tokens. Past that, every token is savings.

Windows do the same thing differently. Gemma 4 31B has 10 global layers that store keys equal to values, 4 heads of 512, so 2,048 values per token, and 50 window layers with 16 KV heads of 256 and a 1,024-token window. At 262,144 tokens the 10 global layers keep 10.7 GB and the 50 window layers 0.84 GB. With every layer full at its own width the same sequence would need 225 GB, because the window layers are the wide ones. Kimi K3 at its 1M context keeps 29.0 GB in its 24 MLA layers (576 values a token each) and 0.43 GB of KDA state, against 112.3 GB if all 93 layers were MLA.

Index-selected sparse is the exception. MiniMax M3.1 Flash keeps every key and value in all 60 layers, plus a 128-value index key per token per layer. At 1M tokens in BF16 that is 145 GB, more than the 128.8 GB the same model would keep with plain GQA, because of the index keys. What it saves is reads: a decoded token reads 2,048 tokens of KV per layer plus every index key, 16.4 GB against 128.8 GB. So the post's "less attention-related memory traffic" holds for sparse models, and "less memory" does not. For a long-context server the reads set the decode speed and the capacity sets how many sessions fit, and sparse attention helps only the first. DeepSeek is the lab attacking both: V4's compressed layers and V4.1's shared CSA2 cache bring the global KV cache to 890 bytes per token, against 48,068 for V3.2.

expand

There is a compute side as well. Full attention over a prompt of $N$ tokens is $O(N^2)$ in prefill, and an indexer that scores every earlier key is still linear per query, so quadratic over a prefill. The paper behind Latent Sparse Attention measured that bill on GLM-5.3-Flash: at half a million tokens the indexer is two thirds of a sparse layer's time, and at 10M it is 98% of prefill. Sparse attention moves the quadratic term into a cheaper unit. It does not remove it.

What gets traded away

Every layer in this census that is not full gives up exact access to some of the past. The kinds give it up differently.

A window forgets by position. MiMo-V2-Flash's 128-token window, one global layer in six, means five layers of six cannot see anything older than a paragraph; the global layers carry all long-range retrieval. A recurrent state forgets by capacity. The linear-attention roundup measured a toy recall task: the delta rule recalls 96.9% of 8 stored pairs and 49.2% of 32. This site's piece on Qwen3.8-Flash-Next puts the design rule plainly: one layer in four stays global "because a finite state cannot reproduce exact token-level retrieval no matter how well it is gated". Every hybrid in this census that mixes full layers with windows, states or convolutions keeps between one layer in ten (Granite 4.0 H Small) and one in two (gpt-oss) as full attention, and most DeltaNet and KDA designs sit at 3 to 1. The state is fragile in another way: LeapQuant shows that storing a Gated DeltaNet state in BF16 instead of FP32 drops Qwen3.5-9B's AIME 2026 score from 87.9 to 72.1, and INT8 drops it to 3.9.

An indexer forgets by ranking. When it ranks well, the loss is small. A.X-K2 published the ablation for adding a top-2,048 indexer to gated MLA, and LongBench went from 62.80 to 62.99. The oracle experiment in the Latent Sparse Attention paper shows where it breaks:

expand

With the true top keys, Qwen3.8-27B keeps 98% of its full-attention RULER score down to $k = 512$, 0.8% of the keys at 128k, and single-needle retrieval survives to $k = 16$. What fails first is anything that needs many scattered tokens at once: multi-key needles and counting common words. A real indexer is worse than an oracle, so these are upper bounds. Needle-in-a-haystack scores, the number most launch posts quote, are the test least likely to show the loss.

One more design in the census answers the precision problem from the other side. Motif 3 keeps 14 full layers of 53, and its attention class, GDLA in modeling_motif.py, is a form of differential attention: two softmax maps, the second scaled by a learned gate and subtracted, so a head can put zero weight on irrelevant context. Fewer global layers, each asked to be sharper.

Where I land

The trend in the post is real and steeper than I expected: in 2025 most open releases were full attention in every layer, and since August only five of 35 are. The headline number is not. "0% used full attention throughout" describes a curated gallery of 21 cards, and a list of 35 releases from the same weeks has five that do, from three labs. If you quote the chart, quote it as Raschka's gallery and say so.

The "48% used no full-attention layers" bar is the more misleading half, because it reads as "48% gave up global attention". None of them did. They kept a softmax that can reach every token and put an indexer, a window or a compression in front of it, and the sparse ones still store the whole cache. What is disappearing is dense, unselected attention in every layer. What is not disappearing is the ability to look anywhere: every model in this census since January keeps it in at least some layers.

What I would watch is the first frontier-scale open model with no global layer at all, only states and windows. xLSTM 7B is the only one in 180 rows, and it is from 2025.

The table

The full census, one row per release, with every layer count, the pattern string, the config repo and the derivative flag, is below and downloadable as JSON. The widget above reads the same rows.

How this was counted: I fetched the config.json of every row from Hugging Face on 13 November (params.json for Mistral Large 3; six gated originals from public copies named in the dataset; Soofi S and Antares from the gallery card), read three modeling files (Motif, Phi-4-mini-flash, K2 Horizon), and classified every layer with one script; nothing was downloaded beyond configs and code, and no model was run. Download counts are Hugging Face's 30-day figure for the one repo linked per model. The KV and state sizes are arithmetic on config widths, not measurements; the MiniMax read and storage figures match the ones in this site's M3.1 Flash teardown, and the DeepSeek, LongBench, RULER, LeapQuant and indexer-share numbers are the publishers' and the linked articles'.