# Full attention in open LLMs: a 180-config recount of the chart that says it is gone > Satyajit Ghana — Head of Engineering @ Inkers Technology > canonical: https://ai.satyajit.si/articles/full-attention-census > date: 2026-11-13 > tags: attention, hybrid-attention, sparse-attention, linear-attention, kv-cache, long-context On 12 November [@crazyMLguy posted a chart](https://x.com/crazyMLguy/status/2120981423750549746) under the headline "Full Attention Is Disappearing From Open LLMs!" The numbers in the post: > 2025: 59% of models used full attention throughout. Jan–Aug 2026: Only 27% did. Aug–Nov 2026: 0% used full attention throughout; 48% used no full-attention layers. The post names three examples, Mistral Small 4, Kimi K3 and MiniMax M3.1 Flash, and gives the reason in one line: "Less attention-related memory traffic and better efficiency for long-context inference."
The post gives no dataset. The picture gives it away. The three diagrams are watermarked "Sebastian Raschka", and the bar labels say 32, 60 and 21 models. Raschka's [LLM Architecture Gallery](https://sebastianraschka.com/llm-architecture-gallery/) lists 122 models, each card with a release date, a layer mix and a link to its `config.json`. Counting its cards by date gives exactly 32 for 2025 and 60 for 1 January to 25 August 2026. The third window holds 22 cards to 5 November, and the gallery's changelog shows the 22nd, ERNIE 5.5 Flash Base, added on 5 November itself. So the chart is the gallery, drawn before ERNIE went in. I could not find where Raschka first published it: the gallery page, its changelog and the public pages of its six source articles do not contain the chart's labels. A count like this is only as good as two choices: which models are on the list, and what counts as "full attention". I checked both by going below the gallery cards to the configs. ## What a layer has to do to count as full Every decoder layer mixes information across positions somehow. The census sorts each of those token-mixing layers into one of six kinds. - **Full attention.** Each new token's query scores every earlier key with a softmax: MHA, GQA, MQA, and [MLA](/articles/attention-mechanisms), which compresses keys and values into a latent but still keeps one per token and reads all of them. - **Sliding window or chunked.** The query sees only the last $W$ tokens (Gemma, gpt-oss, MiMo) or its own fixed chunk (Llama 4's 8,192-token chunks). - **Linear or recurrent.** The past is folded into a fixed-size state: Gated DeltaNet, Kimi Delta Attention, Mamba-2, lightning attention, mLSTM. - **Short convolution.** Liquid's LFM blocks mix a few neighbouring tokens and nothing else. - **Index-selected sparse.** A small indexer scores the past and the real attention reads only the top $k$ tokens or blocks: DeepSeek Sparse Attention (DSA), MiniMax Sparse Attention (MSA), Qwen Sparse Attention, LongCat's LSA, DeepSeek's CSA. - **Compressed.** DeepSeek V4's heavily compressed attention (HCA) attends densely, but over a sequence squeezed 128 to 1. What each costs is easiest to see per layer. A full layer with $h_{kv}$ key-value heads of width $d$ stores, for every token in the context, $$ \text{KV bytes per token per layer} = 2 \cdot h_{kv} \cdot d \cdot b $$ with $b$ bytes per value, and each decoded token reads all $N$ of those entries. A window layer stores and reads at most $W$ of them. A recurrent layer stores a state whose size has no $N$ in it at all. An index-selected layer stores all $N$ entries plus a small index key per token, reads the index keys of all $N$ and the full entries of only $k$. So "sparse" and "linear" are different bargains: one keeps everything and reads little, the other keeps little. The chart's three classes follow from the per-layer kinds. A model is **all-full** if every token-mixing layer is full attention, **hybrid** if some are, **no-full** if none are. The open question is the sparse kinds. A DSA layer can attend to any earlier token, and below 2,048 tokens of context it attends to all of them. It is full attention with a gate in front. The chart counts it as not full: its 2025 bar has 2 "no full attention" models out of 32, and the gallery has exactly two 2025 cards without a dense full layer, xLSTM and DeepSeek V3.2, whose 61 layers are all DSA. I keep that convention so the numbers compare, and report the other one next to it. ## Reading 180 configs The gallery's layer mixes are hand-written summaries. I wanted the configs. For each card dated 2025 or later I fetched the linked `config.json` from Hugging Face (no weights), and for every model I classified each layer from whatever the config uses to describe the stack. There is no common field. Gemma, gpt-oss, OLMo and Qwen3.5 write a `layer_types` list; Nemotron-H writes a pattern string where `M` is Mamba-2 and `*` is attention; Kimi lists `kda_layers` and `full_attn_layers`, 1-based; MiniMax M3 has a `sparse_attention_freq` list; DeepSeek V4 encodes each layer as a `compress_ratios` entry, 0 for window-only, 4 for CSA and 128 for HCA. The core of the classifier, condensed: ```python # condensed from the census script; one branch per config dialect if t.get("layer_types"): # Gemma, gpt-oss, OLMo, Qwen3.5, Mellum, ... codes = [{"full_attention": "F", "sliding_attention": "S", "linear_attention": "L", "conv": "C", "deepseek_sparse_attention": "P"}[x] for x in t["layer_types"]] elif t.get("hybrid_override_pattern"): # Nemotron-H: M = Mamba-2, * = attention codes = ["L" if c == "M" else "F" for c in t["hybrid_override_pattern"] if c in "M*"] elif "sparse_attention_config" in t: # MiniMax M3 / M3.1 Flash codes = ["P" if x else "F" for x in t["sparse_attention_config"]["sparse_attention_freq"]] elif "compress_ratios" in t: # DeepSeek V4: 0 window, 4 CSA, 128 HCA codes = [{0: "S", 4: "P", 128: "H"}[x] for x in t["compress_ratios"][:n_layers]] ``` Where the config gives only a rule, I read the modeling code. Motif 3's `modeling_motif.py` makes a layer full when `layer_idx % period != 0` is false, so with period 4 it is 14 full and 39 window layers of 53. Phi-4-mini-flash-reasoning's `modeling_phi4flash.py` builds a SambaY stack: Mamba and 512-token window layers, one global layer at index 17, and seven cross-attention layers that reuse that one layer's keys and values. Llama-3.1-Nemotron-Ultra-253B is a pruned stack in which 98 of 162 blocks have no attention at all; its other 64 are full. Falcon-H1 runs attention and Mamba-2 side by side in every layer, so it counts as all-full. Draft and MTP layers, vision towers and FFN-only blocks are not counted. Two gallery models have gated configs that I could not read, Soofi S and Antares 1B, and for those the census uses the gallery's own layer mix and says so in the table. For the Ling family the config gives a group size (one MLA layer per group of 4, 6 or 8) but not where in the group it sits, so those rows carry counts and no layer order. On the 115 gallery cards the configs and the cards agree on every model's class; I parsed each card's layer mix and attention line and compared. Comparing each card's layer numbers with the config's turned up two real disagreements, and neither moves a model between classes. MiMo-V2-Flash's `hybrid_layer_pattern` has nine global layers of 48, where the card says eight. Step 3.5 Flash has 12 full and 33 window layers in its 45 `num_hidden_layers`; the card's 36 window layers include three MTP entries from the end of `layer_types`. The DeepSeek V4 cards list every layer as CSA/HCA, where `compress_ratios` makes V4-Flash's first two layers window-only; that is a summary, not an error. ## The chart reproduces exactly Classified from configs, the gallery's cards in the chart's three windows give: | Window | Models | All-full | Hybrid | No-full | |---|---|---|---|---| | 2025 | 32 | 19 (59.4%) | 11 (34.4%) | 2 (6.2%) | | 2026, 1 Jan to 25 Aug | 60 | 16 (26.7%) | 39 (65.0%) | 5 (8.3%) | | 2026, 26 Aug to 5 Nov | 21 | 0 | 11 (52.4%) | 10 (47.6%) | Those are the chart's bars to the rounding: in the gallery, 59.4% of the 2025 models, 26.7% of those from January to August 2026 and none of the last 21 use full attention in every layer. The chart is an honest picture of the gallery. The question is whether the gallery is a picture of open LLMs. ## A wider list, and the 0% goes The gallery is curated for architectures worth drawing. That is its job, and it skews the count in two ways. It counts sizes of one family as separate models: 2025 has six Qwen3 cards, all full attention, and they are nearly a third of that year's 19 all-full models. And its late-2026 window is mostly large releases with new attention, because those are the ones worth a new diagram. So I built a second list. The spine is the gallery's 115 cards from 2025 on. To it I added 65 open-weight releases from this site's own model table and coverage that the gallery does not have, each with a public config and a text decoder. I usually took one size per family, but kept sizes that were announced as separate models (K2 Horizon's three) or that the site covers on their own (Llama 4 Scout beside the gallery's Maverick). That brings in, among others, MiniMax-M1, Nemotron-H 56B, Nemotron Nano 9B v2, Granite 4.0 H Small, Command A, EXAONE 4.0 and 4.5, K-EXAONE, GLM-5.3, DeepSeek V3.1, Kimi K2 Thinking, Qwen3.8's 2.4T model, LongCat 2.0, A.X-K2, Inkling Small, IQuest-Q1, the K2 Horizon models, Nemotron 3.5 Bolt and Raon-1 nano. Then I took out 13 rows whose architecture is another release's, unchanged: third-party fine-tunes where the card's `base_model` or a site article says so (INTELLECT-3, BTL-3, Arete, Ember-1, Atria Dawn, Nex-N2.6 and others), and preview-to-final or dated re-uploads whose configs are byte-identical (Hy3, Motif 3, DeepSeek V4 Pro 0813, DeepSeek V4.1 Flash 1017). AI21's Jamba 1.7 is missing because both its repos are gated. Release dates are the gallery's for its cards and the Hugging Face repo creation date for the rest, which can run a little ahead of the public release. That leaves 167 releases. Same classes, same windows, except that the last one runs to today: | Window | Models | All-full | Hybrid | No-full | Mean share of full layers | |---|---|---|---|---|---| | 2025 | 63 | 38 (60.3%) | 23 (36.5%) | 2 (3.2%) | 69.0% | | 2026, 1 Jan to 25 Aug | 69 | 17 (24.6%) | 44 (63.8%) | 8 (11.6%) | 38.8% | | 2026, 26 Aug to 13 Nov | 35 | 5 (14.3%) | 19 (54.3%) | 11 (31.4%) | 26.0% | The first two rows barely move. The third does. Five releases since 26 August use plain global attention in every layer: - **K2 Horizon 375B-A23B, 7B and MoVA 36B-A4B** from MBZUAI's IFM. All three configs set `sliding_window` to null, and the 375B repo's `modeling_k2_horizon.py` builds the mask with `create_causal_mask` in that case, so every layer is ordinary causal GQA. The MoVA variant's novelty is [routed value experts inside attention](/articles/k2-horizon-mova), not a cheaper mask. - **[Nanbeige4.3-33B](/articles/nanbeige-4-3-33b)**, a 52-layer `LlamaForCausalLM`. - **[MiniCPM5-2B](/articles/minicpm5-2b)**, a 42-layer `LlamaForCausalLM`. None of them is on the gallery. Neither are most of my other additions to that window: IQuest-Q1 (25 full layers of 88), Inkling 2 Small, Ripple-65B, Nemotron 3.5 Bolt, Raon-1 nano, LFM2.5-24B-A2B, Mellum2.1, Spark-X2.6 and Intern-S2, all hybrids. The gallery's 0% is real for the gallery and false for open LLMs. The direction survives every way I cut it. All-full falls from 60.3% to 24.6% to 14.3%, and the average model's share of full layers falls from 69.0% to 26.0%. Collapsing each lab to one row per class per window, so that six Qwen3 sizes or three K2 Horizons count once, gives 54.3%, 29.4% and 10.3% all-full over 98 rows. Weighting by total parameters makes the last window look like the chart again: 3.2% all-full and 62.9% no-full, because the all-full survivors are small except K2 Horizon 375B and the no-full models are the giants (Kimi K3.1 Pro at 2.8T, LongCat-2.5 at 1.6T, Mistral Large 4 at 1T). Weighting by last month's downloads gives 88.0% no-full in that window, which mostly says that GLM-5.3-Flash and Qwen3.8-Flash-Next are popular; three repos carry most of the weight, so I would not lean on that cut. One boundary case: GLM-5.3's repo was created on 25 August, the last day of the middle window, and Z.ai had promised the weights two weeks after its 14 August launch, so they may have gone public inside the last window. Moved into the last window, it makes no-full 12 of 36, 33.3%. G9v3-39A5B is the other row I could not pin: its repo dates from 21 July and Artificial Analysis lists it for August. If it belongs in the last window, all-full there becomes 6 of 36.