Hashed n-gram memory (Engram) · 2026 · Other · 11 min
- lookup-table
- hashing
- conditional-memory
- offload
- mixture-of-experts
- inference
- explainer
A 1:54 narrated explainer, drawn in code. Every number and picture in it is this page's own; the sources are below.
› transcript
Only Alexander the Great could tame the horse. A Transformer spends layers working out that those three tokens are one name. What if it could just look the phrase up? Hi, I'm Solveig! This is conditional memory, and two of this year's largest open models ship it. Take the last two or three tokens. Hash their ids. Each head reduces the hash modulo its own prime, and picks one row of its own table. Glue the rows together. The hidden state gates them, and they join the residual stream. Under one prime, two unrelated phrases can land on the same row. Under another prime, they almost never collide as well. The rows together still tell them apart. A fetched row knows nothing about context. So the model checks it. The hidden state acts as a query. The memory becomes a key. If they disagree, the gate closes, and a collision stays out. Here is how big this gets. One table holds over a quarter of the parameters. A token reads sixteen rows of it. The addresses are known before the layer runs, so rows can be prefetched from the CPU. A hundred billion parameters cost under three percent. Split a fixed budget between experts and memory, and the best loss is not all experts. About a fifth to a quarter goes to memory. And a bigger table keeps lowering the loss, at no extra compute. Some parameters only need to be looked up. And a lookup can live far from the GPU. Hash, fetch, gate. Thanks for watching! Every source is in the full article. I'm Solveig. Bye!
The last few token ids are hashed, and each head reduces the hash modulo its own prime to pick one row of a very large table. Two n-grams that collide under one prime almost never collide under another, so the concatenated rows still identify the phrase. A gate computed from the hidden state decides how much of that memory enters the residual stream, at one layer or a few.
Qwen3.8-Flash-Next's checkpoint has a single layer that holds 28.4% of every parameter in the model. It is not attention or an expert bank. It is a table of 320,001,536 rows, each 160 numbers wide, and a token reads sixteen of those rows and nothing else. DeepSeek-V4.1-Flash ships two tables like it, 196.61B parameters together, more than a quarter of the model. Both are versions of one idea from DeepSeek's Engram paper (Peking University and DeepSeek-AI): give the Transformer a lookup primitive, so that it stops spending layers on reconstructing phrases it could have looked up.
This page builds the module from the paper, then checks it against the shipped configs that this site has read: Tinfield 1 decoded Qwen's table, the V4.1-Flash teardown read DeepSeek's, and Edge0-145B moved one onto an SSD.
Why a model would want a lookup table
The paper's argument starts from what text is made of. Some of it needs reasoning. A lot of it is local and stereotyped: named entities, idioms, formulaic phrases, the second half of "Alexander the Great". A Transformer has no instruction for "recognise this phrase". To resolve a multi-token entity it spends several early layers of attention and feed-forward work rebuilding the association every time it sees it. The paper calls that "an expensive runtime reconstruction of a static lookup table".
Mixture-of-experts already makes most parameters inactive per token, but an expert is still a matrix multiply, and which expert fires depends on the hidden state, so it is only known at the last moment. The paper proposes a second axis of sparsity next to MoE's conditional computation: conditional memory, parameters fetched by a key known from the token ids alone.
The module, step by step
The paper's figure shows the pieces, and the diagram above redraws them as one path from tokens to the residual stream.

1. Canonical tokens. Subword tokenizers give "Apple" and " apple" different ids. Engram first maps every raw id through a precomputed function that collapses ids with the same normalised text (NFKC, lowercasing and so on). The paper reports this shrinks a 128K vocabulary's effective size by 23%. DeepSeek's reference inference/engram.py does the same: lowercase, strip accents.
2. Suffix n-grams. At position , form the n-grams ending there, , for up to a maximum .
3. Multi-head hashing. The space of all n-grams is far too large to index directly, so each n-gram order gets hash heads, and head maps the n-gram to a row of its own table , whose size is prime:
is a cheap multiplicative-XOR hash. The rows from every head of every order are concatenated into one memory vector .
4. Context-aware gating. A static row knows nothing about context, and it may be a hash collision or the wrong sense of a word. So the current hidden state , which has already seen the context through the layers below, acts as a query against a key made from the memory:
If the retrieved memory contradicts the context, goes towards zero and the row is suppressed.
5. A short convolution, then the residual. A depthwise causal convolution of width 4 with SiLU refines the gated values, , and the result is added to the hidden states before that layer's attention and MoE. The convolution starts at zero, so at initialisation the module is an identity and training decides how much memory to let in.
With hyper-connections, the backbone the paper uses by default (mHC with 4 streams; the xHC article explains them), the table and are shared across streams and each stream gets its own and its own gate.
Why a different prime for every head
A single hash table of any size collides: two unrelated n-grams land on the same row and share an embedding forever. Several heads with different prime moduli make collisions independent. Two n-grams that collide under one prime almost never collide under another as well, so the concatenation of all heads' rows still identifies the phrase, and the gate can suppress the head that went wrong.
The Tinfield 1 article read the three small integer tensors that drive Qwen's version. The sixteen table sizes are 20,000,003, 20,000,023, 20,000,033 and so on up to 20,000,171: sixteen distinct primes just above twenty million. The hash multipliers are three odd constants of 43 to 45 bits, one per n-gram position. The article's description: take the last two or three token ids, hash them with a polynomial whose coefficients are those multipliers, reduce modulo each head's prime.
Why the table can live somewhere slow
Everything about a lookup's address is known as soon as the tokens are. Nothing depends on a hidden state. That is the system property the paper builds on. During training the tables are sharded across GPUs and rows are gathered with an all-to-all. At inference they can sit in host memory, and the runtime prefetches the rows over PCIe while the layers before the Engram layer compute.
The paper measured it with a 100B-parameter Engram table entirely in host DRAM, inserted at the second block of dense 4B and 8B models on H800s. Throughput went from 9,031.62 to 8,858.28 tokens per second on the 4B model and from 6,315.52 to 6,140.02 on the 8B: a penalty of at most 2.8%. Traffic scales with the rows a token reads, not with the table. And because n-gram use is Zipfian, the paper suggests a cache hierarchy: hot rows in HBM or DRAM, the long tail on NVMe.
Edge0-145B takes that literally. Its "PLE", which is Qwen's n-gram table, sits on the SSD and is read with direct I/O. Sixteen rows of 160 FP8 bytes is 2,560 bytes a token; in 4 KiB pages the device sees about 64 KiB a token, under 5 MB/s at 75 tokens per second. The 51.2B parameters never occupy RAM or VRAM.
Experts or memory: the U-shaped split
If both MoE experts and Engram rows are inactive parameters, a fixed parameter budget has to be split between them. The paper defines as the fraction of the inactive budget given to routed experts and sweeps it at two compute budgets, holding total and active parameters fixed.

Both curves are U-shaped. Pure MoE is not the optimum: moving roughly 20 to 25% of the sparse budget into Engram gives the best loss, and in the 10B regime the loss improves from 1.7248 to 1.7109. Push too far the other way and the model loses conditional computation, and memory cannot replace it. Separately, with compute fixed and the table growing from 258,000 slots to 10 million, loss follows a power law in the table size. The memory is a scaling knob that costs storage and nothing else.
At scale, the paper's Engram-27B takes MoE-27B, cuts routed experts from 72 to 55, and spends the freed parameters on a 5.7B-parameter table at layers 2 and 15 (n-grams up to 3, 8 heads), keeping the total at 26.7B. The abstract lists the gains over the iso-parameter, iso-FLOPs MoE: MMLU up 3.4, BBH up 5.0, HumanEval up 3.0, and Multi-Query NIAH from 84.2 to 97.0. That last one is the surprising result. The paper's reading is that once local patterns come from lookups, attention has more capacity left for global context.
Where to put it: a single module works best at layer 2 in the paper's sweep, and splitting the same memory across layers 2 and 6 did slightly better. Earlier is better for modelling, deeper gives the prefetch more time to hide. Shipped models pick early layers.
What actually ships
| Table | Per token | Where | |
|---|---|---|---|
| DeepSeek-V4.1-Flash | 2 tables, ~384M rows each, 256 FP8 values per row, 196.61B parameters | 2, 3 and 4-grams × 8 heads = 24 rows per layer, about 12.7 KB from 203 GB | two Engram layers |
| Qwen3.8-Flash-Next | 320,001,536 rows × 160 BF16, 51.2B parameters | 2 and 3-grams × 8 heads = 16 rows, concatenated to 2,560 | one layer |
| Edge0-145B | Qwen's table in FP8 | 2,560 bytes | on the SSD |
| KLM 1.6 | 125,000 bigrams and 38,000 trigrams by exact match, 32-wide rows | 2 rows | beside the embedding |
KLM is the small, unhashed version: it looks the bigram and trigram ending at each token up in a sorted list of the most frequent ones, with row 0 for anything missing, and lifts the two rows to model width with small projections (KLM 1.6). At 99M parameters, exact match is affordable and collisions are not a concern.
A related idea is easy to confuse with this one. A per-layer embedding, as in Gemma 3n, Gemma 4's small models and Raon-1 nano, is also a table that is only indexed, but its key is the single token id, and it gives every layer a private slice of that token's row. Raon's table is 259,840 rows by 6,144, half the 3.1B checkpoint. Edge0's article notes that its "PLE" is Qwen's n-gram table, not a per-layer embedding in Raon's or Gemma's sense, despite the name in the files.
What it costs
Storage, plainly. Qwen's table is 95.4 GiB in BF16. DeepSeek's two are 202.76 GB on disk, more than half of what the experts take. Somebody has to hold them, and while a host or an SSD is cheaper than HBM, a laptop deployment still has to ship and page tens of gigabytes the model barely touches per token.
Hashing also makes the memory unreadable. A row is the sum of whatever n-grams collided onto it under that prime, so the table cannot be edited or inspected per phrase, and the gate is the only thing standing between a collision and the residual stream. The paper's gating visualisation shows the gate firing on completed phrases, "Alexander the Great", "By the way", which is the behaviour you want; I have not seen a measurement of how often it fails.
And the gain is front-loaded. The memory replaces early-layer work on local patterns. It does nothing for reasoning that has no stereotyped surface form, which is why the U-curve turns back up when memory takes too much of the budget.
Where I land
This is the most interesting new parameter type since mixture-of-experts, and for the same reason MoE was: it changes the ratio between parameters you store and parameters you compute with. An expert still costs a matmul and wants to be near the accelerator. A hashed row costs a pointer chase and can be anywhere, with its address known ahead of time. Qwen's own card says embeddings are "more amenable to offloading than Mixture-of-Experts", and Edge0 shipped it from an SSD.
The open question for me is how far the U-curve's bottom moves at larger scale, and whether the long-context gain holds outside the paper's own runs. Two of the largest open releases of 2026 have bet on it, which is better evidence than a paper. Edge0's latent experts show the other way to make sparse parameters cheaper, by shrinking each expert instead of adding a table.