# Computers compiled into Qwen, the second wave: a GPU, a filesystem, a lab and an x86 core
> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.satyajit.si/articles/qwen-compiled-computers
> date: 2026-11-09
> tags: qwen, linear-attention, hybrid-attention, compilers, agents
Ten days ago I took apart [Qwenputer](/articles/qwenputer), a RISC-V CPU compiled into the weights of Qwen3.5-4B with its RAM held in the
state of the model's Gated DeltaNet layers. That article covers the core idea: pin a DeltaNet head's gate open and its write strength at
one, use one-hot keys, and each row of the head's state becomes an exact memory cell. I will not repeat it here.
Within a week, four more machines were compiled into Qwen3.5:
- **qwenGPU** ([@TheVixhal](https://x.com/TheVixhal/status/2116938733815738516), [repo](https://github.com/vixhal-baraiya/qwenGPU)): "Qwen3.5-2B with a SIMT GPU inside its weights,
45 compiled layers in front of Qwen's own", running CUDA kernels compiled to PTX.
- **qwenFS** ([post](https://x.com/TheVixhal/status/2117641713036759494), [repo](https://github.com/vixhal-baraiya/qwenFS)): a filesystem that "adds 8 layers in front
of Qwen's own 24", files in attention, tables in DeltaNet state.
- **qwenLab** ([@deepaksinghml](https://x.com/deepaksinghml/status/2118047922403324115), [repo](https://github.com/DeepakSingh98/qwenlab)): Qwen3.5-9B "with six small
transformers living inside it", each a 420k-parameter model learning modular addition, trained by the model's own forward passes.
- **Qwenx86** ([@TeksEdge](https://x.com/TeksEdge/status/2118857211992854892) on a release by Quazim0t0, [model](https://huggingface.co/Quazim0t0/Qwenx86-4B-compact)):
"an actual 64-bit x86 CPU compiled into the neural network weights of Qwen3.5-4B", in 13 added layers.
All four are open. qwenGPU and Qwenx86 use Qwenputer's compiler (`chipformer`) as a library; qwenFS and qwenLab say they share no code with
it. This article asks three questions of each: how many layers and parameters it adds, what one generated token buys, and whether the one
new trick it brings holds up when rebuilt on the reference code. Every claim carries a label: **measured** (I ran it), **read** (I read it in
the code at the commit named), **reported** (the author says so) or **reasoned** (my inference).
## What I could and could not run
I cloned the three GitHub repos shallow (qwenGPU at `e2695f6`, 1 November; qwenFS at `b124354`, 5 November; qwenlab at `925af4a`,
4 November) and Qwenputer's compiler at the commit qwenGPU pins (`f07a0b2`). For Qwenx86 I read the model card, the runtime code and the
config from the Hugging Face repo at `964daef`, and the tensor header of the 8.75 GB `compact.safetensors` by HTTP range request. Nothing large
was downloaded.
The plan was to run each repo's circuit tests on a CPU, since qwenGPU's and qwenFS's circuit tests need no Qwen weights. The sandbox's
permission layer refused to execute the downloaded repositories' code, so I did not work around it. What I ran instead are two small
scripts of my own that rebuild one compiled head from each of qwenFS and qwenLab, using the head's weights as the source sets them, on
the reference Gated DeltaNet recurrence and gated RMSNorm that ship in `transformers` 5.20.2 (with `torch` 2.14.1, CPU, two threads). The
scripts and their outputs are in `public/articles/qwen-compiled-computers/verify/`. So none of the merged models, benchmarks or test suites below
is verified by me; the numbers about them are **reported**.
## The four machines, side by side
The table hides how different the four are, so in turn:
**qwenGPU** is the closest relative of Qwenputer. It is a 4-lane SIMT machine with "15 registers per lane, a 4-stage pipeline, a 32-block
program ROM and 126 words of global memory", built with Qwenputer's compiler and placed in front of Qwen3.5-2B's 24 layers (**reported**,
README). The host compiles CUDA to PTX with clang, lowers the PTX to machine blocks (`qgpu/lower.py`, 1,424 lines), and feeds the program and the input
arrays as tokens. From then on the model writes the launch: each `EXEC(b, w)` token runs block `b` for warp `w` in one forward pass. The
code (**read**, `qgpu/circuit/kernel.py` and `qgpu/merge.py`) fixes the token cost exactly. A block holds up to four instructions, one per
pipeline stage, and the merge defaults are 4 lanes and 4 stages. After the `EXEC` token come two 16-bit `VAL` tokens per lane for each register a later
block reads, so 8 tokens per live register. A warp that only stores costs one token, because a store is a write into a memory head
above the stage that computes it, in the same forward pass. The best case is 16 thread-instructions per generated token, and every value that
has to survive to the next block costs 8 more (**reasoned** from those two rules). The README reports "tens to a few hundred generated tokens" per launch at about 10 to 15
tokens a second on Apple silicon. The 45 layers and the KL of "about 7e-4" against base Qwen are **reported**; no merged checkpoint is
published, you build it with `qgpu.merge`, so I could not count its weights.
One detail did not match the README. `kernel.py` defaults to `LANES = 8` and documents "16 VALs per live-out", while the README and
`merge.py` say 4 lanes (**read**). Both are consistent: the circuit is parameterised, and the merge passes `lanes=4`. If you read the
circuit tests, which use the defaults, you are reading an 8-lane machine.
**qwenFS** is not a CPU at all, and that is what makes it interesting. Its 8 layers are hand-written in Qwen3.5's own layer types with no
compiler: five Gated DeltaNet layers and three full-attention layers, in the order `linear, linear, linear, full, linear, linear, full, full`
(**read**, `LAYERS` in `qfs/circuit.py`). The merge widens Qwen3.5-2B's residual from 2,048 to 4,096 and scales Qwen's norm gains to match. The
filesystem uses each part of the transformer for what it is good at. A file's text stays in the copy layer's attention log, keyed by
(file index, version, offset), and a read is attention copying it back "exactly token by token". A lookup layer maps a path to a file index.
A seek layer holds a row per word for search and per link. The DeltaNet state holds 128-row tables for versions, bases and paths (the 128 is the file limit), and the MLPs
parse Qwen's tool calls and force the response tokens. So a read costs one token per token of file text, and the response is the model's
own output, with the forced token winning "by hundreds of logits" (**reported**).
The offset in that key is the part I rebuilt below. Attention can only copy the right token if the query knows how far into the
segment it is, and a DeltaNet head has no integer counter. qwenFS makes one out of the gate.
**qwenLab** is the outlier. It is described as six transformers "living inside" Qwen3.5-9B, and the repo is candid that the word is loose.
The lab is "2,382 layers appended to Qwen's own 32", with "its own residual stream, far wider than Qwen's 4,096" (2.7 M fed-back lanes and 51 M
lanes per position), and its memory heads are "Gated-DeltaNet-type with no q/k L2-norm, no conv, no output gated RMSNorm, alpha=1 and fp32
state" (**reported**, `docs/REPORT.md`). It couples to Qwen through token reads and an additive term on Qwen's logits, and while the lab runs a
command Qwen's 32 layers are skipped. So this is a second program sharing Qwen's tokens rather than layers inside Qwen. The engineering is still
remarkable: forward pass, cross-entropy, full backward pass and AdamW for a 419,712-parameter learner, with a float64 twin matching
PyTorch autograd to about 1e-15, at 193 ticks and 0.644 s per update on an H100, against 5.5 ms for the same update in plain PyTorch
(**reported**). The weight bank is the idea I rebuilt.
**Qwenx86** is a fork of Qwenputer's RV32 core with x86 flag micro-ops and conditional jumps added, 64 KB of RAM (up from 32 KB) and a
translator from x86-64 to the core's 32-bit ops (**reported**, model card). That last part matters for the headline. The circuit does
not decode x86. "MSVC-compiled C is translated once into the core's own ops", and "raw x86 bytes are decoded by firmware, not by circuit
layers". The 2,000 of 2,000 lockstep checks against Unicorn test the semantics of that translation plus the core, which is a real result;
calling the circuit "an x86 CPU" compresses it.
## What the Qwenx86 checkpoint is
Everything in this section is **measured** from `model/config.json` and the header of `compact.safetensors`. The script that reads it is in
`verify/`, with its output in `qwenx86_headers.json`.
- **61 layers, not 45.** The model card says 13 layers ahead of Qwen's own 32, which would make 45. The config has 61. Layers 0 to 12 are the CPU: the six
baked "boot" states each fill exactly these 13 layers (32 heads of 128 by 128 fp32 per layer, 27,262,976 bytes of state). Qwen's 32 layers
are 29 to 60, with full attention at 32, 36, ..., 60, Qwen's own pattern shifted by 29.
- **Layers 13 to 28 are state-only.** They are 16 Gated DeltaNet layers whose output projection and MLP are stored as empty blocks, so they cannot
write to the residual stream. Their `in_proj_qkv` reads residual dimensions 0 to 127 and their write strength reads dimension 2,776, which
`chipformer.json` names `mu_dim`. My reading is that these are the slots of the v2 memory, which the card says "run beside the model and
are driven by host code" (**reasoned**; the card names "layers 13-16", and I did not resolve the numbering). They do not change what the 13 CPU
layers compute.
- **The circuit is 45,288,507 stored values.** The compact format keeps, for every matrix, only the block of rows and columns that hold a
nonzero. Across layers 0 to 12 those blocks hold 45.3 M bf16 values, against 175,013,824 weights in each dense layer of this width (the
count from my Qwenputer article). That is 2.0% of the dense size, and an upper bound on the true nonzero count, since a kept block can still
hold zeros inside. The busiest layer, 7, keeps 10.2 M; layer 2 keeps 86 thousand.
- **The state-only layers keep 11.1 M values; Qwen's layers keep 3.57 B,** the full matrices over Qwen's own 2,560 columns. The widened
residual is 3,968, as in Qwenputer, and the vocabulary adds 230,336 chip-token rows. Text rows keep 2,775 columns (Qwen's 2,560 plus 215
circuit features) and chip rows 216.
So the compact checkpoint is honest about its size: about 9 GB for Qwen's text model plus a circuit that, laid out dense, would be 13
layers of 175 M weights, and stored compactly is 45 M.
The card's costs are worth reading as the model's own disclaimer (**reported**): `(1200 - 350) / 5` took 56,667 machine tokens and 41.0 s,
a C program printing `6 * 7` took 169,293 machine tokens and 71.2 s, and block verification (checking up to 256 machine tokens per pass) cut
a 168,578-token C chat from 582 s to 44.5 s. On the first 28 GSM8K test questions, Qwen with the calc tool scored 26/28 against 23/28 for plain
Qwen. As with Qwenputer, the gain is having a calculator; where it runs is the novelty.
## A counter in the exponent: qwenFS's offset head
The qwenFS circuit describes its offset counter in one line: "a decaying DeltaNet counter, alpha = 2^(-1/16) per token, read in the head norm's
eps regime and decoded by thresholds (exact to 600 tokens)". Here is what that means, from the weights `counter()` sets in `qfs/circuit.py` (**read**):
- At an **anchor** token (the start of a segment), the write strength is $\sigma(30) \approx 1$ and the decay logit is 200, so the gate
$\exp(-\mathrm{softplus}(200))$ wipes the state to zero. The head then writes the value 20 into one cell.
- At every other token the write strength is $\sigma(-30) \approx 0$, nothing is written, and the decay logit is $-3.125$, so the cell is
multiplied by $\alpha = \exp(-\mathrm{softplus}(-3.125))$.
- After $n$ tokens the cell holds $20\,\alpha^n$. The count lives in the exponent.
The reading side is the clever part. Qwen3.5's head output goes through a gated RMSNorm, which would normally divide a single nonzero
value by its own magnitude and return a constant. qwenFS gives the query a tiny component (1e-5) so the output is far below the norm's
epsilon of 1e-6. In that regime the norm divides by $\sqrt{\epsilon}$, a constant, and passes the value through linearly. The MLP then decodes
$n$ with a thermometer: 600 pairs of units, the pair for $j$ firing when the value is below the geometric mean of the values at $j-1$ and $j$.
Why this works in bf16 at all is the point worth taking away. A bf16 number has 8 significant bits at every exponent from about $2^{-126}$
to $2^{127}$. A value that halves every 16 tokens moves down the exponent range at a steady rate and never runs out of mantissa, so neighbours stay distinct for hundreds
of tokens. A linear counter in bf16 would be exact only to 256.
I rebuilt the head with exactly those weights on `transformers`' `torch_recurrent_gated_delta_rule` and `Qwen3_5RMSNormGated`, with bf16
activations, the conv tap followed by SiLU as in the real layer, and `A_log` and `dt_bias` at zero as `common()` sets them (**measured**,
`counter.py`):
| Quantity | Value |
|---|---|
| decay per token, $\alpha$ | 0.95791, which is 1.00032 times $2^{-1/16}$ |
| value written at the anchor, after norm, gate and out-projection | 0.03247 |
| distinct values for n = 0 to 600 | 601 of 601, strictly falling |
| ratio between neighbours, n = 1 to 600 | 0.945 to 0.971 (bf16 rounding on top of 0.958) |
| first n where the value stops falling | 1,737 |
| first n where it reads exactly 0 | 1,820 |
So the "2^(-1/16)" is an approximation that works: softplus of $-3.125$ is 0.0430, against $\ln 2 / 16 = 0.0433$. The 600-token limit is a
design choice and not a numerical wall. The head stays strictly decreasing to 1,736 tokens, and fails there because the head's output,
about $2^{-25}$ times the cell, has fallen into fp32's subnormal range before the norm (**reasoned** from the magnitudes; the run shows
where, not why). The thermometer's size is the reason to stop at 600: 1,200 MLP units for one counter.
I also ran the decode. Using the thermometer's own formula, two SiLU units per threshold with gate scale $126/\tau$, evaluated in bf16 on my
measured values (**measured**, `thermo.py`), the units that should read 1 read 0.75 to 1.25, and the units that should read 0 read $-0.097$ to
0.061. They are not clean 0/1 values, because adjacent counter values are only about 4% apart, so a unit next to its threshold sits on
SiLU's curve and not on its flat part. Thresholded at 0.5 they decode every $n$ from 0 to 600 exactly, and their sum is within 0.37 of $n$.
That is enough for what follows, where the next layer's MLP turns the thermometer into offset bits, but "decoded by thresholds" is the honest
wording, and it is the circuit's own.
## A matmul in a state: qwenLab's weight bank
qwenLab stores "every 128x128 weight block of every unit" in a memory head, and the forward pass is "reading it with the activation as the
query". The arithmetic is a line: with one-hot keys and full write strength, the state after 128 writes is $S = W^\top$, and the head's read
is $o = S^\top q$. Put the activation $x$ in as $q$ and $o = Wx$. Reading a transposed copy with the output gradient gives $W^\top\,dy$, the
backward pass.
The catch is in Qwen3.5's stock layer, which L2-normalises $q$ and $k$ and divides $q$ by $\sqrt{d}$. With the norm on, the read returns
$Wx / (\lVert x \rVert \sqrt{d})$: the right direction, at a scale the layer cannot undo, since $\lVert x \rVert$ changes every token. I checked both
paths on `torch_recurrent_gated_delta_rule` with 128 by 128 blocks in fp32, five random blocks and activations (**measured**, `bank.py`):
| Path | Relative error against the float64 matmul |
|---|---|
| stock q/k L2-norm, raw read | 0.997 (the scale is gone) |
| stock, rescaled by $\lVert x \rVert \sqrt{128}$ after the fact | 4.3 to 5.6 x 10-7, and the stored state is not bit-exact |
| no q/k norm, as qwenLab builds it | 0.86 to 1.20 x 10-7, state bit-exact |
| plain fp32 matmul, for scale | 0.87 to 1.02 x 10-7 |
| backward, transposed copy read with a random gradient | 1.2 x 10-7 |
Two things follow. First, qwenLab had to change the layer type. Without q/k normalisation the read is a matmul with fp32 accuracy;
with it, the read loses the activation's norm. That is why the repo lists "no q/k L2-norm" among its modifications, and why "inside Qwen"
should be read as "built from layers shaped like Qwen's". Second, the stock path also damages the write: the L2 norm's epsilon makes a one-hot
key 1/sqrt(1 + 1e-6) long, not 1, so the state is off in the seventh digit. My Qwenputer rebuild hit the same detail, and the compiled CPUs
avoid it by scaling keys up. qwenLab keeps fp32 state and drops the norm, which removes the problem.
The cost of the bank is visible in the same numbers. Writing one block takes 128 ticks, one per row, which is where the repo's "129 row ticks" per fork or optimizer
step comes from. A head answers one query per position, so the lab keeps "2 layouts x 32 read copies": 1,686 heads per unit and 11,892 in all, 790 MB of
recurrent state for six learners of 1.26 M floats each (parameters plus both Adam moments) (**reported**).
## How Qwen uses them
All four keep Qwen's own weights unchanged and reach the machine through Qwen3.5's native tool-call format. They differ in who drives.
- **qwenFS** is the one where the weights do the most. The circuit parses the call as Qwen writes it, keying on single tokens like
`=name`, `=text` and ``, then prints the tool response and opens the next assistant turn itself (**reported**, README). Its
scenario eval passed 57 of 57 turns with the reference doc in the system prompt and 49 of 57 without it, and the misses were Qwen's calls,
not the filesystem's answers.
- **qwenGPU's** host compiles CUDA and lowers PTX; the model writes every step of the launch. Its own README is plain about the language
model's half: normal mode answered 20 of 24 sampled questions with GPU-computed numbers, and in advanced mode Qwen wrote a correct kernel 4 of 5 times,
with "A 2B model's CUDA is still unreliable" (**reported**).
- **Qwenx86** runs six firmwares from baked boot states (calc, MicroPython, a C compiler called qcc, notes, a science table and Python docs), and its v2
memory is driven by host Python, which the card says plainly (**reported**).
- **qwenLab** had Qwen pick learning rates and weight decays in a two-round study; the best unit reached 49.09% test accuracy against 15.68% for the control, from one
session, and the repo calls the strategy "textbook, not a discovery" (**reported**).
## Limits
- **I ran no merged model and no repo test.** The permission layer refused the repos' code. My two rebuilds use each head's weights
as written in the source, on the reference recurrence; they do not show the rest of either circuit working.
- **The 45 qwenGPU layers and the KLs are not checked.** No qwenGPU or qwenFS checkpoint is published, and building one needs the base model and 16 GB of
memory. qwenGPU's interpreter matching a Tesla T4 "on 49 of 49 test kernels" is **reported**.
- **My counter rebuild assumes the residual's `ONE` dimension reads 1 after the input norm**, which is what qwenFS's fixed-scale ballast is for.
The calibrated values in the real model differ from mine by that norm's scale; the ratios, the 601 distinct values and the place where it breaks do not
depend on it (**reasoned**).
- **The Qwenx86 state-only layers are read from headers alone.** I know they cannot write to the residual; what drives them is my reading of
the card.
- **Speed is the price everywhere.** 0.644 s against 5.5 ms per training update in qwenLab, 41.0 s for one division in Qwenx86, 10 to 15 tokens a second for
qwenGPU on Apple silicon. None of the authors claims otherwise.
## What it shows
Read together, the four repos make Qwenputer's point more general. A pretrained hybrid model contains three different kinds of exact
storage, and these projects found a use for each: DeltaNet rows as registers and tables, an attention log as an append-only file, and,
with one modification, a DeltaNet state as a weight matrix you can multiply by. The two tricks I rebuilt are small and both hold on the
reference code. One stores a count in a float's exponent, so bf16 holds 600 positions where a linear count would stop at 256. The other turns a
recurrent read into a matmul once the query norm is removed.
What they do not show is that a language model gains anything from having a computer inside rather than beside it. Every benchmark in
these READMEs is a tool-use benchmark, and where a host-run tool was the baseline, as in Qwenputer, it scored the same. A related approach,
[Program-as-Weights](/articles/program-as-weights), learns its program into an adapter; these machines are written by hand, and none of them
learns anything about its own machinery. The open question is still the one the Qwenputer article ended on: whether a model can learn to use such a memory without having it compiled in.
## Reproducing it
`public/articles/qwen-compiled-computers/verify/` holds `counter.py` (the qwenFS head on `transformers`' reference recurrence and gated norm),
`thermo.py` (the thermometer decode, in bf16), `bank.py` (the weight bank, with and without the q/k norm) and their JSON outputs, plus
`qwenx86_headers.json` (per-layer stored values from the compact checkpoint's header). They need `torch` and `transformers` 5.20 or later, and
run in under a minute on a CPU. The three repos are MIT-licensed; Qwenx86's code is MIT and its weights Apache-2.0, as a modified Qwen3.5-4B.
## Sources
- qwenGPU: [github.com/vixhal-baraiya/qwenGPU](https://github.com/vixhal-baraiya/qwenGPU), commit `e2695f6`; [launch post](https://x.com/TheVixhal/status/2116938733815738516)
- qwenFS: [github.com/vixhal-baraiya/qwenFS](https://github.com/vixhal-baraiya/qwenFS), commit `b124354`; [launch post](https://x.com/TheVixhal/status/2117641713036759494)
- qwenLab: [github.com/DeepakSingh98/qwenlab](https://github.com/DeepakSingh98/qwenlab), commit `925af4a`, `docs/REPORT.md`; [launch post](https://x.com/deepaksinghml/status/2118047922403324115)
- Qwenx86: [huggingface.co/Quazim0t0/Qwenx86-4B-compact](https://huggingface.co/Quazim0t0/Qwenx86-4B-compact), revision `964daef`; [post](https://x.com/TeksEdge/status/2118857211992854892)
- Qwenputer, the compiler two of them use: [github.com/divagr18/qwenputer](https://github.com/divagr18/qwenputer), and [my teardown](/articles/qwenputer)