Computers compiled into Qwen, the second wave: a GPU, a filesystem, a lab and an x86 core

mdjsonmcpcopy↗ai

2026-11-09 · 18 min · qwen · linear-attention · hybrid-attention · compilers · agents

Why read this

Solid· top 85%

Two compiled mechanisms rebuilt and measured on transformers' own recurrence, and the Qwenx86 headers read; none of the four merged models was run.

• Original analysis
• A new technique
• Explained from first principles

·LLM architecture·Runs on a consumer GPU·MIT·Research teardown

How this was scored

Is it new?

2 of 3: A real new idea, method or capability

Can I trust it?

2 of 3: Measures key facts from files, code or configs

Can I run it?

1 of 3: API-only, gated or restrictive licence

Will I understand it?

2 of 3: Mechanism from first principles with figures

Can I act on it?

1 of 3: General advice

Will it last?

1 of 3: Relevant for months

Does it affect many?

1 of 3: A specialist community

Only here?

2 of 3: A teardown or measurement few others did

Score 55 of 100, ranked 530 of 701 rated articles. Each question is answered 0–3 by hand, and a 3 is rare. How articles are scored

Ten days ago I took apart Qwenputer, a RISC-V CPU compiled into the weights of Qwen3.5-4B with its RAM held in the state of the model's Gated DeltaNet layers. That article covers the core idea: pin a DeltaNet head's gate open and its write strength at one, use one-hot keys, and each row of the head's state becomes an exact memory cell. I will not repeat it here.

Within a week, four more machines were compiled into Qwen3.5:

• qwenGPU (@TheVixhal, repo): "Qwen3.5-2B with a SIMT GPU inside its weights, 45 compiled layers in front of Qwen's own", running CUDA kernels compiled to PTX.
• qwenFS (post, repo): a filesystem that "adds 8 layers in front of Qwen's own 24", files in attention, tables in DeltaNet state.
• qwenLab (@deepaksinghml, repo): Qwen3.5-9B "with six small transformers living inside it", each a 420k-parameter model learning modular addition, trained by the model's own forward passes.
• Qwenx86 (@TeksEdge on a release by Quazim0t0, model): "an actual 64-bit x86 CPU compiled into the neural network weights of Qwen3.5-4B", in 13 added layers.

All four are open. qwenGPU and Qwenx86 use Qwenputer's compiler (chipformer) as a library; qwenFS and qwenLab say they share no code with it. This article asks three questions of each: how many layers and parameters it adds, what one generated token buys, and whether the one new trick it brings holds up when rebuilt on the reference code. Every claim carries a label: measured (I ran it), read (I read it in the code at the commit named), reported (the author says so) or reasoned (my inference).

What I could and could not run

I cloned the three GitHub repos shallow (qwenGPU at e2695f6, 1 November; qwenFS at b124354, 5 November; qwenlab at 925af4a, 4 November) and Qwenputer's compiler at the commit qwenGPU pins (f07a0b2). For Qwenx86 I read the model card, the runtime code and the config from the Hugging Face repo at 964daef, and the tensor header of the 8.75 GB compact.safetensors by HTTP range request. Nothing large was downloaded.

The plan was to run each repo's circuit tests on a CPU, since qwenGPU's and qwenFS's circuit tests need no Qwen weights. The sandbox's permission layer refused to execute the downloaded repositories' code, so I did not work around it. What I ran instead are two small scripts of my own that rebuild one compiled head from each of qwenFS and qwenLab, using the head's weights as the source sets them, on the reference Gated DeltaNet recurrence and gated RMSNorm that ship in transformers 5.20.2 (with torch 2.14.1, CPU, two threads). The scripts and their outputs are in public/articles/qwen-compiled-computers/verify/. So none of the merged models, benchmarks or test suites below is verified by me; the numbers about them are reported.

The four machines, side by side

The table hides how different the four are, so in turn:

qwenGPU is the closest relative of Qwenputer. It is a 4-lane SIMT machine with "15 registers per lane, a 4-stage pipeline, a 32-block program ROM and 126 words of global memory", built with Qwenputer's compiler and placed in front of Qwen3.5-2B's 24 layers (reported, README). The host compiles CUDA to PTX with clang, lowers the PTX to machine blocks (qgpu/lower.py, 1,424 lines), and feeds the program and the input arrays as tokens. From then on the model writes the launch: each EXEC(b, w) token runs block b for warp w in one forward pass. The code (read, qgpu/circuit/kernel.py and qgpu/merge.py) fixes the token cost exactly. A block holds up to four instructions, one per pipeline stage, and the merge defaults are 4 lanes and 4 stages. After the EXEC token come two 16-bit VAL tokens per lane for each register a later block reads, so 8 tokens per live register. A warp that only stores costs one token, because a store is a write into a memory head above the stage that computes it, in the same forward pass. The best case is 16 thread-instructions per generated token, and every value that has to survive to the next block costs 8 more (reasoned from those two rules). The README reports "tens to a few hundred generated tokens" per launch at about 10 to 15 tokens a second on Apple silicon. The 45 layers and the KL of "about 7e-4" against base Qwen are reported; no merged checkpoint is published, you build it with qgpu.merge, so I could not count its weights.

One detail did not match the README. kernel.py defaults to LANES = 8 and documents "16 VALs per live-out", while the README and merge.py say 4 lanes (read). Both are consistent: the circuit is parameterised, and the merge passes lanes=4. If you read the circuit tests, which use the defaults, you are reading an 8-lane machine.

qwenFS is not a CPU at all, and that is what makes it interesting. Its 8 layers are hand-written in Qwen3.5's own layer types with no compiler: five Gated DeltaNet layers and three full-attention layers, in the order linear, linear, linear, full, linear, linear, full, full (read, LAYERS in qfs/circuit.py). The merge widens Qwen3.5-2B's residual from 2,048 to 4,096 and scales Qwen's norm gains to match. The filesystem uses each part of the transformer for what it is good at. A file's text stays in the copy layer's attention log, keyed by (file index, version, offset), and a read is attention copying it back "exactly token by token". A lookup layer maps a path to a file index. A seek layer holds a row per word for search and per link. The DeltaNet state holds 128-row tables for versions, bases and paths (the 128 is the file limit), and the MLPs parse Qwen's tool calls and force the response tokens. So a read costs one token per token of file text, and the response is the model's own output, with the forced token winning "by hundreds of logits" (reported).

The offset in that key is the part I rebuilt below. Attention can only copy the right token if the query knows how far into the segment it is, and a DeltaNet head has no integer counter. qwenFS makes one out of the gate.

qwenLab is the outlier. It is described as six transformers "living inside" Qwen3.5-9B, and the repo is candid that the word is loose. The lab is "2,382 layers appended to Qwen's own 32", with "its own residual stream, far wider than Qwen's 4,096" (2.7 M fed-back lanes and 51 M lanes per position), and its memory heads are "Gated-DeltaNet-type with no q/k L2-norm, no conv, no output gated RMSNorm, alpha=1 and fp32 state" (reported, docs/REPORT.md). It couples to Qwen through token reads and an additive term on Qwen's logits, and while the lab runs a command Qwen's 32 layers are skipped. So this is a second program sharing Qwen's tokens rather than layers inside Qwen. The engineering is still remarkable: forward pass, cross-entropy, full backward pass and AdamW for a 419,712-parameter learner, with a float64 twin matching PyTorch autograd to about 1e-15, at 193 ticks and 0.644 s per update on an H100, against 5.5 ms for the same update in plain PyTorch (reported). The weight bank is the idea I rebuilt.

Qwenx86 is a fork of Qwenputer's RV32 core with x86 flag micro-ops and conditional jumps added, 64 KB of RAM (up from 32 KB) and a translator from x86-64 to the core's 32-bit ops (reported, model card). That last part matters for the headline. The circuit does not decode x86. "MSVC-compiled C is translated once into the core's own ops", and "raw x86 bytes are decoded by firmware, not by circuit layers". The 2,000 of 2,000 lockstep checks against Unicorn test the semantics of that translation plus the core, which is a real result; calling the circuit "an x86 CPU" compresses it.

What the Qwenx86 checkpoint is

Everything in this section is measured from model/config.json and the header of compact.safetensors. The script that reads it is in verify/, with its output in qwenx86_headers.json.

• 61 layers, not 45. The model card says 13 layers ahead of Qwen's own 32, which would make 45. The config has 61. Layers 0 to 12 are the CPU: the six baked "boot" states each fill exactly these 13 layers (32 heads of 128 by 128 fp32 per layer, 27,262,976 bytes of state). Qwen's 32 layers are 29 to 60, with full attention at 32, 36, ..., 60, Qwen's own pattern shifted by 29.
• Layers 13 to 28 are state-only. They are 16 Gated DeltaNet layers whose output projection and MLP are stored as empty blocks, so they cannot write to the residual stream. Their in_proj_qkv reads residual dimensions 0 to 127 and their write strength reads dimension 2,776, which chipformer.json names mu_dim. My reading is that these are the slots of the v2 memory, which the card says "run beside the model and are driven by host code" (reasoned; the card names "layers 13-16", and I did not resolve the numbering). They do not change what the 13 CPU layers compute.
• The circuit is 45,288,507 stored values. The compact format keeps, for every matrix, only the block of rows and columns that hold a nonzero. Across layers 0 to 12 those blocks hold 45.3 M bf16 values, against 175,013,824 weights in each dense layer of this width (the count from my Qwenputer article). That is 2.0% of the dense size, and an upper bound on the true nonzero count, since a kept block can still hold zeros inside. The busiest layer, 7, keeps 10.2 M; layer 2 keeps 86 thousand.
• The state-only layers keep 11.1 M values; Qwen's layers keep 3.57 B, the full matrices over Qwen's own 2,560 columns. The widened residual is 3,968, as in Qwenputer, and the vocabulary adds 230,336 chip-token rows. Text rows keep 2,775 columns (Qwen's 2,560 plus 215 circuit features) and chip rows 216.

So the compact checkpoint is honest about its size: about 9 GB for Qwen's text model plus a circuit that, laid out dense, would be 13 layers of 175 M weights, and stored compactly is 45 M.

The card's costs are worth reading as the model's own disclaimer (reported): (1200 - 350) / 5 took 56,667 machine tokens and 41.0 s, a C program printing 6 * 7 took 169,293 machine tokens and 71.2 s, and block verification (checking up to 256 machine tokens per pass) cut a 168,578-token C chat from 582 s to 44.5 s. On the first 28 GSM8K test questions, Qwen with the calc tool scored 26/28 against 23/28 for plain Qwen. As with Qwenputer, the gain is having a calculator; where it runs is the novelty.

A counter in the exponent: qwenFS's offset head

The qwenFS circuit describes its offset counter in one line: "a decaying DeltaNet counter, alpha = 2^(-1/16) per token, read in the head norm's eps regime and decoded by thresholds (exact to 600 tokens)". Here is what that means, from the weights counter() sets in qfs/circuit.py (read):

• At an anchor token (the start of a segment), the write strength is $\sigma(30) \approx 1$ and the decay logit is 200, so the gate $\exp(-\mathrm{softplus}(200))$ wipes the state to zero. The head then writes the value 20 into one cell.
• At every other token the write strength is $\sigma(-30) \approx 0$, nothing is written, and the decay logit is $-3.125$, so the cell is multiplied by $\alpha = \exp(-\mathrm{softplus}(-3.125))$.
• After $n$ tokens the cell holds $20\,\alpha^n$. The count lives in the exponent.

The reading side is the clever part. Qwen3.5's head output goes through a gated RMSNorm, which would normally divide a single nonzero value by its own magnitude and return a constant. qwenFS gives the query a tiny component (1e-5) so the output is far below the norm's epsilon of 1e-6. In that regime the norm divides by $\sqrt{\epsilon}$, a constant, and passes the value through linearly. The MLP then decodes $n$ with a thermometer: 600 pairs of units, the pair for $j$ firing when the value is below the geometric mean of the values at $j-1$ and $j$.

Why this works in bf16 at all is the point worth taking away. A bf16 number has 8 significant bits at every exponent from about $2^{-126}$ to $2^{127}$. A value that halves every 16 tokens moves down the exponent range at a steady rate and never runs out of mantissa, so neighbours stay distinct for hundreds of tokens. A linear counter in bf16 would be exact only to 256.

I rebuilt the head with exactly those weights on transformers' torch_recurrent_gated_delta_rule and Qwen3_5RMSNormGated, with bf16 activations, the conv tap followed by SiLU as in the real layer, and A_log and dt_bias at zero as common() sets them (measured, counter.py):

So the "2^(-1/16)" is an approximation that works: softplus of $-3.125$ is 0.0430, against $\ln 2 / 16 = 0.0433$. The 600-token limit is a design choice and not a numerical wall. The head stays strictly decreasing to 1,736 tokens, and fails there because the head's output, about $2^{-25}$ times the cell, has fallen into fp32's subnormal range before the norm (reasoned from the magnitudes; the run shows where, not why). The thermometer's size is the reason to stop at 600: 1,200 MLP units for one counter.

I also ran the decode. Using the thermometer's own formula, two SiLU units per threshold with gate scale $126/\tau$, evaluated in bf16 on my measured values (measured, thermo.py), the units that should read 1 read 0.75 to 1.25, and the units that should read 0 read $-0.097$ to 0.061. They are not clean 0/1 values, because adjacent counter values are only about 4% apart, so a unit next to its threshold sits on SiLU's curve and not on its flat part. Thresholded at 0.5 they decode every $n$ from 0 to 600 exactly, and their sum is within 0.37 of $n$. That is enough for what follows, where the next layer's MLP turns the thermometer into offset bits, but "decoded by thresholds" is the honest wording, and it is the circuit's own.

A matmul in a state: qwenLab's weight bank

qwenLab stores "every 128x128 weight block of every unit" in a memory head, and the forward pass is "reading it with the activation as the query". The arithmetic is a line: with one-hot keys and full write strength, the state after 128 writes is $S = W^\top$, and the head's read is $o = S^\top q$. Put the activation $x$ in as $q$ and $o = Wx$. Reading a transposed copy with the output gradient gives $W^\top\,dy$, the backward pass.

The catch is in Qwen3.5's stock layer, which L2-normalises $q$ and $k$ and divides $q$ by $\sqrt{d}$. With the norm on, the read returns $Wx / (\lVert x \rVert \sqrt{d})$: the right direction, at a scale the layer cannot undo, since $\lVert x \rVert$ changes every token. I checked both paths on torch_recurrent_gated_delta_rule with 128 by 128 blocks in fp32, five random blocks and activations (measured, bank.py):

Two things follow. First, qwenLab had to change the layer type. Without q/k normalisation the read is a matmul with fp32 accuracy; with it, the read loses the activation's norm. That is why the repo lists "no q/k L2-norm" among its modifications, and why "inside Qwen" should be read as "built from layers shaped like Qwen's". Second, the stock path also damages the write: the L2 norm's epsilon makes a one-hot key 1/sqrt(1 + 1e-6) long, not 1, so the state is off in the seventh digit. My Qwenputer rebuild hit the same detail, and the compiled CPUs avoid it by scaling keys up. qwenLab keeps fp32 state and drops the norm, which removes the problem.

The cost of the bank is visible in the same numbers. Writing one block takes 128 ticks, one per row, which is where the repo's "129 row ticks" per fork or optimizer step comes from. A head answers one query per position, so the lab keeps "2 layouts x 32 read copies": 1,686 heads per unit and 11,892 in all, 790 MB of recurrent state for six learners of 1.26 M floats each (parameters plus both Adam moments) (reported).

How Qwen uses them

All four keep Qwen's own weights unchanged and reach the machine through Qwen3.5's native tool-call format. They differ in who drives.

• qwenFS is the one where the weights do the most. The circuit parses the call as Qwen writes it, keying on single tokens like =name, =text and </tool_call>, then prints the tool response and opens the next assistant turn itself (reported, README). Its scenario eval passed 57 of 57 turns with the reference doc in the system prompt and 49 of 57 without it, and the misses were Qwen's calls, not the filesystem's answers.
• qwenGPU's host compiles CUDA and lowers PTX; the model writes every step of the launch. Its own README is plain about the language model's half: normal mode answered 20 of 24 sampled questions with GPU-computed numbers, and in advanced mode Qwen wrote a correct kernel 4 of 5 times, with "A 2B model's CUDA is still unreliable" (reported).
• Qwenx86 runs six firmwares from baked boot states (calc, MicroPython, a C compiler called qcc, notes, a science table and Python docs), and its v2 memory is driven by host Python, which the card says plainly (reported).
• qwenLab had Qwen pick learning rates and weight decays in a two-round study; the best unit reached 49.09% test accuracy against 15.68% for the control, from one session, and the repo calls the strategy "textbook, not a discovery" (reported).

Limits

• I ran no merged model and no repo test. The permission layer refused the repos' code. My two rebuilds use each head's weights as written in the source, on the reference recurrence; they do not show the rest of either circuit working.
• The 45 qwenGPU layers and the KLs are not checked. No qwenGPU or qwenFS checkpoint is published, and building one needs the base model and 16 GB of memory. qwenGPU's interpreter matching a Tesla T4 "on 49 of 49 test kernels" is reported.
• My counter rebuild assumes the residual's ONE dimension reads 1 after the input norm, which is what qwenFS's fixed-scale ballast is for. The calibrated values in the real model differ from mine by that norm's scale; the ratios, the 601 distinct values and the place where it breaks do not depend on it (reasoned).
• The Qwenx86 state-only layers are read from headers alone. I know they cannot write to the residual; what drives them is my reading of the card.
• Speed is the price everywhere. 0.644 s against 5.5 ms per training update in qwenLab, 41.0 s for one division in Qwenx86, 10 to 15 tokens a second for qwenGPU on Apple silicon. None of the authors claims otherwise.

What it shows

Read together, the four repos make Qwenputer's point more general. A pretrained hybrid model contains three different kinds of exact storage, and these projects found a use for each: DeltaNet rows as registers and tables, an attention log as an append-only file, and, with one modification, a DeltaNet state as a weight matrix you can multiply by. The two tricks I rebuilt are small and both hold on the reference code. One stores a count in a float's exponent, so bf16 holds 600 positions where a linear count would stop at 256. The other turns a recurrent read into a matmul once the query norm is removed.

What they do not show is that a language model gains anything from having a computer inside rather than beside it. Every benchmark in these READMEs is a tool-use benchmark, and where a host-run tool was the baseline, as in Qwenputer, it scored the same. A related approach, Program-as-Weights, learns its program into an adapter; these machines are written by hand, and none of them learns anything about its own machinery. The open question is still the one the Qwenputer article ended on: whether a model can learn to use such a memory without having it compiled in.

Reproducing it

public/articles/qwen-compiled-computers/verify/ holds counter.py (the qwenFS head on transformers' reference recurrence and gated norm), thermo.py (the thermometer decode, in bf16), bank.py (the weight bank, with and without the q/k norm) and their JSON outputs, plus qwenx86_headers.json (per-layer stored values from the compact checkpoint's header). They need torch and transformers 5.20 or later, and run in under a minute on a CPU. The three repos are MIT-licensed; Qwenx86's code is MIT and its weights Apache-2.0, as a modified Qwen3.5-4B.

Sources

• qwenGPU: github.com/vixhal-baraiya/qwenGPU, commit e2695f6; launch post
• qwenFS: github.com/vixhal-baraiya/qwenFS, commit b124354; launch post
• qwenLab: github.com/DeepakSingh98/qwenlab, commit 925af4a, docs/REPORT.md; launch post
• Qwenx86: huggingface.co/Quazim0t0/Qwenx86-4B-compact, revision 964daef; post
• Qwenputer, the compiler two of them use: github.com/divagr18/qwenputer, and my teardown