llama-modchip: a 284B MoE in 64 GB of RAM, and what actually makes it fit

Two posts from the Chinese creator @mingzi206 landed on 12 November. The first introduced synchopate/llama-modchip, a patched llama.cpp, with the line "no GPU, 64 GB of RAM, a 284B model". It listed 56.9 GB for DeepSeek-V4, 9 to 14 tok/s, up to 25 tok/s with speculative decoding, 99.15% cosine similarity, a 13.3 GB 30B model that "runs on a 16 GB laptop", and 111 tok/s. Three hours later he quoted it with a video: "it really works. On a free 8-core, 32 GB CPU-only machine I got NVIDIA's modified Nemotron 30B-A3B VQ3-64 running."

A 284B model in 64 GB sounds like a trick with the disk: map the file, let the page cache hold the hot experts, read the rest from NVMe on demand. I expected that and went looking for the mmap path. There isn't one. The repo loads every weight into anonymous heap (--load-mode none --repack) and says "zero disk streaming" in so many words. The 284B fits for a duller reason, and finding it took reading the checkpoints rather than the README.

I did not run the code. I parsed the headers of seven GGUF files from three published checkpoints with HTTP range requests (a GGUF header lists every tensor's name, shape, type and byte offset, so sizes fall out of consecutive offsets) and counted bytes. Everything below that says "the header" comes from that.

Two different sizes: what a model holds, and what a token reads

A mixture-of-experts layer replaces one feed-forward block with many, plus a router. For each token the router scores every expert and runs the top few. DeepSeek-V4-Flash, per its config.json, has 43 layers, 256 routed experts per layer, one shared expert, and num_experts_per_tok: 6. Each expert is a SwiGLU block of width 2048 on a hidden size of 4096, so three matrices of 4096 x 2048:

The header agrees: 284.335B parameters in all, 277.025B of them in the ffn_*_exps tensors. Add attention, the shared expert, the output head and one embedding row, and a token touches 13.27B parameters, close to the 13.17B the README quotes.

Those two totals answer different questions. Total parameters set capacity: every expert must live somewhere you can reach in time, because the router may pick any of them next token. Active parameters set bandwidth: on a CPU, decoding one token is a sequence of matrix-vector products, each weight is read once and used for two flops, and no CPU has the memory bandwidth to make those flops the bottleneck. Time per token is, to first order, bytes read divided by bytes per second.

So two separate numbers decide whether a 284B model is usable on a desktop. Does the whole file fit in RAM? And how many bytes does one token pull through the memory bus?

Where the 56.93 GiB goes

The default 284B checkpoint is DeepSeek-V4-Flash-0731-MaxQ-Fractal, three shards totalling 61,129 MB, the README's 56.93 GiB. By tensor type, from the header:

Routed experts are 92.7% of the file and average 1.64 bits per weight. The whole file averages 1.72. That is the answer to "how does 284B fit in 64 GB": 225.5B parameters at 1.5625 bits is 41 GiB, and nothing else is large.

IQ1_S, IQ1_M and IQ2_XXS are upstream llama.cpp formats, the same family Unsloth used for the UD-IQ1_S build the README itself lists at 76.87 GiB. What modchip adds for the 284B is choosing which layers get the 2-bit treatment, an importance weighting for the 1-bit ones (the MaxQ build uses @antirez's 1.5M-token importance matrix; an earlier build used a closed-form "SwiGLU-H2" weighting from the product of each gate and up row norm), and moving attention from Q8_0 to its own 3.75-bit Fractal-64. The README credits the re-quantized experts and the attention transcode together with taking the Unsloth build from 76.87 GiB to 56.56 GiB. The attention part is the smaller share: 4.9B parameters at Q8_0's 8.5 bits are about 4.85 GiB against 2.14 GiB now, so roughly 2.7 of the 20.31 GiB saved. The rest came out of the experts.

The format the repo is named for, LUT-VQ3-64 (GGML_TYPE_VQ3_0 = 43), does not appear in the MaxQ header. Not one tensor of type 43. Per the README it ships in the separate 56.40 GiB "VQ3" 284B build, on 68 shared-expert matrices (I did not parse that one), and in the Nemotron 30B, where the header shows it. The post that credits VQ3 with the 284B result has the cause wrong.

What a token reads

Capacity explains the fit. Speed needs the second number. Per token, the runtime reads every non-routed tensor once, one row of the embedding table, and 6 of 256 slices of each routed tensor. Applying that to the header's byte counts:

The experts dominate the file and are a quarter of the traffic. Squeeze them to 1.6 bits and you have solved capacity, but the bus spends most of each token on the 7% of the file that every token touches. The attention projections alone, attn_q_b, attn_output_a and attn_output_b, are 2.03 GB a token. This is why the repo's kernel work concentrates on the 3.75-bit Fractal-64 matmuls for attention rather than on the experts.

Nemotron-3.5-30B-A3B shows the same shape at a smaller size. Its config is a hybrid: 52 blocks, 23 of them MoE (128 experts, top 6, each expert an up and a down matrix of 2688 x 1856 with a squared-ReLU between, 9,977,856 parameters), 23 Mamba-2 and 6 attention. In the VQ3-64 GGUF the routed experts are 88.5% of the file and 29.1% of a token's 2.04 GB. The Mamba layers read more per token (0.631 GB) than the experts do (0.592 GB).

The widget holds every per-token byte count from the headers. Pick a checkpoint and a memory system and it divides. Switch the experts to "mmap from an SSD" and it charges the expert reads that miss the page cache at SSD speed.

The arithmetic of a ceiling

The README's benchmark machine is a Ryzen 9 9955HX, 16 Zen 5 cores, two channels of DDR5-5600. Peak bandwidth is channels times bytes per transfer times transfers per second: $2 \times 8 \times 5600\text{M} = 89.6$ GB/s. The README says about 75 GB/s sustained, which is a believable 84% of peak. Then:

The README measures 8.75 tok/s for the MaxQ build, single stream, all six experts, no speculation. That is 62% of the bandwidth ceiling. For Nemotron, 2.04 GB over 75 GB/s is 27.2 ms, a 36.8 tok/s ceiling, against a measured 22.44: 61%. Two models, two formats, the same fraction. On a CPU that is respectable. The remaining third goes to dequantization arithmetic, thread synchronization across 43 layers of small matmuls, and the parts of a token that are not weight reads.

The repo's own profile in docs/DEEPSEEK_V4_OPTIMIZATIONS.md labels the decode read as "3.20 GB -> 2.85 GB" per token. I cannot get there from the headers. The same document puts the wo_a and wo_b pair at 991.6 MB a token; the header says those 86 tensors are 2.886B parameters at 3.75 bits, 1.35 GB. The runtime repacks weights at load, so on-disk bytes are a proxy, but repacking a 3.75-bit block does not make it 27% smaller. If their 3.20 GB were right, the ceiling would be 23.4 tok/s and the measured speed 37% of it, which would say the kernels are far from the bus. My count says the kernels are close to where llama.cpp usually lands. Either way the measured number sits under the ceiling, so nothing is impossible here. The headers are public; anyone can redo the count.

The post's "9 to 14 tok/s" mixes two modes, and the bus says which. At 5.36 GB a token, 9 tok/s pulls 48.2 GB/s through memory: 64% of the 75 GB/s the desktop sustains, 54% of its 89.6 GB/s peak. Fine. 14 tok/s would pull 75.0 GB/s, every byte the bus sustains, with nothing left for activations, the KV cache or the operating system. A single stream cannot do that on two channels of DDR5. The README never claims it: its single-stream range is 8.75 to 9.55, and 10.74 with optional top-4 expert pruning, which reads fewer expert bytes and measurably changes the output. The 14 matches the repo's 14.69 peak with speculative decoding on, in ARCHITECTURE.md. Two more numbers in the post do not belong on this axis at all. The "25 tok/s" is the README's 25.3, a speculative-decoding burst: a drafter proposes tokens and one forward pass verifies several, so weights are read once for several tokens and a per-token bandwidth ceiling no longer applies. The "111 tok/s" is 111.62 on a 64-core Threadripper PRO 7985WX with eight channels of DDR5-6000 (384 GB/s peak, 188 tok/s ceiling for Nemotron by the same arithmetic), warm, with the speculative drafter, running the 30B. It is a real number about different hardware, a different model and a different decoding mode.

The first post, claim by claim

Satyajit reposted the first of the two posts as well, so here is every claim in it against the repo. Where a phrase is the poster's and not the README's, the table says so.

The poster's framing ("a few eight-GPU H100 servers" for the full model) is about BF16 serving; at 568.6 GiB in BF16 that is fair, and it is also why the comparison means little. The 284B here is a 1.72-bit model.

Why nothing streams from disk

The obvious design for a model bigger than RAM is mmap: map the file, let the kernel fault pages in on first touch, and let the page cache keep whatever was used recently. Upstream llama.cpp does this by default. For an MoE it is attractive, because each token touches 6 of 256 experts per layer and the rest can sit on disk.

The arithmetic is unkind. In the MaxQ build one expert's three matrices in one layer average 5.15 MB, and a token reads 258 of those slices (6 experts, 43 layers), 1.33 GB. A PCIe 4.0 NVMe drive reads about 7 GB/s sequentially and less for scattered 5 MB reads. Put all the experts on disk with nothing cached and a token costs 53.8 ms of dense reads from RAM plus 190 ms of expert reads from the SSD: 4.1 tok/s. A 90% hit rate in the page cache brings that back to 11.3 tok/s, still under the resident 14.0.

The counterfactual that matters is the one where you keep more bits. DeepSeek ships the routed experts in FP4 ("expert_dtype": "fp4"); at MXFP4's 4.25 bits the routed tensors alone are 147.2 GB, and the file would be about 151.6 GB. That does not fit, so streaming would be mandatory. RAM could cache roughly 41% of the experts. If routing were uniform, about 40% of expert reads would hit, and a token would run at 2.7 tok/s on the same desktop, and even with everything resident it would only reach 10.0. Squeezing experts to 1.6 bits buys both: the file fits, and the expert slice of each token shrinks from 3.45 GB to 1.33 GB.

Choosing anonymous heap over mmap has a cost the README is candid about. Page-cache pages are clean and the kernel can drop them under pressure; anonymous pages cannot be dropped without swap. The MaxQ build allocates 59.13 GiB of resident memory at a 32K context, and a 64 GB Linux box has about 62.5 GiB usable. The launch script forces -c 32768 -ub 512 on hosts of 68 GB or less and suggests zram or closing the browser. That is a 3.4 GiB margin. Treat "runs on 64 GB" as "runs on a 64 GB machine that does nothing else".

The streaming design does exist elsewhere. Edge0 streams a 35B MoE off SSD, and the bytes it cannot stream are the same dense layers counted here. A 200 GB MoE on one RTX 3090 keeps experts in eight-channel DDR4 and found the CPU's dequantization arithmetic, not the bus, setting the speed. modchip sits at the far end of both: no GPU, no disk, everything in DRAM, as few bits as the model survives.

Vector quantization, from the scalar case up

The format with its name on the repo does real work in the Nemotron build, so it deserves a proper look.

A scalar quantizer rounds each weight on its own. Q4_0 takes a block of 32 weights, stores one FP16 scale $d$, and replaces each weight $w_i$ with a 4-bit integer $q_i$ so that $w_i \approx d \cdot q_i$: 18 bytes for 32 weights, 4.5 bits per weight. Go to 3 bits and each coordinate gets 8 levels. Three consecutive weights then live on a grid of $8^3 = 512$ points spread evenly through a cube, and spending 9 bits on that triple buys one of those 512 points.

The waste is geometric. Weights are roughly bell-shaped and heavy-tailed, so a triple of them is far more likely to sit near the centre than in a corner of the cube. The README makes the volume argument: a ball inscribed in the cube $[-M, M]^3$ has volume $\tfrac{4}{3}\pi M^3 \approx 4.189 M^3$ against the cube's $8 M^3$, so 47.64% of the cube lies outside it, and a uniform grid spends that share of its points where triples rarely land.

Vector quantization quantizes the triple as a unit. Instead of 512 grid points, keep a codebook of $K$ points placed where triples actually fall, and store the index of the nearest one. Lloyd-Max is the classical way to place them: alternate between assigning every training triple to its nearest centroid and moving each centroid to the mean of its triples, which is k-means with a quantization goal. With $K = 128$ the index is 7 bits for three weights, 2.333 bits per weight before scales.

LUT-VQ3-64 packs 64 weights into a 24-byte block_vq3_0:

• 16 bits: an FP16 block scale $d$, multiplied at decode by a constant GGML_VQ3_G = 8.6373016238e-3.
• 140 bits: twenty 7-bit indices covering 60 weights, three at a time, at stride 22 (weights $j$, $j+22$, $j+44$).
• 12 bits: two 6-bit indices for the 4 weights left over, which the README describes as indices into a 2D table.
• 24 bits: eight 3-bit sub-scales, one per group of 8 weights, read as a multiplier $m_b = 2 + u_b$, so 2..9.

$16 + 140 + 12 + 24 = 192$ bits, 24 bytes, exactly 3.000 bits per weight. A weight decodes as $w = d \cdot G \cdot m_b \cdot c_{\text{idx}}[j]$, where $c$ is an int8 codebook entry.

The decode trick is where this becomes a CPU format. The 128 entries times 3 int8 coordinates are 384 bytes: six 64-byte AVX-512 registers, one pair per coordinate. AVX-512 VBMI's VPERMI2B (_mm512_permutex2var_epi8) treats a pair of registers as a 128-byte table and looks up 64 bytes of it at once with 7-bit indices. So three permutes expand a block's indices into int8 weights without touching memory, and the existing int8 dot-product path takes over. The patch patches/lut-vq3-64-avx512-vbmi.patch does exactly this with _mm512_mask2_permutex2var_epi8, and falls back to AVX2 and scalar paths on CPUs without VBMI.

The widget draws the table as shipped in vq3/vq3_codebook.h. Two README claims about it do not survive contact with the file. The README calls the codebook "antipodally symmetric $c_{127-k} = -c_k$". In the shipped table no entry's negation is in the table at all, let alone at index $127-k$; the set is balanced (the coordinate means are 0.09, -0.69 and -0.95, and each octant holds 15 to 17 points) but not symmetric. The README also says the centroids sit "on the populated spherical/cross-polytope shell inside" the cube. Measured from the origin, 80 of the 128 entries lie outside the ball of radius 127 and the largest reaches 205.7, close to a corner at $127\sqrt{3} \approx 220$. The codebook spreads into the corners. With per-group scales doing the radial work, that is a reasonable place for it to be; it is just not what the prose describes.

Fractal-64, the other format, is a scalar 3-bit quantizer with a non-uniform level set {-14, -8, -4, -1, +1, +4, +8, +14} and eight 4-bit sub-scales: $16 + 32 + 192 = 240$ bits per 64 weights, 3.75 bits per weight. Eight levels fit a VPSHUFB lookup. It is the format on the 284B's attention and on all of Nemotron's down projections.

One check ties the formats to the files. The Nemotron VQ3 build puts 12.133B parameters (19 interior ffn_up_exps) in VQ3 and the rest of the experts in Fractal-64. Moving those 12.133B from 3.75 to 3.00 bits should save $12.133\text{B} \times 0.75 / 8 = 1.14$ GB, which is 1.06 GiB. The README's two Nemotron files are 14.36 and 13.30 GiB: 1.06 GiB apart. The format does what its block layout says.

What the quality numbers measure

The post's "cosine similarity 99.15%" reads like a statement about the 284B's output. It is something narrower. The README measures it by quantizing and dequantizing one tensor, blk.1.ffn_up_exps.weight, from a "59.0 GiB BF16 master checkpoint". 59.0 GiB is the Nemotron 30B in BF16 (the 284B in BF16 is 568.6 GiB), and 99.15% belongs to Fractal-64 on that tensor; VQ3 scores 97.78% on the same test. The numbers are self-consistent: a signal-to-quantization-noise ratio of 17.75 dB is a power ratio of 59.6, a relative error of $1/\sqrt{59.6} = 12.95\%$ against the README's 12.96%, and a cosine of $1/\sqrt{1 + 1/59.6} = 0.9917$. Weight reconstruction is the right first test for a format. It says nothing about what a 1.6-bit 284B model writes.

For that the repo has perplexity, KL divergence and a 100-point gate, and each has a limit:

• The 284B has no full-precision reference. BF16 does not fit the test machine, so the KL divergence of every 284B build is measured against the MaxQ build. MaxQ scores 0.0000 by definition.
• The VQ3 284B build is clearly worse. Against MaxQ it has a KL divergence of 0.8302 nats per token with 75.0% top-1 agreement, and WikiText-2 perplexity of 12.2345 against MaxQ's 9.3896, on 8,192 tokens. The README still lists it as "compact safety-aligned". It is 0.53 GiB smaller.
• The 100-point gate does not separate anything on the 30B. It is 30 points of tool calls, 30 of code generation, 20 of needle-in-a-haystack and arithmetic, and 20 of throughput and memory telemetry. BF16, Q4_0, Fractal-64, VQ3 and the abliterated VQ3 all score 100.0 / 100.0. A test every candidate aces cannot rank them. The perplexity column can: 7.3610 for Fractal-64, 7.4400 for VQ3, 7.2927 for MXFP4.

On the 284B the gate does spread out (MaxQ 70/80, VQ3 64/80) and the harder 60-point suite more so (40 against 30). That is the evidence I would weigh, and it favours the build without the VQ3 tensors.

The free 8-core box in the video

The quoted post's run is the more interesting of the two, because it is the cheap case. He named the machine in a reply: when someone asked where to get an 8-core, 32 GB CPU instance, he linked his earlier post about ModelScope's free notebooks. His screenshot of the console shows it.

expand

His coding assistant, in the first seconds of the video, gets the model's provenance right where the post does not: "NVIDIA's Nemotron 3.5 Lightning model, a CPU quantized version made by the third party Synchopate". NVIDIA publishes the BF16 weights; nobody at NVIDIA modified anything. The same screen gives the decode speed as "we measured about 11.3 token/s".

expand

The request panel in his client logs a token count, a time and a rate per call, and they don't divide (333 tokens in 4.5 s is shown as 22 tok/s), so I would not read a decode speed off it. The server ran with an 8K context, and an earlier call failed with "request (125334 tokens) exceeds the available context size (8192 tokens)": a coding agent's prompt does not fit.

expand

Take the 11.3 tok/s at face value. At 2.04 GB a token that is 23.0 GB/s of weight traffic, plausible for a shared cloud host. I don't know whether the instance has AVX-512 VBMI; without it the VQ3 tensors take the AVX2 fallback. Either way, a 30B MoE holds a conversation on a free CPU instance at reading speed, as long as the prompts stay short.

The launch animation

Every panel is typed on at a fixed cadence and the telemetry is static. As marketing it is fine. As evidence it is not.

Run the MaxQ build, skip the VQ3 one

The README packs a dozen invented names (RouteCrack, Stockfish-PVS, HARP-Cache, SwiGLU-H2) into every table cell, and several of its derivations do not match its own files. Under that is competent engineering: a model that fits 64 GB at 1.6 bits, bits spent where tokens read them (1.56 on experts, 3.75 on attention, 6.56 on the shared expert and head), and about 62% of the memory bus on both models.

If you have a 64 GB desktop that does nothing else, the MaxQ build is a real way to run DeepSeek-V4-Flash at 8 to 9 tok/s, and its decode floor is set by attention, not experts. If you have 16 to 32 GB, the Nemotron VQ3 build is the more interesting artefact; it is 1.06 GiB smaller than Fractal-64 for a 1.07% perplexity cost. I would skip the VQ3 284B build, which loses a quarter of top-1 agreement to save half a gigabyte. And I would not quote the 100/100 gate for anything.

So a 284B MoE runs from 64 GB because its experts are compressed until they fit, not because they stream. For the same bandwidth reasoning elsewhere, see CPU performance engineering, PS5LM on a console's memory bus, and DeepSeek-V4.1-Flash-1017, whose UD-IQ1_S build shows the same split: 1.80 GB of experts against 6.60 GB of everything else per token.