# Colibri: running a 744B model on a 25 GB machine by streaming experts from disk
> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.satyajit.si/articles/colibri
> date: 2026-07-10
> tags: inference, mixture-of-experts
[GLM-5.2](/articles/glm-5-2) is a 744-billion-parameter model — backbone-only; counting
its multi-token-prediction layer it is 753B, which is why some write-ups quote that instead. Loaded the normal way, its
weights want hundreds of gigabytes of fast memory — data-center territory. [Colibri](https://github.com/JustVugg/colibri)
is a single-file, pure-C engine, zero external dependencies, that runs that same model on a
consumer machine with about **25 GB of RAM**. Not a distillation, not a smaller sibling — the
full 744B checkpoint, answering correctly, on a box that costs less than one H100's cooling fan.
**Updated August 2026.** This piece was written in July, when 0.37 tok/s was the best result anyone
had reported and the project was a single-file C engine for one model. Both are now out of date: the
repo reports **5.8–6.8 tok/s on 6× RTX 5090** with the experts fully resident, supports DeepSeek V4
Flash, Qwen3.6 and OLMoE alongside GLM-5.2, and has grown GPU backends, NUMA placement and dual-SSD
striping. The floor is unchanged and so is the thesis. [What changed](#what-changed-since-july) is at
the end.
Before the "how," the honest headline: **this is a feasibility feat, not a usable-speed setup.**
Cold, Colibri decodes at roughly **0.05–0.1 tokens per second** — that is *10 to 20 seconds per
token*. Warm, with every trick engaged, the best real community result is about **0.37 tok/s**,
still ~3 seconds per token. And "25 GB of RAM" is only true if you *also* have ~**370 GB of fast
NVMe** for the experts. Keep both numbers in the same sentence or the claim is misleading.
**Read the fine print before you get excited.** (1) It is *slow* — 0.05–0.1 tok/s cold is 10–20
seconds **per token**; the best community case, ~0.37 tok/s, is still ~3 s/token. (2) "Runs on 25 GB
RAM" **requires ~370 GB of fast disk** for the experts — the RAM number alone is misleading. (3) The
numbers here are **community-reported / single real test cases**, not official benchmarks — treat them
as existence proofs, not spec sheets. (4) It works only because GLM-5.2 is an **extremely sparse MoE**
*and* because of **int4/int8 quantization** — remove either and the trick collapses.
## Why this is even possible: extreme sparsity
Colibri does not compress 744B parameters into 25 GB. It exploits the fact that, at any moment,
almost none of those parameters are doing work. GLM-5.2 is a
[Mixture of Experts](/articles/mixture-of-experts-from-scratch): each MoE layer holds **256 experts**,
but a router picks only a small **top-k** of them per token. Across the model, only about **40B of the
744B parameters activate for a given token**, and of those, only ~**11 GB of weights** actually *change*
from one token to the next — the routed experts. Everything else (attention, shared experts, embeddings —
the "dense" ~17B) is used on *every* token.
That split is the whole design. Split the model by how often each piece is touched:
- **The dense core (~17B params)** is touched every token → keep it **resident in RAM**, int4-quantized
to **~9.9 GB**.
- **The routed experts (21,504 of them: 75 MoE layers × 256, plus the MTP head, ~19 MB each at int4)**
are touched rarely and unpredictably → leave them **on disk (~370 GB)** and fetch only the handful a
token actually routes to.
The second enabler is quantization. The original FP8 checkpoint is ~**756 GB**; Colibri's offline
converter requantizes it to int4 (the experts) so the dense core fits in ~9.9 GB of RAM and the expert
store shrinks to ~370 GB on disk. Sparsity says *you rarely need most experts*; quantization says *the
ones you do need are small enough to stream*. You need **both**.
## The three-tier memory, made visible
So the memory hierarchy has three tiers: the **int4 dense core resident in RAM**, an **LRU cache of hot
experts in RAM**, and the **full expert store on disk**. When the router picks an expert, one of two
things happens. If that expert is already in the RAM cache (or pinned), it is a **hit** — served at
memory speed. If not, it is a **miss**: a disk read of ~19 MB that sits **on the critical path** of that
token. The token cannot finish until the bytes arrive.
Route a token through one MoE layer and watch it resolve. Scrub the token, resize the cache, toggle
hot-expert pinning, and watch the LRU fill and evict:
This is the entire performance story in one picture. The router picks its experts; the cache absorbs the
ones you keep re-using; everything else is a disk fetch. A cold token — nothing cached — reads about
**11 GB from disk** (75 layers × ~8 experts × ~19 MB). At a typical NVMe rate of ~1 GB/s, that read alone
is ~10 seconds, which is exactly why cold decode lands at 0.05–0.1 tok/s. **Decode is disk-bound**, not
compute-bound: the CPU sits waiting on I/O. This is the same memory-wall intuition as ordinary
[LLM inference](/articles/how-llm-inference-works), pushed to its limit — except the "memory" the decode
waits on is a spinning queue of NVMe reads instead of GPU HBM.
Colibri softens the disk with the usual systems tricks: an **LRU cache** so repeat experts stay hot, an
**async readahead** that reads the *next* block of experts while the current one is still multiplying (so
compute and I/O overlap), the **OS page cache** acting as a free second-level cache, and a **RAM safety
budget** — the cache is auto-sized from `MemAvailable` at startup so it fills spare memory without ever
triggering an OOM kill. None of these change the physics of a cold miss; they just make misses rarer.
## Budget the RAM and the disk together
Because the "25 GB" number is the seductive, misleading one, it is worth drawing to scale. On a 25 GB
machine the resident footprint is the 9.9 GB int4 core plus an auto-sized LRU expert cache plus a safety
headroom — and the 370 GB of experts sit on disk, roughly **15× larger than the entire RAM**. Slide the
cache size and see how little of the model is ever in memory at once:
That 15× gap is the point. Colibri did not shrink GLM-5.2 to fit in RAM; it arranged for only the
in-use ~4% to be in RAM at any instant, and made disk the backing store for the rest. Take away the fast
NVMe and there is no engine — which is why quoting the RAM figure without the disk figure sells a fiction.
## Buying speed back: warm cache, MTP, and pinning
Cold decode is the floor, not the experience. Three things stack on top of it, and it is worth being
precise about what each one is.
**A warm cache** is just locality: real prompts re-route to the same experts, so after a few tokens the
hottest experts live in RAM and the miss rate drops. **Pinned hot experts** make that permanent — Colibri
records which experts your usage actually routes to (a `.coli_usage` file) and pins the hottest ones in
spare RAM, so the engine *literally gets faster the more you use it*.
**MTP** is the subtle one. GLM-5.2 ships a native [multi-token-prediction](/articles/multi-token-prediction)
head — a lightweight draft model that proposes several future tokens, which the main model then *verifies*
in one batched forward. Colibri runs it natively at **int8** (this matters: at int4 the draft head's
predictions are so degraded that acceptance collapses to 0–4% and speculation never engages; at int8 it
reaches ~39–59% acceptance, community-measured). The payoff is **2.2–2.8 tokens per forward** — but note
that is a *speculation/acceptance rate*, the number of tokens you get out of one main-model pass, **not**
tokens per second. It amortizes the fixed per-forward cost; it does not make the disk faster. On a *cold*
cache MTP can even be a net loss, because verifying extra draft tokens routes to *more* experts
(~660 → ~1100 expert-loads/token) — so speculation only pays once the cache and pins are warm.
Stack them, and honestly label which rungs are measured versus an illustrative split:
The endpoints are real community numbers; the per-factor decomposition in the middle is illustrative
(Colibri's README reports the endpoints, not the split). The takeaway is the seconds-per-token column: the
best result available when this was written, **0.37 tok/s on a Ryzen AI 9 Framework 13** with a warm
cache, MTP and pinning, is still about **one token every three seconds**. (That ceiling has since moved
a long way — see [what changed](#what-changed-since-july) — though the streaming floor has not.) A different community machine — an M5 Max with **128 GB of RAM** —
reaches **1.06 tok/s**, but only because far more RAM lets far more experts stay resident, which just
confirms the thesis: *the disk is the bottleneck, and RAM buys you out of it.*
## Colibri is the engine, not the model
Keep the two things distinct. [GLM-5.2](/articles/glm-5-2) is the *model* — the 744B MoE, its routing, its
MTP head, its training. **Colibri is an *engine*** that runs that model's forward pass in pure C on tiny
hardware. The cleverness is entirely in the *systems* layer: how to lay out weights on disk, when to read
them, what to cache, how to overlap I/O with compute, how to quantize the head so speculation survives.
It reimplements GLM-5.2's forward pass faithfully — MLA attention with a compressed KV cache (~57× smaller
than dense), DeepSeek-V3-style routing, native MTP — but adds nothing to the model's *capability*. Same
weights, same answers; the contribution is fitting them onto a laptop.
## What changed since July
The version of Colibri this article was written against was a single-file C engine that ran one model. Six weeks later it is a different kind of project, and three of the changes bear directly on the argument above.
**The ceiling moved a long way; the floor did not.** The repo now reports **5.8–6.8 tok/s on 6× RTX 5090** with full expert residency (TTFT ~13 s), **~1.8 tok/s on a 128 GB CPU-only desktop**, and **1.07 tok/s on a single RTX 5070 Ti**. The 25 GB dev box is still 0.05–0.1 tok/s cold, described in the README as "the proven floor where this project started, and still the honest baseline."
Read that ladder carefully, because it does not contradict the thesis — it confirms it. Every rung is bought with memory. The six-GPU number is what happens when the experts stop being streamed at all, at which point Colibri is no longer doing the interesting thing; it is a competent C inference engine on hardware that did not need the trick. The interesting regime is still the bottom of that ladder.
**Two drives instead of one.** The most direct attack on the bottleneck: put a second copy of the model on a second SSD and read from both.
**And a research posture that is rarer than any of the engineering.** The README now carries a table of open hypotheses with, for each, the evidence so far and *the experiment still needed* — and the framing is explicit: "Colibrì treats an optimization as a hypothesis until a controlled end-to-end A/B shows otherwise." Contributors are asked to record hardware, commit, exact command, cache state, expert hit rate, bytes read and quality check, change one variable, and **publish the negative results too**. The line I would put on a poster: *"A well-controlled failure is more valuable here than an unexplained fast number."*
That posture produces the single most useful correction to this article. The MTP section above says speculation only pays once the cache is warm. The repo now has a number for the other end: **MTP has measured a 32% loss at around 85% expert hit rate**, listed as an open problem with "map the break-even surface" as the required experiment. So it is not monotone — speculation loses when the cache is cold *and* can lose again near residency, and the profitable band in between has not been mapped. That is a sharper and less comfortable claim than the one I made in July, and it comes from the project reporting against itself.
Two smaller things worth knowing. `O_DIRECT` is documented as drive-dependent rather than a universal win. And the engine now spans CPU, CUDA, Metal and NUMA placement in one runtime, with the honest caveat attached — a fast CPU and low residency can erase the GPU wins entirely.
## The take
Colibri is a lovely demonstration of a real principle: **a sparse MoE's active footprint, not its parameter
count, is what a runtime has to hold in fast memory.** Because GLM-5.2 fires only a few of its 256 experts
per layer per token, and because int4/int8 quantization shrinks both the resident core and each streamed
expert, the working set collapses from hundreds of gigabytes to ~10 GB of RAM plus on-demand disk reads.
That is genuinely clever, and it is pure C with zero dependencies, which makes it a beautiful object to read.
But be clear-eyed about what it buys. It buys *access*, not *speed*: you can hold a conversation with a
744B frontier model on a 25 GB machine, at the pace of a few seconds per token, provided you also own a
370 GB fast disk. The bottleneck is not going away — it is the physics of pulling ~11 GB across NVMe for
every cold token — and the caching, pinning and speculation only push against it. As an existence proof
that extreme sparsity plus aggressive quantization can put a frontier model on consumer hardware, Colibri
is compelling. As a way to actually *use* one at interactive speed, it is not there, and it is refreshingly
honest about that.
---
*Built from the [Colibri repository](https://github.com/JustVugg/colibri) (JustVugg; Apache-2.0) and its
README. All throughput and hardware figures are **community-reported single test cases**, not official
benchmarks: cold ~0.05–0.1 tok/s and MTP 2.2–2.8 tok/forward on the dev machine; 0.37 tok/s (Ryzen AI 9,
Framework 13), 1.06 tok/s (M5 Max, 128 GB), 0.11 tok/s (Core Ultra 7, 24 GB) from community runs.
Architecture figures — 744B total, ~17B dense, 9.9 GB int4 core, 21,504 experts, ~370 GB on disk — are from
the repository README. The interactive diagrams are illustrations of the mechanism, not measurements.*