# Qwen3.8-27B at 764, 1,035 and 1,623 tok/s: one GPU, many streams, and what each number divides by
> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.satyajit.si/articles/qwen3-8-27b-throughput-week
> date: 2026-11-12
> tags: inference, qwen, speculative-decoding, kv-cache, linear-attention, vllm, benchmarks
On 11 November my feed carried three throughput records for the same model on one consumer
card each. One was a fortnight old: HyperQwen's figure dates from late October, and a thread
that day recirculated it.
- [@QwenAIFans](https://x.com/QwenAIFans/status/2120316633860452657) and
[@plotarmordev](https://x.com/plotarmordev/status/2120375275863359716) reposted a video by
@ashxhart: Qwen3.8-27B on one RTX 5090 through TensorFold, NVFP4 weights, "700 tok/s". The
original post now returns not-found through the mirror, so the reposted video is the source.
- [@lemonDefi1](https://x.com/lemonDefi1/status/2120335852849029383), quoting
[a longer thread](https://x.com/Oluwaphilemon1/status/2120317269804826749): one RTX 3090 at
a 250 W cap, *"1,035 tokens/s across 64 concurrent Qwen3.8-27B sessions"*, from HyperQwen.
- [@satellitedown](https://x.com/satellitedown/status/2120574539381756375), the author of
[Cinference](/articles/cinference): *"does anyone want 1600tps on a single 5090 running Qwen
3.8 27b?!"*
Unsloth shipped new GGUFs for the same model the same afternoon, with
[*"Run on 12GB RAM"*](https://x.com/UnslothAI/status/2120536946774769860) in the post. That one
is a different kind of claim, and I come back to it at the end.
A reply under the TensorFold repost says what everyone should ask first: *"原来是 8 路，那我就不慌了"*
("oh, it's eight streams, then I'm not worried"). Every one of these numbers is a sum over
several streams. That is not cheating. It is a different quantity from the speed one person
sees in a chat window, and the gap between the two is set by arithmetic you can do from the
model's `config.json`.
| Claim | Card | Streams | Total decode | Per stream | Source of the number |
|---|---|---:|---:|---:|---|
| "700 tok/s" (TensorFold) | RTX 5090 | 8 lanes of one prompt | 764 | about 95 | the video's last line |
| "1600tps" (Cinference) | RTX 5090 | 8 lanes of one prompt | 1,099 average, 1,623 best second | about 137 | the video's stats line and counter |
| "1,035 tok/s" (HyperQwen) | RTX 3090, 250 W | 64 independent requests | 1,035 to ~1,094 | about 16 | the repository's READMEs and demo |
The per-stream column is the total divided by the stream count, my division. The rest of this
piece is why the columns come out the way they do, then each claim in turn.
## One weight read, shared by every stream
Generating a token is a pass through the whole network for one new position. At batch size
one that pass does very little arithmetic per byte it loads: every weight matrix is read
from memory, multiplied by a single activation vector, and thrown away until the next token.
The card's compute units wait on its memory bus. That is the regime
[the inference primer on this site](/articles/how-llm-inference-works) calls memory-bound,
and it sets a ceiling you can compute with a pocket calculator:
$$
\text{tok/s}_{\,B=1} \;\le\; \frac{\text{bandwidth}}{\text{bytes read per token}}
$$
HyperQwen's 3090 checkpoint is W4A16 at about 16 GB, and the card moves 936 GB/s, so one
stream cannot beat 936 / 16 = 58 tok/s. HyperQwen's own batch README says the same in
words (*"Raw single-stream decode for 16 GB of weights on a 936 GB/s card is bounded near ~50
tok/s"*) and measures 46 at one concurrent request, 79% of the line. On the 5090 the NVFP4
weight arena NInfer documents is 19.729 GiB, or 21.18 GB, against 1,792 GB/s: a ceiling of 85,
and NInfer's no-speculation run measured 71.2 at 8K context, which is 84%.
Now put $B$ streams in the same step. The weights are still read once. Each matrix multiply
now takes a $B$-row activation block instead of one row, and the extra rows cost arithmetic,
which the card had spare, but almost no extra bytes. So the step time barely moves while the
tokens it produces multiply by $B$:
$$
t_\text{step}(B) \;=\; \max\!\left(\frac{W + B\,(2S + \kappa L)}{\eta \cdot \text{BW}},\;
\frac{B \cdot r \cdot F}{P}\right),
\qquad
\text{aggregate} = \frac{B\,\tau}{t_\text{step}},\quad
\text{per stream} = \frac{\tau}{t_\text{step}}
$$
$W$ is the weight bytes, $\text{BW}$ the bandwidth and $\eta$ the fraction of it an engine
achieves. The per-stream terms are a recurrent state of $S$ bytes that is read and written
each step, and $\kappa$ bytes of KV cache per token of context $L$. $F$ is the arithmetic per
position, $P$ the compute ceiling, $r$ the positions each stream verifies per step and
$\tau$ the tokens it keeps. Without speculation $r = \tau = 1$.
At small $B$ the first term wins and the aggregate rises almost linearly with $B$ while each
stream's speed hardly drops. That is the whole trick behind every number in the table above.
HyperQwen's cohort table on the 3090 is the cleanest demonstration I found: 45.6, 83.7, 162.7
and 321.0 tok/s at one, two, four and eight concurrent chat requests. Eight users get 7.0 times
the throughput of one, and each of them still sees about 40 tok/s. The scheduling side of this,
how vLLM keeps the batch full as requests come and go, is in [the DGX Spark batching
piece](/articles/dgx-spark-batching); here the interest is the arithmetic.
## What one more stream costs on this model
The per-stream term is where Qwen3.8-27B differs from the usual transformer, and it is
worth getting from the config rather than from a rule of thumb.
[`config.json`](https://huggingface.co/Qwen/Qwen3.8-27B/blob/main/config.json) has 64 layers,
of which `layer_types` marks 16 as `full_attention` and 48 as `linear_attention`, with
`full_attention_interval` 4.
The 16 attention layers keep a KV cache. With `num_key_value_heads` 4 and `head_dim` 256, one
token stores
$$
16 \times 2 \times 4 \times 256 = 32{,}768 \text{ values}
$$
which is 64 KiB in BF16, 32 KiB in FP8 and 25.125 KiB in Cinference's K8V4 format (the
[Cinference article](/articles/cinference) derives that last one from its kernels). For a
dense 64-layer model with the same heads it would be four times as much.
The 48 Gated DeltaNet layers keep no KV cache. Each keeps a fixed matrix per value head instead:
`linear_num_value_heads` 48, each `linear_key_head_dim` 128 by `linear_value_head_dim` 128.
That is
$$
48 \times 48 \times 128 \times 128 = 37{,}748{,}736 \text{ values per stream}
$$
or 144 MiB at the FP32 `mamba_ssm_dtype` the config asks for, 72 MiB in FP16. It does not grow
with context, which is why this model holds up at 256K. It is also allocated before the
stream's first token, and it is read and written on every decode step, so at short context it
is the dominant per-stream cost in both memory and traffic.
HyperQwen found this the hard way, and its `docs/optimizations.md` reports it plainly: *"On
this architecture that state — not the KV cache — is what bounds concurrency: with
`--max-num-seqs 64` only 37 requests were ever actually running."* Switching the state to FP16
let all 64 run, and the batch README says about 70 short requests fit. Those two observations
agree with each other: 37 × 144 MiB is 5.2 GiB and 70 × 72 MiB is 4.9 GiB, so the pool
left for streams on that card is about five to six GiB either way.
That also settles the thread's "configured context up to 150K tokens". It is true, and it is
a different configuration from the 64 sessions. HyperQwen's batch README records the KV pool on
vLLM 0.27.1 as 223,821 tokens with prefix caching off and 193,298 with it on, the default, and
puts that at *"1.29x concurrency at 150k instead of 1.49x"*: barely more than one full-length
request. Sixty-four
sessions at 150K would need 64 × 150,000 × 32 KiB, about 293 GiB of FP8 cache. The 64-stream
number was measured at 128 tokens in and 512 out, 640 tokens of context per stream, which uses
about a fifth of that pool.
On the 5090 the same wall shows up in NInfer's own corpus run, 75 requests of long generations.
With NVFP4 weights, eight concurrent slots left room for 187,712 tokens of device KV, and the
average batch over the corpus fell to 2.36; NInfer's docs put it down to memory pressure. Eight
slots finished the corpus slower than four, 334.2 against 432.9 tok/s. A smaller groupwise-int
checkpoint (16.672 GiB) left more room and averaged 4.76 in flight at eight slots. Fewer weight
bytes help twice: less to read per step, and more memory left for streams.
## Where the line bends
Batching stops being free when the arithmetic catches up with the memory traffic. Counting the
linear layers from the config (attention projections with their output gate, the DeltaNet
projections, the 17,408-wide MLPs and the untied 248,320-row output head) gives 25.6 billion
weights touched per token, or about 51.2 GFLOP. That count closes against the repository: add
the 1.27 billion input embeddings, roughly 0.46 billion in the vision tower (its 0.93 GB BF16
`mmproj` file in Unsloth's repo) and roughly 0.42 billion in the MTP module (850 MB in BF16, per
HyperQwen), and you land within 0.01 billion of the 27,781,427,952 parameters Hugging Face
lists. HyperQwen measured the 3090's ceiling for this
workload directly: *"At 40-64 concurrent sequences the decode step is bound by fp16 tensor-core
math (~63 TFLOPS sustained at 250 W)."*
So the 3090 step costs 17 ms of weight reads plus about 0.16 ms of state traffic per stream, and
0.81 ms of tensor-core time per stream. The two meet around 26 streams at full bandwidth, or in
the mid-30s at the 79% an engine actually gets. Past that point every extra stream adds its
full share of arithmetic, the aggregate flattens toward $P / F$, and per-stream speed falls
roughly as $1/B$. The model puts the 64-stream ceiling at about 1,230 tok/s; HyperQwen's batch
table measures ~1,094 steady with int8 activations on the MLPs. Those int8 tensor cores are the
reason the repository patched vLLM's W4A8 Marlin path at all: past the bend, the only way up is
cheaper arithmetic.
The calculator runs that model with four presets. On the 3090 it lands 1% to 9% over
HyperQwen's measured C1 to C4 cohorts, 6% over at C8 (340 against 321), and 12% over the
64-stream figure (1,230 against ~1,094). Drag the state slider to
144 and the memory wall moves from 69 streams to 38, which is the 37-of-64 log line.
It does much worse at long context, and the 8K preset shows where. HyperQwen's "raw" cells,
speculation and prefix caching off, use 7,681-token prompts: 46.9 tok/s at one stream, 94.8 at
four, 111.5 at eight (143.7 with int8 prefill). That is 2.4× for eight streams, not 7×. The
model, with each stream's 8K of KV added to the traffic, still predicts about 300 at eight. I
do not know where the rest goes. One candidate is that those cells start all eight requests at
once, so their decode steps share the GPU with the other requests' chunked prefill (2,048 tokens
a step), and a mean TPOT includes those steps; I cannot separate the two from the published
table. (The 12 November update below looks at this on a real 3090, and the engine log there points
at prefill.)
The 7× from the cohort table is a short-context result, and it should be quoted as one.
On the 5090 MTP3 preset it overshoots NInfer's measured points, by 15% at one stream and 53% at
eight. The model has no term for drafting. HyperQwen's notes give a reason that would fit: every
draft token also runs the 248k-row output head, and on the 3090 *"each extra draft cost ~3 ms"*
before they shrank it. I have not verified that NInfer's drafter costs the same, so treat the
gap as unexplained rather than explained.
## Speculation spends the same arithmetic
Every 5090 number this week used speculative decoding as well as batching, and the two pull on
the same resource. A speculating stream drafts $k$ tokens cheaply and then verifies $k+1$
positions in one target pass. At batch one that pass is still memory-bound, so the extra rows
are close to free: the step reads the same weights and commits $\tau = 1 + a k$ tokens instead of
one, for acceptance $a$. That is how Cinference got 450.78 tok/s out of one stream in September, and how
[qwen-warp](/articles/qwen-warp-llamampere) reached 276.7 tok/s on a single 3090 by committing
about six tokens per weight pass.
In a batch, those extra rows are extra arithmetic. Eight streams verifying sixteen positions each
put 128 rows through every matrix, the same arithmetic as 128 plain streams. HyperQwen's README
draws the conclusion from measurement: *"Speculation wins below ~8 concurrent users, plain batching
above."* Its batch mode ignores the speculation setting. NInfer's MTP3 saturation run shows the
squeeze from the other side: 143.8 tok/s at one stream and 766.6 at eight, a 5.33× gain for 8×
the streams.
So the eight-lane runs sit in the one place where both multipliers still pay. TensorFold has not
published the size of its NVFP4 checkpoint or its state dtype, so for its run I borrow NInfer's
19.729 GiB and an FP32 state: the plain ceiling for eight streams is then about 640 tok/s at full
bandwidth, or 538 at 84% of it, and 764 needs at least 1.19 tokens per stream per step, or 1.42.
The Cinference run's model card gives 20.9 GiB resident, which puts its plain ceiling at about
577, or 485. Its 1,099 needs 1.90 and 2.27 tokens per lane per step; its 1,623 best second needs
2.81 and 3.35. None of those is hard for a good drafter
on code. With a 16-position verify, the second term starts to matter too: 128 rows × 51.2 GFLOP
is 6.55 TFLOP per step, 31 ms even at the 209.5 TFLOP/s dense BF16 reference that Cinference's
own benchmarks use for this card. At that step time, 1,099 tok/s across eight lanes needs about
4.3 tokens per lane per step, which is roughly what fafstmobel's model card measures for its
drafter on coding requests (4.016 accepted per round, plus the target's own token). That last
step of reasoning assumes the full 16-position block was verified every step, and adaptive
drafting would not do that, so read it as a consistency check, not a measurement.
## TensorFold's 764: eight lanes of one prompt
The video shows a terminal running `omp` against a TensorFold server on an RTX 5090 with the
model `TensorFold-Fanout/qwen38-fanout`. The prompt is *"build me a html three.js website for
harry potter, it should be a masterpiece."* The status bar reads `TensorFold RTX 5090
Qwen3.8-27B NVFP4 (uLane)`. Code streams in sections labelled by bucket, and the run ends on one
line of statistics.
So "700 tok/s" was the honest rounding of two printed numbers, and @QwenAIFans's repost said
what they were: *"8 路同时在写代码，13.32 秒里一共 8859 个 token，解码加起来 764 tok/s"*,
eight lanes writing code at once, 8,859 tokens in 13.32 s, decode summed to 764.
The two rates check against each other only halfway. 8,859 tokens over 13.32 s is 665.1 tok/s,
which is the printed end-to-end figure to rounding. The decode figure implies 8,859 / 764 =
11.60 s of decoding, so 1.72 s of the wall clock was excluded. The line names two other
intervals, and neither accounts for it: taking out the 1,051 ms plan gives 722 tok/s, taking out
the 2,333 ms tail gives 806, both gives 892. Most likely the plan plus the lanes' prompt
processing, about 0.67 s, is outside the decode window, but the formula is in `omp`'s code, which
is not in any repository I cloned for this piece. TensorFold's own repository has no record of this run; its
changelog describes concurrent streams sharing every round, each drafting with DFlash2, for this
model, which is the mechanism, but records no 5090 lane benchmark.
"Lane" needs care here, because the site's [TensorFold and Strata
article](/articles/tensorfold-strata-engines) defines TensorFold's lanes as the rows of one
verify pass, and the engine's own docstring says those rows *"can hold the near future of one
stream (drafts), alternative futures (a draft tree), or rows of other streams."* The video's
"code lanes" are the last kind: one prompt split into sections, each section written by its own
stream, shown with its own heading, merged into one file. Each of those streams presumably
carries its own draft rows as well, which is how the total gets past the plain ceiling below.
That is fan-out, a legitimate agent pattern, and for throughput purposes it is eight concurrent
streams sharing every weight read. Per lane it is 764 / 8 = 95.5 tok/s on
average, and the "tail" is the 2.3 s when fewer than eight lanes were still writing.
One coincidence is worth putting beside it. NInfer's MTP3 saturation run on the same card,
published weeks earlier and retained in Cinference's docs, measured 766.6 tok/s at eight
concurrent streams. Different engine, different drafter, same number to within half a percent.
That is what you would expect if both are sitting near the same roofline.
## The 1,600 tok/s was a fastest second, and its author said so
Within the hour, answering people under his own post, @satellitedown
[said what it was](https://x.com/satellitedown/status/2120588862422737004): *"NOPE! … this is
a magic trick i'm running 8 small context lanes at once through one output since it's what the
other guy did."* In an earlier reply he was blunter: *"i'm copying/making fun of what the other
guy did where you can run 8 lanes at once and sum their tps to get silly high numbers that dont
reflect any real use case. realistically you'll get like 350tps with bursts of up to 900 when
coding with a nice 257k context window."* That last sentence is a single-stream claim nobody
has measured publicly; his September results file measured 450.78 on recall at 8K.
The video is careful about this, more careful than the post. The giant number is labelled
*"FASTEST SECOND · 8 LANES COMBINED"*. The run's own statistics line, in the same format as
TensorFold's, reads 14,539 tokens in 14.70 s, 1,099 tok/s decode, 989.3 end to end, plan 1,421
ms, tail 5,745 ms. The end-to-end figure closes: 14,539 / 14.70 is 989.0. Taking only the plan out of the wall
clock gives 1,095 here, close to the printed 1,099 but 0.05 s off, and the same rule gives 722
rather than 764 for TensorFold. Plan-only reproduces neither line exactly.
His comparison card gets the like-for-like ratio right: 1,099 / 764 is 1.44 and 989.3 / 665.3
is 1.49. The reply's *"twice as fast as his"* is a different ratio, 1,623 / 764 = 2.12, a best
second set against someone else's average. The tail is also much longer here. For 5.7 of the 14.7
seconds, 39% of the run, fewer than eight lanes were decoding, so the eight-lane rate in the
middle of the run was well above 1,099 and the fastest second shows how far.
Three details matter for anyone trying to reproduce it.
- **The model is not the one in the Cinference article.** The status bar names `fafstmobel 27B`,
[a separate artifact](https://huggingface.co/satellitedown/fafstmobel): an abliteration delta
transplanted onto another fine-tune, MLP weights in NVFP4 and attention and DeltaNet
projections in FP8, 22,884,131,844 bytes with a bundled DFlash2 drafter. The Cinference
article covered the Huihui NVFP4 build with MTP-10.
- **Both published installers run one request at a time.** The
[`fast-long-context-cinference`](https://github.com/satellitedown/fast-long-context-cinference)
manifest, re-pinned the day of the post, and
[`fafstmobel-cinference`](https://github.com/satellitedown/fafstmobel-cinference) both set
`"max_concurrency": 1`. The engine's `docs/serving.md` gives `--max-concurrency` a valid range
of `1..8`, so eight lanes is the engine's maximum, set by hand for this video.
- **The `ninfer` link in the replies is a sibling, not the source.** A reply pointed at
[`dylanbrodiefafard/ninfer`](https://github.com/dylanbrodiefafard/ninfer). Its README calls it
*"This fork of Neroued/NInfer"*; Cinference is another fork of the same upstream. It has its
own concurrency numbers, and they are the most carefully defined of the week: 312 tok/s for
one request on a Python-package prompt and 1,057 total for six, where *"the six-request total
counts only intervals with all six decoding. Divide by six for the average per active
request."* That rule removes the tail on purpose and says so.
## HyperQwen's 1,035: sixty-four short requests on a 3090
[`syv-ai/HyperQwen`](https://github.com/syv-ai/HyperQwen) is the one claim of the three built for
serving many users, and it is the one whose repository explains its number best. The site's
[DFlash2 article](/articles/dflash2) covered its single-user mode in October. It is a patch series
on a pinned vLLM plus a model-preparation pipeline; the link in the replies,
`syv-ai/qwen38-27b-rtx3090`, is the same repository under its old name (same commit, `3d6c954`).
The batch mode is the textbook case from the first section. Its headline is decode throughput at
64 concurrent random requests of 128 tokens in and 512 out. The repository states it three ways:
*"~1,035 tok/s aggregate decode"* on the front page, *"~1,094"* steady-state decode in the batch
README's table (which notes the figures *"now predate four pins"*), and 1,046 in the demo video,
computed as 64 × 1000 / a median time per output token of 61.2 ms. The end-to-end figures,
counting everybody's prefill and the ramp, are 942 to 959.
The per-stream number falls straight out of the definition: 1000 / 61.2 ms is 16.3 tok/s for each
of the 64 users. The same card does 46 for one user in this mode, and 127 for one user in the
repository's single-user mode, where DFlash2 speculation is on. A person typing to this server
under full load reads at about a third of the speed of the plain single stream, and an eighth of
the single-user mode's. As a backend for an eval run or a document pipeline that is an excellent
trade. As a chat server it is the wrong mode, and the README's own decision table says to pick
`--profile single` for *"one or a few people chatting"*.
Two smaller things the thread did not mention. The tok/s chart in the demo has four plateaus with
dips between them: 256 requests in four waves of 64, with the prefill and ramp of each wave in
the dips. That is the same effect as the lanes' tail, at a different scale. And the steady figure
is the engine's own log line once all 64 are resident, which the README defines carefully; the
end-to-end number is the one a client measures.
## Update, 12 November: HyperQwen's 64 streams reproduce on a 3090, at 390 W
The evening this piece went up I ran HyperQwen's batch mode, and then qwen-warp, on an RTX 3090.
Everything above this section was read. In it, the 3090 numbers marked as mine were run on one
card on 12 November; the NInfer part at the end is from its release notes.
The stack was the repository at `3d6c954`, the vLLM 0.33.0 its `docker/requirements.txt` pins
with the full patch series applied by `patches/apply.sh`, torch 2.14.0+cu130, FlashInfer
0.7.2.post1 and Python 3.12.11, on Ubuntu 20.04 with driver 580.126.09 and no Docker. The
checkpoint is `dbirks/Qwen3.8-27B-W4A16-AutoRound` at `1f05c44`, put through the three prepare
steps in `docs/install.md` (output head, embeddings and MTP module requantized). The server is
`batch/start_qwen.sh` with its defaults: FP8 KV, FP16 DeltaNet state, int8 activations on the MLP
GEMMs, 64 slots, prefix caching on, except that `GPU_UTIL` is 0.92, not 0.94: at 0.94 the engine ran out of memory inside a Marlin GEMM on the very first eight-request prefill,
with 330 to 380 MiB of the card held by the machine's desktop session. Separately, FlashInfer's
FP8-KV kernel would not link until I gave the linker a `libcudart.so` symlink, because pip's CUDA 13
runtime ships only `libcudart.so.13`.
The power limit is the part that does not match. HyperQwen measured at 250 W. This card runs at
its default limit of 390 W (the board allows 480), I was not allowed to change it, and
`nvidia-smi`, sampled once a second, read a mean draw of 383 to 388 W in every run, one stream
included, with the SM clock averaging 1,760 to 1,900 MHz and the die peaking at 75 to 80 °C. The card was
at its power limit the whole time, and its limit is 140 W higher than the claim's. So a number
here at or above 1,035 shows that the recipe reproduces. It does not show what the same card does
at 250 W.
### The headline workload
Same protocol as the README: `vllm bench serve`, random dataset, 128 tokens in and 512 out,
`--ignore-eos`, 256 requests, 64 at a time, a fresh seed per call so the prefix cache never
hits. I ran `bench/run_benchmarks.sh batch` twice, as the README says to, then the same workload
three more times.
| Run | End to end | 64 × 1000 / median TPOT | Median TPOT | Engine log, 64 running, no prefill | Mean TTFT |
|---|---:|---:|---:|---:|---:|
| HyperQwen front page, 250 W | 948 | ~1,035 | | | |
| HyperQwen batch README, 250 W, vLLM 0.27.1 | 942 | | 62.2 ms | ~1,094 | |
| HyperQwen demo video, 250 W | 959 | 1,046 | 61.2 ms | | |
| Mine, `run_benchmarks.sh` first pass | 1,087.0 | 1,179 | 54.27 ms | 1,248 | 2.41 s |
| Mine, second pass (the one the README keeps) | 1,073.3 | 1,165 | 54.94 ms | 1,232 | 2.44 s |
| Mine, a separate run after the ladder below | 990.5 | 1,076 | 59.48 ms | 1,139 | 2.64 s |
| Mine, `PREFIX_CACHE=0`, two runs | 1,056.6 and 1,052.7 | 1,150 and 1,148 | 55.66 and 55.76 ms | 1,222 and 1,216 | 2.59 and 2.61 s |
The engine-log column is the median of vLLM's ten-second "Avg generation throughput" lines that
show all 64 requests running and no prompt tokens in the window, which is how the batch README
defines its steady figure. All 256 requests finished in every run, and the server logs record
no preemptions.
Every one of my runs clears 1,035 by the repository's own arithmetic, and the end-to-end column
is 3% to 15% above the 942 to 959 HyperQwen printed. The same card, the same commands and a
fresh seed moved the result by 10% between runs, from 990.5 to 1,087.0 end to end, so the
README's choice of a single "~" figure is about as precise as the measurement allows. Turning
prefix caching off, which was the default on vLLM 0.27.1 when the README's table was measured,
landed inside that spread.
Per stream, the second pass gives each of the 64 users 1000 / 54.94 = 18.2 tok/s, and the slowest
run 1000 / 59.48 = 16.8. The 16.3 I worked out above from the demo's 61.2 ms holds to within a
couple of tokens a second. The engine's steady figure ran from 1,139 to 1,248, and four of the
five runs land within 1.5% of the batching model's ~1,230 for 64 streams from "Where the line
bends". That model used HyperQwen's 63 TFLOPS at 250 W, though, and this card had 140 W more to
spend.
The launcher's seat-budget line at boot did not predict this workload. With prefix caching on it
said only 24 to 29 requests of 1,024 tokens would fit; all 64 of these 640-token requests ran at
once anyway, filling the KV pool to 95.2%. With caching off it said 54 to 91.
### Where the bend actually is
The last Falsifier below asked for cohorts at 16 and 32 in batch mode. This is that ladder, same
workload, with 16 requests at one stream, 64 at eight, 128 at sixteen and at thirty-two, and 256
at sixty-four:
| Streams | Median TPOT | Per stream, 1000 / TPOT | Aggregate, streams × per stream | End to end | Median TTFT |
|---:|---:|---:|---:|---:|---:|
| 1 | 23.04 ms | 43.4 | 43.4 | 43.0 | 133 ms |
| 8 | 25.22 ms | 39.7 | 317 | 304.2 | 585 ms |
| 16 | 27.71 ms | 36.1 | 577 | 536.5 | 1,089 ms |
| 32 | 36.30 ms | 27.5 | 882 | 803.6 | 1,700 ms |
| 64 | 59.48 ms | 16.8 | 1,076 | 990.5 | 2,698 ms |
Mean power was 387 to 388 W at every row, and `nvidia-smi` read about 23,800 MiB in use throughout,
because vLLM reserves its pool when it starts. The step grows by 9% from one stream to eight and
another 10% to sixteen, which is the memory-bound region: each new stream is nearly free. From 16
to 32 it grows 31%. From 32 to 64 it grows 64% in this run, the slowest of my five at 64 streams;
against the other four (54.27 to 55.76 ms) it grows 50% to 54%. A fully compute-bound step would
grow 100%. So the bend starts between 16 and 32 streams on this card, a little earlier than the
mid-30s I estimated, and by 64 it is roughly half to two-thirds of the way to $1/B$: doubling the
streams from 32 bought 22% to 34% more aggregate (882 to between 1,076 and 1,179) and cost each
user 33% to 39% of their speed (27.5 to between 16.8 and 18.4).
One stream decodes at 43.4 tok/s, 6% under the README's 46 at 21.7 ms. The 23.04 ms step is 74%
of the 58 tok/s bandwidth line from the first section. The repository's real-prompt cohorts came
out at 43.6, 84.6, 165.4 and 315.3 tok/s for one to eight chats on the kept pass (47.8, 91.0,
175.1 and 333.6 on the first), against the README's 45.6, 83.7, 162.7 and 321.0. The box is
shared, and another user's desktop session ran throughout with a load average of 3 to 6 on its 12
cores; I cannot tell how much of the single-stream gap is that and how much is the newer pin.
### What long prompts do to the aggregate
The open question in "Where the line bends" was why HyperQwen's raw cells with 7,681-token prompts
only reach 111.5 tok/s at eight streams when the model predicts about 300. I ran the same shape in
batch mode, 7,681 tokens in and 511 out, with four requests at one stream and two per stream
above it:
| Streams | Requests | Streams × 1000 / mean TPOT | End to end | Median TTFT | One engine log line with no prefill |
|---:|---:|---:|---:|---:|---:|
| 1 | 4 | 41.2 | 30.4 | 4.39 s | 41.3 |
| 8 | 16 | 116 | 82.4 | 8.03 s | 279.2 |
| 16 | 32 | 128 | 93.3 | 8.08 s | 443.0 (15 running) |
By the README's definition, eight streams give 116 tok/s here and 111.5 there, but the setups
differ more than that closeness suggests. The README's cells start C requests at once with prefix
caching off, in single-user mode on its fast checkpoint. Mine ran in batch mode with int8 MLP
activations, caching on (the fresh seeds kept it from ever hitting), and twice as many requests
as streams, so a second wave's prefills land on the first wave's decode. Read the two as the same
shape, not the same measurement.
The last column is the more useful one, though it is thin: vLLM logs one line per ten seconds,
and each figure there is a single line with no prompt tokens in it (at one stream the better of
two, 41.3 against 35.0; at sixteen, 15 were still running). In that line, eight 8K streams decoded at 279.2 tok/s, within 7% of the model's ~300. In
20 of the 24 lines that did carry prompt tokens, generation was 17 tok/s or less for all the
streams together; the other four show prefill and a few hundred tokens of decode in the same ten
seconds. A step that carries a 2,048-token prefill chunk takes on the order of a second at this
card's prefill rate and gives each running stream one token, and the mean TPOT averages the two
regimes. So the engine log points at prefill sharing the steps as the gap I could not explain, with
decode itself close to the line at 8K. A run that admits all eight at once and then measures only
the decode would settle it.
### qwen-warp's 425 at four clients: the scaling reproduces, the level is prompt-bound
[qwen-warp](/articles/qwen-warp-llamampere)'s README now gives 425 tok/s for *"4 users at once, in
total"* on a 3090 at 350 W. I ran it on the same card after HyperQwen: `tuan3w/qwen-warp` at `031d02a`
(v0.10.0, a release newer than the one the site's qwen-warp article read), the
`ukisai/Swift-1.5-Qwen3.8-27b-INT4` checkpoint at `81a28ee`, and `--profile multiuser`, which gave
five slots of 15,360 tokens each, the same as its table. The engine wants a system CUDA 13.0
toolkit and the box has 12.8; the repository's own lock resolves nvcc 13.4.92 against CCCL 13.3
headers, which fail CUDA's version check, so I compiled it with pip's matched 13.0.88 set
instead. Power was the same 390 W limit, at 377 to 387 W mean.
Its `docs/PERFORMANCE.md` describes the protocol: C closed-loop clients, three greedy chat
requests each, thinking off, `max_tokens` 512, eight short code, prose and math prompts, aggregate
as all output tokens over wall time, per request as tokens over the time between first and last
chunk. The prompts live in a script in the author's research repository, which is not public, so
I wrote two sets of eight: short questions like the ones described, and prompts that ask for long
answers so that every request runs to 512.
| Clients | qwen-warp docs: aggregate, per request | Mine, short prompts | Mine, long answers |
|---:|---|---|---|
| 1 | 154.4, 162.8 | 140.2, 165.4 | 138.3, 146.8 |
| 2 | 278.1, 149.7 | 209.8, 148.3 | 245.3, 133.8 |
| 4 | 424.9, 114.8 | 351.8 and 353.0, 118.3 and 115.8 | 378.1 and 374.2, 105.2 and 104.0 |
| 8 | 440.5, 97.8 | 386.6, 92.0 | 407.7, 90.0 |
The per-request speeds on the short set land within 3% of the docs at one, two and four clients,
which says the engine does what its table says. The aggregate is where the prompts show. The short
set's answers averaged 242 tokens at four clients, so a closed loop spends a larger share of its
wall time between requests and on the last one finishing alone: 352 tok/s from four streams that
each decode at 118. With long answers that idle share shrinks and the aggregate rises to 374 to
378, about 3.6 streams' worth of the 105 each one gets, where the docs' 424.9 is 3.7 streams of
114.8. The scaling matches: four clients give 2.73 times one client on my long set against 2.75 in
the docs' multiuser row, and eight give 2.95 against 2.85. What I could not match is the absolute level, and the
gap sits in per-request speed, which for an engine that drafts depends on how predictable the text
is. So I read "425" as true for its own prompts and 11% to 17% above what mine got, and the 2.7×
from one user to four as the number that transfers.
### NInfer's 2,439.6 on a 5090 is 32 requests at about 76 each
The day the three posts went round, Dylan997S tagged
[`v1.2.5-dspine.3`](https://github.com/Dylan997S/ninfer-dspine-rtx5090/releases/tag/v1.2.5-dspine.3)
of NInfer for the RTX 5090: Qwen3.8-27B in NVFP4, Tencent's DSpine proposal head at K11 for
speculation, FP8 KV, prefix reuse off. NVFP4 needs Blackwell, so it cannot run on the 3090, and
everything in this part is from the release notes. Those notes now open by pointing to a newer
release, `v1.2.5-dspine.5` from 12 November, which I have not read.
The headline is 2,439.6 tok/s on Windows 11 and 2,345.6 under Ubuntu 24.04 in WSL2, for 32
concurrent requests alternating two Python prompts, 2,048 output tokens each, one round. The notes
define it as end-to-end aggregate including prefill, the stricter of the two definitions this
article keeps separating. Divided by 32 it is 76.2 tok/s per request on Windows, my division. With
32 distinct Python prompts per round and five rounds pooled, the figures are 2,368.2 and 2,349.1.
One request alone on the same 32-lane build decodes English at 155.5 tok/s and Python at 511.0.
That gap is speculation, and the reading is mine rather than the release's: code is predictable
enough that most of the eleven drafted tokens (`--draft-tokens 11` in the release's benchmark
instructions) survive verification, and prose is not. At 32 lanes
each request gets about 76 end to end, near a seventh of its solo Python decode speed, which is the squeeze from "Speculation
spends the same arithmetic": the verify rows that were free at batch one are arithmetic at
batch 32.
Read the context before comparing. The 32-request benchmark profile gives each request 2,304
tokens of context with a 128-token prefill chunk, and the notes say the 32-lane 8K profile with a
1,024-token prefill chunk exceeded the calibrated device budget on the tested card. All 32 of 32
single-round requests and 160 of 160 five-round requests finished. The Linux build is verified
under WSL2 only and carries a `v1.2.5-dspine.4` build identifier, and the notes say the benchmark
checks output length and throughput, which says nothing about whether the Python it wrote runs.
## All of it on one chart
On log scales, "aggregate = streams × per-stream speed" turns every per-stream speed into a
straight diagonal, so each claim's user-visible speed can be read off its position.
The single-stream speculative points sit high on the left at 127 to 451 tok/s, each one a user who
has the card to themselves. The eight-lane 5090 runs sit between the 50 and 200 per-stream
diagonals. The 64-stream 3090 point sits near 16. Which of those is "fast" depends on whether you
are one person waiting or a pipeline counting tokens, and none of the three posts said which one
it was written for.
## Unsloth's "12 GB" is a batch of one
The Unsloth release is a quantization claim, not a throughput one, and the site has already
[read the Qwen3.8 files](/articles/qwen3-8-open-weights) and
[looked at what fits in 12 GB](/articles/qwen3-8-27b-on-12gb), so this is short. Two things in
the announcement read differently against the files.
The "+5–10%" is a comparison between quantizers at equal size: Unsloth's new files against its
older V3, ISTA's GSQ-RCO and an EXL3 build. It is not distance from the full model. On the
Terminal Bench 2.1 panel two of the green points sit above the BF16 line of 59.15, which says the
per-point noise on that benchmark is at least as large as the gaps it is drawn to show. Unsloth's
own docs say benchmarks should be run more than once for that reason.
"Run on 12GB RAM" is true of the smallest files. Unsloth's own docs page for the model says
*"runs locally on 17GB RAM/VRAM setups"*. The repository's files below 12 GB are the 1-bit and
2-bit ones (6.19 to 9.95 GB) and `UD-IQ3_XXS` at 10.93 GB, and the docs say the 3-bit files were
not updated for V3.1. On a 12 GB card the arithmetic from the first section is all that is left:
a 9.95 GB file and a few hundred MiB of state and cache leave no room for a second stream, and the
card's bandwidth divided by 9.95 GB is the most one stream can get before any speculation.
## What I want printed next to every tok/s
None of this week's numbers is false. Two of the three were labelled correctly in the place they
were first printed, and lost the label on the way to a headline. What a claim needs is short:
- **Streams, and how they relate.** Eight lanes of one prompt, 64 random requests and one user
are three different workloads.
- **Per-stream speed at that concurrency**, which is the number a person feels.
- **Context per stream.** 640 tokens and 150K tokens are different memory budgets by a factor of
234, and on this model the recurrent state comes first anyway.
- **Rows verified per step and acceptance**, whenever there is speculation, because that is
arithmetic the batch cannot also have.
- **Decode or end to end, and what was left out.** TensorFold's decode window excluded 1.72 s that
its own line does not name; the ninfer fork excludes the tail and says so.
My own reading: HyperQwen's 1,035 is the most useful number of the three, because it comes with a
protocol, a per-token latency and a README that tells chat users to use the other mode. The
eight-lane videos are real batch-8 results on a card that has the bandwidth for it, and both sit
where the roofline says they should. NInfer had 766.6 at eight streams before either was posted. I
don't buy "twice as fast": like for like, the Cinference run was 1.4× TensorFold's, and its
author, to his credit, called the whole format a magic trick before anyone else did.
If you have one RTX 5090 and you are the only user, the number to compare is single-stream decode
on your own prompts: 71 without drafts and somewhere between about 140 and 450 with them,
depending on how predictable your output is. If you are serving a pipeline, batch until the
per-stream speed hits the floor your clients tolerate, and read the memory wall off the state size
before the KV size.
The videos show lanes labelled 3/8, 4/8 and 5/8 writing different sections at the same time, the
stats lines say "8 parallel code lanes", and Cinference's `--max-concurrency` tops out at 8. If
TensorFold's uLane or omp's fan-out turns out to be a single sequence with a structured draft tree
that merely renders as lanes, the per-stream column for those two rows is wrong by a factor of
eight and the comparison with NInfer's C=8 point is coincidence.
Computed from `config.json` (48 DeltaNet layers × 48 value heads × 128 × 128) and matched by
HyperQwen's 37-of-64 log line at FP32 and ~70 at FP16. If an engine shares or compresses the
DeltaNet state across streams, or stores it below 16 bits, this stops being true for that engine.
HyperQwen's own description (fp16 tensor-core bound at 40-64 sequences, ~63 TFLOPS) and the
51.2 GFLOP per token I count from the config. A cohort run at C16 and C32 in batch mode would
settle where the bend actually sits; the README publishes C1 to C8 and C64 only. My own ladder in
the update puts the start of the bend between 16 and 32 streams on a 3090 at 390 W, with 32 to 64
still buying 22% to 34% more aggregate.
---
Read, not run, except the 12 November update. The model geometry is from `Qwen/Qwen3.8-27B`
`config.json` (commit `37561816`).
HyperQwen at `3d6c954`, Cinference at `383e5db`, `fast-long-context-cinference` at `bae8a31`,
`fafstmobel-cinference` at `dd3c3c2`, the ninfer fork at `c3bb9df` and TensorFold at `000b534`
were cloned and read. The video figures are read off frames of the posted videos, and the X
threads through the fxtwitter mirror. Every per-stream figure and every ceiling is my
arithmetic on those numbers. I have no RTX 5090. The update's runs were on one RTX 3090 at its
390 W default limit: HyperQwen at `3d6c954` on its pinned vLLM 0.33.0 with
`dbirks/Qwen3.8-27B-W4A16-AutoRound` at `1f05c44`, and qwen-warp at `031d02a` with
`ukisai/Swift-1.5-Qwen3.8-27b-INT4` at `81a28ee`, each with the protocol described there. I did not
run HyperQwen's quality battery; one arithmetic prompt to each server came back right. The NInfer
DSpine figures are from its `v1.2.5-dspine.3` release notes.