Ripple-65B Preview: a dense 65B from scratch, read from its headers and run on a DGX Spark budget

mdjsonmcpcopy↗ai

2026-11-09 · 16 min · attention · kv-cache · speculative-decoding · agentic-coding · quantization · inference · tokenization

Why read this

Notable· top 60%

Ripple-65B counted from its headers, its value-residual weights read from the checkpoint, and the DGX Spark claim turned into bytes per token.

• Original analysis
• Widely used
• Concrete numbers to act on

·LLM architecture·Needs a workstation GPU·Apache-2.0·Practitioner model

How this was scored

Is it new?

1 of 3: An incremental tweak

Can I trust it?

2 of 3: Measures key facts from files, code or configs

Can I run it?

2 of 3: Open code or weights with real limits

Will I understand it?

2 of 3: Mechanism from first principles with figures

Can I act on it?

2 of 3: A concrete recipe, numbers or comparison

Will it last?

1 of 3: Relevant for months

Does it affect many?

2 of 3: A widely used model, tool or lab release

Only here?

2 of 3: A teardown or measurement few others did

Score 62 of 100, ranked 369 of 701 rated articles. Each question is answered 0–3 by hand, and a 3 is rare. How articles are scored

On 4 November Ricursive Intelligence posted: "Today we release Ripple-65B Preview, our 65B dense LLM we pretrained from scratch." The same post claims "a new state-of-the-art for U.S. open-weight models in agentic coding tasks, surpassing significantly larger models including Muse Spark Open-816B, Beam-501B, Inkling-975B, and Nemotron 3 Ultra-550B." An hour later a widely shared repost added that it "even runs locally on my DGX Spark".

Every MoE I have covered this autumn was a statement about sparsity. This one is the opposite bet: every parameter runs for every token. That makes the hardware question simple arithmetic, and the architecture is unusual enough to be worth reading out of the files. Three things first:

• The count holds. The 16 shard headers list 64,782,689,360 parameters, all BF16, the same number Hugging Face's API reports. "65B" is that, rounded, with the vocabulary tables included.
• BF16 does not fit a DGX Spark. The tensors alone are 129.57 GB. Ricursive's own Spark recipe runs a separate NVFP4 checkpoint, 57.21 GB, plus a 4B drafter.
• The coding lead is 0.4 points. 80.5 against 80.1 on Terminal-Bench 2.1, in a table whose own harness row moves Beam from 80.1 to 70.8.

Throughout, measured means I computed it from a file or a run. Reported means the publisher's figure, not re-run. Reasoned means my arithmetic on the other two.

What is public

Four repositories under ricursiveAI, plus a launch blog with no paper (measured, file sizes from the Hugging Face API on 10 November):

The BF16 repository also carries modeling_ricursive.py (a transformers implementation, trust_remote_code), an SGLang port under inference/server/, and chat_template.jinja. The NVFP4 repository adds inference/local/spark.py and a notebook titled "Ripple on DGX Spark". The LICENSE file is the stock Apache 2.0 text. There is no training code and no GitHub repository.

Within six days the community had posted more than fifty derivatives. The most downloaded is a llama.cpp GGUF set, which says Ripple support was merged into llama.cpp as release b12167. I use two of its files below for Q8 and Q4 sizes.

The architecture, read from the headers

A safetensors file opens with an 8-byte length and a JSON header naming every tensor's dtype, shape and offset. Two range requests per shard give the whole model without downloading 130 GB. The script is twenty lines of standard-library Python.

Here is where the 64.78B parameters sit (measured, sums of tensor shapes over 80 layers):

Read against config.json and the modeling code, that is:

• Thin and deep. 80 layers at width 5,120. The blog calls it "a thin-and-deep model aspect ratio". Llama-class 65-70B models use 8,192 wide and 80 deep, so Ripple puts the same depth on a narrower stream.
• An 8x MLP. intermediate_size is 40,960, eight times the width, against roughly 3.5x in Llama-style SwiGLU. Four of every five parameters are MLP, which is why the NVFP4 build quantizes only those and still saves 56% of the file.
• Twice as many query heads as the width needs. 80 heads of 128 make a 10,240-wide query against a 5,120-wide stream. The blog's phrase is "we overprovision the number of attention heads". Only 8 KV heads exist, so ten query heads share each one.
• A gated output. g_proj is a third 5,120 x 10,240 matrix whose sigmoid multiplies the attention output elementwise before o_proj, computed in FP32 (gate_full_precision: true). The gate alone is 4.19B parameters, 6.5% of the model.
• Four norms per block. The code calls it PeriNorm: RMSNorm before and after attention, before and after the MLP, plus per-head RMSNorm on Q and K.
• Clipped activations. The MLP clamps its gate branch above 20 and its up branch to plus or minus 20 before SwiGLU (ffn_clamp_threshold: 20.0).
• Scaled embeddings. embed_scale is 71.554, which is the square root of 5,120.

Four sliding layers, then one global

layer_types is four sliding_attention layers followed by one full_attention, repeated sixteen times. The sliding layers see the last 1,024 tokens and use RoPE with theta 10,000. The 16 global layers see everything and have no positional encoding at all: the code builds no rotary embedding for them. max_position_embeddings is 2,097,152.

This is the same trick as Muse Spark Open, with a harder ratio (4:1 instead of 3:1) and a shorter window (1,024 instead of 2,048). For the cache it means (measured from the config, reasoned into bytes):

• Each layer stores K and V for 8 heads of 128 in BF16: 4,096 bytes per token.
• The 64 sliding layers stop growing at 1,024 tokens: 0.27 GB, whatever the context.
• The 16 global layers grow by 64 KiB per token: 8.86 GB in total at 131,072 tokens, 68.99 GB at 1,048,576.
• With every layer global, 1M tokens would need 343.6 GB.

So the claimed 1M-token context costs 69 GB of BF16 cache, more than the NVFP4 weights. On a Spark the official recipe caps it at 131,072 (--context-length 131072 in spark.py).

Value residuals, read out of the checkpoint

The last addition in the blog's list is "value residuals". In the code it is three lines. Every layer has one learned scalar, value_res_lambda, and computes the input to its V projection as a blend:

Here $x$ is the layer's normed input and $e$ is the token's own embedding from layer 0, scaled by 71.554. Q and K still read $x$. So each layer can pull its values straight from the token identity, skipping 79 layers of mixing.

Those 80 scalars are in the checkpoint, 2 bytes each, so I range-read them (measured). One layer stands out: layer 6 has $\lambda = -11.94$, so $s$ is zero to four decimals and its values come entirely from the embedding. Layers 14 to 39 mostly sit above $\lambda = 5$, where $s > 0.99$, and ignore the embedding.

The raw weight understates the embedding's pull, because the two inputs are not the same size. The scaled embedding has an RMS of 3.08 (median over 48 sampled vocabulary rows, measured). The normed input has roughly the RMS of its norm weight, 0.14 to 0.46 by layer (measured). Weighting each side by its size gives an estimate of the embedding's share of the V input (reasoned: it ignores how the two vectors correlate).

The pattern is a U. The first ten layers take most of their values from the token embedding. The middle third ignores it. From layer 40 on, the embedding comes back: it carries more than half the V input in 34 of the 80 layers, and under a tenth in 28. Ricursive does not discuss this. My reading, which I have not tested, is that the late layers want a clean copy of "which token is this" for predicting what comes next, and the value path is a cheap place to get it.

Running the tokenizer and the chat template

The tokenizer is a byte-level BPE with 100,000 merges and 100,278 entries, the same cl100k-style vocabulary Olmo 3 uses, with ChatML markers and fill-in-the-middle tokens added (measured, tokenizers 0.23.3). The embedding has 100,352 rows, so 74 are padding. One of them is used: the DFlash2 drafter's mask_token_id is 100,278, the first spare row.

On my samples (measured): modeling_ricursive.py encodes at 4.44 characters per token, the model card's Markdown at 2.86, the Apache licence at 5.00. All five tokenizer probe strings from ripple_format.py, including digits, contractions, whitespace runs and mixed scripts, round-trip exactly.

I rendered chat_template.jinja with jinja2 3.1.6 and encoded the result:

Four things show up only when you run it (measured):

• The tags are plain text. <think> is three ordinary pieces, <th, ink, >; <tool_call> is four. Only <|im_start|> and <|im_end|> are special. Worse for a streaming parser, the closing > merges with following newlines: > is id 29, >\n is 397, >\n\n is 1363. That is why ripple_format.py parses on text and works hard to make an unchanged assistant message re-render to the same tokens.
• Tools cost 165 tokens before the user speaks. One small tool definition plus the fixed instructions, with no system prompt.
• Tool results come back as a user turn. The template wraps consecutive tool messages into one <|im_start|>user block of <tool_response> elements.
• There are two templates. tokenizer_config.json embeds a five-line ChatML template with no tools and no reasoning. chat_template.jinja is the real one. A runtime that reads only the embedded one will drop every tool definition and every <think> block without an error.

clear_thinking=True drops earlier reasoning from history; the default keeps it. The template has no thinking switch and no effort levels.

Does it fit a DGX Spark?

A DGX Spark has 128 GB of unified LPDDR5x and 273 GB/s of memory bandwidth. Single-stream decoding on a dense model reads every weight once per token, so the bandwidth is the speed limit. The batching piece covers why that limit moves with concurrency; here I stay at one stream.

The fit, first (measured file sizes, reasoned fit):

• BF16 never fits. 129.57 GB of tensors. Even reading 128 GB as 128 GiB, 137.4 GB, that leaves 7.9 GB for the operating system, the runtime and the cache.
• The official Spark build is NVFP4 for the MLPs only. Its config quantizes every Linear with 4-bit floats in groups of 16, and its ignore list keeps all 80 attention blocks, the embedding and the head in BF16. The card calls it "a conservative quantization". It is 57.21 GB; the drafter adds 8.08 GB.
• GGUF covers the rest. The community set has Q8_0 at 68.84 GB and a plain Q4_K_M at 38.80 GB. Ricursive's own thread says the model fits "locally on a MacBook or DGX Spark", which is true of the 4-bit files on a 64 GB laptop.

Then the speed. The bytes read per token are every weight except the embedding table, from which one row is gathered (measured for the official files, estimated for the GGUFs). Divide 273 GB/s by that and you have a ceiling no engine can beat without speculation (reasoned):

The cache is part of the read: at 128K it adds 8.86 GB to every pass, which is why the right-hand column drops.

What speculation has to deliver

Neither Ricursive's card nor the launch posts give a Spark speed. Ricursive's Spark recipe turns on DFlash2 with 32 draft tokens, one request at a time. DFlash 2 drafts a whole block in one pass and has the target verify it in one pass, so each cycle reads the target, the 8.08 GB drafter and the 1.03 GB output head the drafter borrows: about 65.3 GB, or 4.13 cycles a second at 8K (reasoned).

The only measured Spark numbers I found are from a third-party server kit, rlaope/Ripple-65B-Spark-Server, on one GB10 at temperature 0 (reported): code 20 to 36 tok/s, average 26.5 over three prompts; prose about 9; and "plain decoding near ~5 tok/s". The plain figure sits right on my 4.8 ceiling. Its authors also found that cutting draft tokens from 32 to 16 helped, because "most steps accept far fewer than 32".

Divide their numbers by my cycle rate and you get a floor on how many tokens each verify pass keeps (reasoned). 26.5 tok/s needs at least 6.4 tokens a pass; 36 needs 8.7; prose at 9 needs 2.2. Those are floors, since no engine hits the bandwidth ceiling. So the Spark story is really a drafter story: code, which is repetitive and predictable, runs five times faster than the dense bandwidth allows, and prose barely doubles.

The datacenter number

The card also says the model reaches "250 tok/s on 2xB200 batch=8 prior to quantization or custom kernels" (reported). Two B200s give 16 TB/s, so BF16 weights cap one stream at 124.5 tok/s without speculation (reasoned). If 250 is per stream, it comes from the DFlash2 drafter that serve.py turns on by default; if it is the batch total, each stream gets about 31. The card does not say which.

The training claims

The model card's stage table (reported):

Some of the rest is checkable and some is not:

• From Olmo 3 to here. The blog describes a ladder of small-scale runs from Olmo 3's baseline: pre+post norm in v1.2, sigmoid-gated attention in v1.3, a rejected 2-stream mHC in v1.4, global NoPE in v1.5, value residuals in v1.6, then a code-heavy dataset. It claims 16x better compute efficiency on code at 4B scale, and 55x "for the full hero run" in the card (reported). There is no chart in the files to check it against.
• 40% MFU, BF16 throughout, "no custom kernels" (reported). Plausible for a dense model with simple sharding; that is the argument for dense in the first place.
• The compute is small. At $6ND$, 12T tokens on 64.78B parameters is $4.66 \times 10^{24}$ FLOPs of pretraining. Assuming B200s at 2.25 PFLOP/s dense BF16 and the stated 40% MFU, that is about 1.44M GPU-hours (reasoned; Ricursive does not name its hardware).
• The money is stated two ways. The launch thread says it "was built on a $32 million compute budget". The blog says "$32M is our full R&D cost starting from scratch", including ladders, evals, debugging and synthetic data. The second is the stronger claim. Neither is checkable.
• 1.4M RL rollouts, "roughly 70x fewer rollouts than Reflection Beam" (reported).

The benchmark table

The card compares Ripple with four US open-weight models on 22 rows. Two caveats are printed in it: other labs' missing scores were filled in with "our own in-house eval flow" (in italics), and "Replicating benchmark scores has been challenging, particularly for modern agentic evals".

Counting the table (measured, from the card):

• Ripple tops 3 of 22 rows: both Terminal-Bench 2.1 rows and SWE-bench Pro v2.0 Hard. Muse Spark Open tops 12, Reflection Beam 5, Inkling 2, Nemotron 3 Ultra 1.
• Against Muse Spark Open, Ripple wins 8 of 22. In the 10 coding and agentic rows it wins 5: both Terminal-Bench rows, SWE-bench Pro v1.0 and v2.0 Hard, and tau-3 banking. Muse wins DeepSWE by 15.7 points and MCP Atlas by 17.5.
• The headline margin is the smallest number in the argument. Terminal-Bench 2.1 is 80.5 against Beam's 80.1, both self-reported. One row down, Ricursive ran everyone through the official Terminus 2 harness: Ripple stays 80.5, Muse rises to 78.7, Beam falls to 70.8. A 9.3-point harness swing for one model, next to a 0.4-point lead, says the lead is inside the noise of how the test is run. Ricursive says as much: "our results represent a best-effort attempt at a fair comparison."

What survives is the size claim. A 65B dense model is in the same band as 500-975B MoEs on agentic coding, at 23B to 55B active parameters for them against 65B for Ripple. On active parameters per token the comparison is close to even; on memory it is not, and memory is what decides what fits on a desk.

What I ran, and what I did not

Ran (measured):

• Range reads of all 16 BF16 shard headers and one NVFP4 shard header; parameter counts by tensor group.
• Range reads of 80 value_res_lambda scalars, 240 norm weight vectors and 52 embedding rows from the BF16 checkpoint.
• The checkpoint's tokenizer.json in tokenizers 0.23.3, and chat_template.jinja in jinja2 3.1.6, on the probes and conversations above.
• Counts and comparisons over the card's benchmark table.

Did not run:

• The model. 130 GB of BF16 on a 4-vCPU box with no GPU is not a run; neither is the NVFP4 build, whose kernels need Blackwell.
• modeling_ricursive.py itself. I read it and re-implemented the pieces I needed in arithmetic; I did not install torch to run it.
• Any benchmark. Every score in this piece is Ricursive's, or another lab's, as printed.
• The Spark speeds. Those are a third party's, from one GB10, and I could not reproduce them.

Should you run it?

If you have a 128 GB unified-memory box and your work is agentic coding, yes: it is Apache 2.0, the official NVFP4 recipe is one notebook, and code is exactly the workload where the drafter pays. Without the drafter, budget 4 to 5 tok/s. With it, expect about 9 on prose and 20 to 30 on code, per the only measurements so far. If you have a 64 GB machine, the 4-bit GGUFs fit with modest context; the ceiling there is set by your memory bandwidth, and the table above gives the bytes.

If what you want is the most capable open model per byte of memory, this is the first dense entrant in a while that makes the argument. If you want the most capable open model, the card's own table says Muse Spark Open, by 12 rows to 3.