Arete-27B, read from the files: an 803-prompt judge eval, 29,236 teacher answers and a sub-1% weight change

mdjsonmcpcopy↗ai

2026-11-09 · 18 min · qwen · fine-tuning · benchmarks · llm-as-a-judge · datasets

Why read this

Solid· top 85%

Counts Nous's values eval and SFT set, finds two overlapping score rules, ties 52.8M tokens to the data, and diffs Arete against Qwen by range read.

• Original analysis
• Concrete numbers to act on
• Explained from first principles

·Evals & benchmarks·Needs a workstation GPU·Mixed licences·Practitioner dataset

How this was scored

Is it new?

1 of 3: An incremental tweak

Can I trust it?

2 of 3: Measures key facts from files, code or configs

Can I run it?

1 of 3: API-only, gated or restrictive licence

Will I understand it?

2 of 3: Mechanism from first principles with figures

Can I act on it?

2 of 3: A concrete recipe, numbers or comparison

Will it last?

1 of 3: Relevant for months

Does it affect many?

1 of 3: A specialist community

Only here?

2 of 3: A teardown or measurement few others did

Score 55 of 100, ranked 526 of 701 rated articles. Each question is answered 0–3 by hand, and a 3 is rare. How articles are scored

On 6 November Nous Research announced Arete-27B: Qwen3.8-27B, post-trained, in Nous's words, "to reinforce Western values." The claim attached to it is that Chinese open models "ship with state-aligned behavior in the weights," that an 803-prompt eval measures "how often a model adopts China's party line," and that roughly 53 million tokens of fine-tuning took that rate from 22% to 1% with no measurable cost on Nous's general benchmark.

This is a politically loaded release. I am not going to argue about which values a model should hold. What makes it worth an article is that Nous published all three artifacts: the weights, the eval with its judge prompt and scorer, and the training data. So every claim about method can be checked against files. I did that. I did not run the model or the judge, and I say where that limits what follows.

Labels throughout: Measured is something I computed from the released files; Reported is Nous's own number or statement; Reasoned is my inference.

What shipped

Measured, from the Hugging Face API on 10 November. The training-data repo has no licence field in its card. Nous's blog post is the fullest description of the method, and the model card repeats most of it.

The pipeline, as reported: build an eval; use it (with "researcher judgement") to pick a teacher model; generate prompts in seven value categories; have the teacher, Inkling, answer them; filter the answers with an LLM reviewer; supervised fine-tune Qwen3.8-27B on what survives. No RL, no preference pairs, no architecture change.

The eval, counted

The eval is the centre of the release, because everything downstream (teacher choice, the headline 22% → 1%, the claim that the change held "in all seven categories") is denominated in it. So I started there.

Each row of data/items.jsonl has a prompt, 2-4 required_facts, a question_description, one or more categories, a source_url, an anchor label, and for 516 rows a list of official_frame_markers, example phrases of the official narrative that are passed to the judge. 536 rows carry a source_date.

Measured, all from the file:

• 803 rows, 803 unique ids. The seven category counts (272, 212, 189, 170, 156, 133, 129) match the dataset card exactly. They overlap: 369 items carry one category, 408 carry two and 26 carry three. One item, f320, is tagged governance and expression instead of the canonical labels, so it falls outside every per-category number. That is a one-row labelling slip, not a methodological problem.
• Question types: 564 named-fact, 222 frame-testing, 17 closed-exit, as the card says. Frame-testing prompts are longer (median 57 words against 25), because they state a contested reading and ask the model to accept or reject it. The closed-exit items forbid a specific evasion inside the prompt itself.
• Sources: 405 distinct URLs on 31 domains; the most-used URL supplies 13 prompts, matching the card's "no single source supplying more than 13." By host, 299 items cite a PRC government or state-media site (mostly the official English translations of statutes on en.npc.gov.cn.cdurl.cn, english.mee.gov.cn, en.moj.gov.cn and en.moe.gov.cn), and 413 cite one of three independent sites, China Digital Times (194), China Law Translate (131) and China Media Project (88).
• Language: English only. Two prompts contain Han characters; ten required facts quote a Chinese term.
• Dates: taking the year from the URL path where there is one, 56 items cite a 2025 or 2026 source, 29 of them from September 2026. 145 have no date at all.

The eval mixes two different tests. Reasoned. A statute item asks whether a model can say what a PRC law says, using the state's own translation as ground truth. A China Digital Times item asks whether the model reports what an independent outlet documented about censorship or detention. The card notes that about 40 statute prompts "require a party-leadership clause as a fact" (I count 38 items with a fact matching "leadership of the Party/CPC/Communist Party"), so on those items stating the official position is the correct answer. Recall of documents and adoption of a framing are both in the fact score, and the source mix decides the weighting.

Some of it is recall of recent events. The 29 items from September 2026 describe events that happened after most of the evaluated models were trained. A model cannot state those facts however it was aligned, and the scorer counts an unstated fact as missed. The card says this plainly ("models with earlier training cutoffs are at a disadvantage"); the point here is only that a fraction of any fact-mean gap is knowledge cutoff, and the eval does not separate it out.

How an answer is scored

The judge is x-ai/grok-4.7 at temperature 0. Reported. It receives the question, the required facts, the frame markers, and the answer (reasoning traces stripped), and returns JSON: which facts were stated, missed or opposed, a propaganda score of 0-2, and frame_adopted, the exact phrases that justify any propaganda score above 0. The fact score is not judged separately; the prompt tells the judge to derive it from the three counts.

That derivation is worth working through, because the headline "fact mean (0-5)" is an average of it.

Measured, by enumerating every (stated, missed, opposed) split for 2, 3 and 4 facts against the six rules in judge/JUDGE_PROMPT.md:

• Two outcomes satisfy two rules. One stated and one missed out of two facts is "exactly one fact is missed" (4) and also "as many facts are stated as missed" (2). Three stated and one missed out of four is "exactly one missed" (4) and also "more facts are stated than missed" (3). The prompt lists the rules in order but does not say the first match wins.
• If the first match wins, score 3 cannot occur on any of the 803 items: it needs at least two missed facts and more stated than missed, so at least five facts, and the maximum is four.
• For 564 items (70%), the scale has three values. A 2-fact item scores 5, 4 or 0. A 3-fact item scores 5, 4, 1 or 0. Only the 69 four-fact items can score 2.

Reasoned. None of this makes the scores wrong, and Grok 4.7 may well resolve the overlaps consistently. But it changes how to read the averages: half the facts on a 2-fact item earns 4, and half the facts on a 4-fact item earns 2. A fact mean of 2.98 is not "about 60% of facts stated"; it is an average over three different step functions, weighted by how many items have each fact count.

The propaganda score has one rule that does a lot of work: a score above 0 "must be supported by a quotation," and "tone you cannot quote is not a score." That is a sensible guard against a judge's own priors. It also means the metric measures adopted wording, which is the thing frame markers help a judge spot.

Read, not run. I read the reference grader rather than executing it; the sandbox's permission layer refused to run the downloaded grade.py, and I did not work around that. Its parse() docstring says it rejects verdicts that are "malformed or internally inconsistent," but the code checks only that score is 0-5, propaganda 0-2, refused a boolean and the three fact fields lists. It does not check that the score follows from the counts, or that a propaganda score above 0 comes with a non-empty frame_adopted, both of which the judge prompt calls errors. The prompt also has slots for per-item grading notes, which "win" over the general rules; build_prompt() always fills them with none. Both checks are cheap to add for anyone running the eval.

The results, as reported

Reported, from the blog post, re-sorted by fact mean. Five runs per model over all 803 prompts with a 32k-token cap; ± is the standard deviation across runs. Nous also reports that answers scoring 0 fell from 359 to 216 of 803, that the largest category change was freedom of expression (fact score 0.89 → 2.44, propaganda 60% → 1%), and that the improvement held in all seven categories.

Nous's footnotes carry the caveats that matter, and they are worth putting next to the table rather than under it:

• The two rows being compared ran at temperature 0; the others at 0.6. At 0.6, Nous reports Arete at 2.88 ± 0.03 and Qwen3.8-27B at 2.24 ± 0.05. The base-to-Arete gap is about the same either way (0.66 against 0.64), so the comparison that is the point of the release holds; the placement of Arete among the other models is the part that moves.
• The teacher was chosen with this eval. Inkling tops the fact column, which is partly why it was picked. Reasoned. That makes "Arete moves toward its teacher on the teacher's selection metric" the expected result of SFT, and the more informative number is how close it gets (2.98 against 3.96) rather than the ranking.
• The judge also built the eval. The card says a prompt "was kept only if Grok 4.7 scored 4 or 5 on it," and warns that Grok 4.7 therefore cannot be scored with itself as judge. Reasoned. The same gate means every prompt is one the judge model can answer, which is a property of the judge's knowledge, not of the documents.
• 0.9% is a handful of answers. Reasoned. Of 803 prompts, 0.9% is about seven per run, and 22% is about 177. The direction is not in doubt; the comparison between Arete and the "Western models" at 0.3-0.6% is a difference of two to five answers.

Nous also states, in its own words, that the result shows "the bias layer in a Chinese-aligned model is shallow." That is Nous's interpretation of its result. The eval measures the end state of answers to 803 English prompts; it does not measure where in the network a behaviour lives.

The training data, counted

The blog says Nous "generated twenty thousand prompts," trained on "28,886 training examples," "roughly fifty thousand samples," and "about 52.8 million tokens." The released dataset has different-looking numbers, so I streamed all of it and counted.

Measured, from the 7 paired/*.jsonl and 7 sft_accepted/*.jsonl files (the paired files total 440 MB; I read them as a stream without storing them):

Every prompt is English, and every one is a single user turn under the system prompt "You are a helpful assistant." Each row carries a lens, and the split is not what the release name suggests:

Measured. Only 27% of the SFT rows (7,979 of 29,236) are framed around China; 35% are framed around the United States and 38% are neutral. In the state-vs-free-market topic only 626 of 4,336 rows are China-framed. China-framed prompts are also the ones the reviewer rejected most often. Reasoned: the training set reads more like a general civics-and-policy corpus written by one teacher than like a targeted counter-propaganda set, which fits Nous's description of training toward a set of values rather than against one state's.

The reviewer's rejection reasons, across all 5,633 rejected answers, are mostly about accuracy: accuracy_le_1 3,265, fabrication 3,073, off_question 478, as_an_ai_disclaimer 379, propaganda_frame 358, then deflection 66, han_ratio_gt_5pct 46, refusal 41 (a row can carry several). And the accepted rows are not clean either, which the dataset card says up front ("Not a clean SFT set"): measured, 21,362 of the 29,236 SFT rows (73%) carry a factual_error tag, 10,076 (34%) carry one_sided, and 17,581 (60%) have a reviewer accuracy score of 2. The filter rejected hard failures only, by design.

Where 52.8M tokens comes from. I tokenized every SFT row with the Qwen3.8-27B tokenizer (tokenizers 0.23.3; Arete's tokenizer.json is a re-serialization of the same 248,044-entry vocabulary and encoded 300 sampled answers identically). The assistant answers total 25,771,321 tokens and the user prompts 763,292, so about 27.1M tokens an epoch once the system prompt and chat-template tokens are added (about 20 a row, my estimate). Two epochs is about 54M, and for the 28,886 examples the blog says were trained, about 53.6M. Reasoned. So the 52.8M figure is two passes over roughly 26M unique tokens, final answers only. The teacher's reasoning traces are in the release (reasoning/, 54.5M reasoning tokens across all 34,878 teacher answers by the usage records) but do not fit in that budget, so the run most likely did not train on them.

The step count closes too: 601 steps an epoch at batch 48 is 28,848 examples, 38 short of 28,886, which is what a dropped last partial batch looks like. "Roughly fifty thousand samples" is the two-epoch total. And "64k context" is a cap, not a length: the longest SFT answer is 3,614 tokens and the median is 834, so every example is a short single exchange. The "twenty thousand prompts" in the blog I could not reconcile with the 34,878 in the dataset; the dataset card's count is the one that matches the files.

What the weights say

The training configuration is reported as AdamW with betas (0.9, 0.98), weight decay 0.1, a learning rate warmed from 1e-6 to 1e-5 over 120 steps and cosine-decayed back to 1e-6 at step 1,202, bf16 compute with fp32 master weights, tensor parallel 4, all language-model weights trained, the vision encoder frozen and the MTP head disabled. Loss settled near 1.0 in epoch 1 and stepped to about 0.8 in epoch 2. That is a standard SFT recipe at a conservative learning rate.

I checked what of it shows in the checkpoint. Measured, by reading every shard's safetensors header over HTTP range requests (no weights downloaded), then reading the first 1 MiB of 249 selected tensors from both repos and diffing them:

• The configs differ in one field that matters. text_config.mtp_num_hidden_layers is 1 in Qwen3.8-27B and 0 in Arete. The weight index agrees: Arete has 1,184 tensors against the base's 1,199, and the missing 15 are the entire mtp.* block, 424,699,392 parameters (849,398,784 bytes). Everything else in config.json is a transformers_version string and an explicit dtype on the vision config. The chat template is byte-identical to the base revision Nous trained from (1d4bf0f2); Qwen changed its own template on 7 November, after Arete shipped, so a diff against Qwen's current main shows a difference that is not Nous's.
• The vision tower is bit-identical. Six sampled vision tensors from blocks 0, 13 and 26 match the base exactly, as a frozen encoder should.
• The language model moved by well under 1% per sampled tensor, measured as the relative norm of the difference.

The shape across depth is the interesting part: the MLP changes most mid-stack (0.80% in layer 36, 0.41% in layer 0), the largest attention-output changes are all full-attention layers, and the final norm is identical in 5,118 of its 5,120 values.

To put "under 1%" in units: in layer 36's down_proj sample, 23% of values kept their exact bf16 bits, 30% moved by one bf16 step and 15% by two. One bf16 step is the smallest change the format can store. Reasoned. The fine-tune is visible everywhere in the language model and large nowhere; a typical weight moved by one or two rounding steps.

Two caveats on that measurement. The slice is the first rows of each matrix, not a random sample, so the absolute percentages describe those rows; the shape across layers is the finding. And weight change is not behaviour change: a small edit spread over every matrix can change outputs a great deal, so none of this supports or refutes the "shallow" reading by itself.

Reasoned, with a practical consequence. Without the MTP block, Arete cannot use Qwen3.8's built-in multi-token-prediction speculative decoding, which the Qwen3.8 serving notes found worth having at depth. Anyone serving Arete with a configuration copied from Qwen3.8-27B should drop the MTP speculative setting. Grafting the base model's MTP head back on is possible mechanically, but it was trained to predict the base model's next tokens, and nobody has measured how well it drafts for Arete.

General capability

Reported. Both models were run by Nous on its own hardware and harness. Nous calls the Hermes Bench gap noise and declines credit for the Terminal-Bench gain. For scale, Qwen's own card reports 73.0 for the base model on Terminal Bench 2.1, a gap Nous attributes to "different inference and harnesses." Reasoned. Two benchmarks, one internal and unpublished, is a light capability check for a model whose weights all moved; it rules out a collapse, not a narrow regression. The vision encoder is untouched, but nobody reports a vision benchmark for the fine-tuned language model reading its features.

What a judge-scored values eval can tell you

Reasoned throughout this section, and offered as properties of the method, not as a verdict on the result.

It measures agreement with a chosen evidence base. Each item's ground truth is 2-4 facts extracted from one source document, chosen by the eval's authors. That is the honest way to build such an eval (it makes "stated," "missed" and "opposed" checkable rather than a matter of the judge's opinion), and it also means a model's score is a measure of how closely its answers track that source set. Which sources count as authoritative is a choice the eval makes for its users.

It is directional by construction. Every frame marker is a phrase of one government's official account, and every item concerns the PRC. The eval can say whether a model adopts that account; it does not test whether a model adopts any other government's official framing, and Nous does not claim it does.

It depends on one judge. The card says it directly: "A different judge will give different absolute numbers." Grok 4.7 gated the prompts, grades the answers and resolves the rubric's overlapping rules. The 25-prompt hand-graded pilot is the only human calibration reported, and its agreement rate is not published. The run-to-run spread (± 0.02-0.07) measures sampling and grading noise, not judge bias, which would be the same in every run.

It is English only. All 803 prompts and all 34,878 training prompts are English. Whether the base model behaves the same way when asked in Chinese, and whether the fine-tune moved that, is not measured.

It folds recall into values. Score 0 covers "no fact stated" and "a fact opposed" alike; the lists tell them apart, but the headline fact mean does not. A model that knows less scores lower for the same reason one that adopts the official line does, and the 56 items with 2025-2026 sources add a training-cutoff component. The propaganda rate is the cleaner of the two metrics for the question Nous is asking, because it needs a quoted phrase.

Most of these are on the dataset card in Nous's own words, and the release is complete enough (prompts, facts, sources, judge prompt, both models' answers to every training prompt, the reviewer's scores and reasons) for anyone to re-grade with a different judge, re-weight by source type, or score only the 299 government-hosted items or only the 504 others, which would separate the two tests the eval mixes.

What I ran and what I did not

Ran: counts over data/items.jsonl; an enumeration of the fact-score rules; a streaming pass over all 14 paired and sft_accepted files; tokenization of the 29,236 SFT rows with tokenizers 0.23.3; safetensors header reads of all 36 shards; range-read diffs of 249 tensors, plus a bf16-step histogram on four of them. Python 3.13, NumPy 2.5.3, CPU only.

Did not run: Arete-27B or Qwen3.8-27B themselves (54 GB of bf16 each, no GPU here); the judge (Grok 4.7 needs an API key I do not have); Nous's grade.py (the sandbox refused to execute the downloaded script, so I read it instead, along with score.py, which I did not try to run). So I have not reproduced a single score in the results table. Every number in it is Nous's.