tokin: training an agent on the tokens it actually sampled

A coding agent is two programs. One is the model. The other is the loop around it: the thing that sends the conversation, reads the reply, runs pytest, pastes the output back and asks again. People call that loop the harness, and the useful ones (Claude Code, Codex, pi, opencode) were written to talk to a chat API. They send a list of messages and get a message back.

Reinforcement learning does not care about messages. It needs the exact token ids the policy sampled, with the log-probability it sampled each one at, because that is what the gradient is taken through. Put a stock harness in front of a stock inference server and the ids get thrown away on every turn. The server decodes them to text, the harness keeps the text, and the next request is text that the server encodes again. Nothing promises that the second encoding gives back the first ids.

tokin, by Yuan He, released as v0.1.0 on 11 November, is a gateway that sits in that gap. The harness keeps speaking chat completions to it. The engine only ever gets ids. In between, tokin keeps every id the model sampled and never re-encodes one. I read the code at the v0.1.0 tag (504573c, 2,027 lines of Python under src/tokin), the release blog, the announcement thread and the pi_mimo_rl training recipe, and I ran the actual tokenizer and chat template of Qwen3.5-35B-A3B, one of the two models it trained, to see how often the round trip breaks.

Why decode then encode is not the identity

Start with the tokenizer. Qwen3.5 uses byte-level BPE with a vocabulary of about 248,000 entries. Encoding is deterministic: a regex splits text into chunks, and within each chunk the merges are applied in a fixed priority order. For the string " rollouts" that procedure produces two tokens, " roll" (id 6299) and "outs" (id 11336).

Sampling is not that procedure. The model emits one id at a time from a distribution over the whole vocabulary, and nothing stops it from emitting " rollout" (69293) followed by "s" (82). Both sequences decode to the same seven characters. I counted how many ways the vocabulary can spell that word by walking every segmentation of its byte string into vocabulary entries: 189. " tokenization" has 2,326 and " retokenization" has 8,396. The encoder knows exactly one of them. If the model wrote any of the others, decoding and re-encoding hands the trainer a different sequence of the same length.

That case is the subtle one, because the lengths match. A trainer that lines up the recorded logprobs by position would score " roll" with the logprob that belonged to " rollout", and nothing would crash.

The blog's Figure 2 shows exactly this split with ids 71184/82 against 6502/11666. Those ids did not match Qwen3.5 or Qwen3.8 when I tried them. They match GLM-4.5's tokenizer.json, which tokin also supports. Same idea, different vocabulary; the ids in the widget below are Qwen3.5's.

The chat template is a second encoder

The harness does not send ids back, it sends a message: {"role": "assistant", "reasoning_content": …, "tool_calls": […]}. The server turns that structure into text with the model's Jinja chat template, and the template has opinions. In Qwen3.5-35B-A3B's template these are the ones that move ids:

• It trims. Reasoning is written as "<think>\n" + reasoning|trim + "\n</think>\n\n". If the model sampled a blank line before </think>, the trimmed version has one newline, and "\n\n" (id 271) becomes "\n" (id 198).
• It rebuilds tool calls from parsed arguments. Whatever whitespace or argument order the model wrote, the re-render prints the template's canonical <function=…><parameter=…> block.
• It drops old reasoning. Every assistant reply before the last user message is rendered without its think block. The moment a harness appends a new user message, the reasoning of earlier turns vanishes from the prompt.

Special tokens do help a little. <|im_end|>, <think>, </think> and <tool_call> are added tokens, so the pre-tokenizer cuts the text at them and BPE never merges across a message boundary in ChatML. The drift lives inside the segments those tokens fence: the words, the whitespace glued to the tags, the template's rewrites.

I built one assistant turn by hand (a short thought, then a bash tool call that runs pytest -q tests/test_io.py), encoded it as if the model had sampled it, then rendered the next request through the real template and encoded that. The widget shows the result id by id.

With everything canonical, the 48 sampled ids come back unchanged. Turn on the non-canonical split and two ids change, same length. Add the blank line and a third changes. Add a user message and the template deletes 18 ids, the two that open the think block and all 16 reasoning ids the model sampled. The text in this demo is constructed, so it says nothing about how often a real policy samples a non-canonical spelling. It does show that each of the three mechanisms is real for this checkpoint, with its own files.

The control case also checks out. When the next request is a tool result, the prompt ids plus the sampled ids plus the encoded increment (80 ids) are identical to encoding the full re-rendered conversation. The re-render can agree. It just isn't obliged to.

What drift does to the gradient

Write a trajectory as prompt ids $x$ and sampled ids $y_1 \dots y_T$, drawn from the rollout policy $\mu$ (the weights the engine had when it sampled). Policy-gradient methods in the REINFORCE family push on

where $\pi_\theta$ is the policy being trained, $A$ is the advantage (here the trial's reward, centred within its group) and $\rho_t$ is the importance ratio that corrects for the trainer and the sampler disagreeing. The sampler records $\log \mu(y_t \mid \cdot)$ for each id at generation time. The trainer recomputes $\log \pi_\theta$ on whatever sequence it is handed.

Hand it a re-encoded sequence $\hat y$ and every term goes wrong at once. The numerator is evaluated on $\hat y_t$, an id the policy never chose, with a context $\hat y_{<t}$ the policy never saw. The denominator is still the recorded $\mu(y_t)$ for the id that was chosen. If the lengths differ, as when the think block disappears, the recorded logprobs no longer align with anything. And the gradient raises or lowers the probability of $\hat y$, which reinforces a spelling the model did not produce.

Truncated importance sampling clips $\rho_t$ (the 35B recipe uses --tis-clip 2 --tis-clip-low 0), and that hides the damage rather than fixing it: a ratio computed on the wrong token gets clipped to a bounded wrong value. Mixture-of-experts models have one more coupling. Rollout Routing Replay (arXiv:2510.11370), R3 for short, records which experts the router picked for every token during sampling and forces the trainer's forward pass to use the same experts, so $\pi_\theta$ and $\mu$ disagree only through the weights and not through routing noise. R3 is indexed by token position. Shift or replace the ids and the replayed routing belongs to tokens that are no longer there.

What tokin does instead

tokin's blog states two properties. A harness only ever sees messages. And along every trajectory the ids extend as the messages do:

Here $x_t$ is the prompt ids for turn $t$, $y_{t-1}$ is what the model sampled on the previous turn, and $\Delta_t$ is the encoding of only what the template writes after that sample's stop token: the end of the turn, the new observation (a tool result, a user message) and the opening of the next assistant turn. tokin stores each turn's $\Delta$ and $y$, and the only ids it ever encodes are $\Delta$s. The sampled ids are reused verbatim. That is "token-in, token-out at every prefix", TITO in the literature.

expand

Each request goes through four steps in Gateway.chat:

1. Attach. Session.attach finds the turn this request continues: the deepest turn whose path the request's messages begin with, latest on a tie.
2. Render. ChatTemplate.apply_increment renders only the messages past that turn's path. It does this by rendering them after a stub history and subtracting the stub's own render, then strips the stop token the model already sampled, so $\Delta$ does not repeat it. If the template rewrites earlier text when a message is appended, it raises instead of guessing.
3. Generate. The engine (SGLang's native /generate) gets rollout.token_ids + prompt.token_ids and returns ids, logprobs and whatever else the session asked for. A generation that reports stop must end on one of the template's stop ids, or the gateway refuses it, because the next $\Delta$ assumes that id is there.
4. Parse. The template's parser reads reasoning and tool calls back out of the sampled ids, and that message goes to the harness. The turn is recorded only after all of this succeeds, so a failed request leaves the session as it was.

The parse step is where tokin depends on the harness's good behaviour. attach matches a previous reply by its text, its reasoning and its tool-call ids, and nothing else. It does not compare tool-call names or arguments, because harnesses rewrite them (the docstring names pi writing {} and opencode writing invalid), and the ids the model sampled stand for them anyway. But a harness that drops the reasoning from its history fails the match. The test suite says so in a test name: test_a_reply_whose_reasoning_was_dropped_opens_a_root. Such a harness still works, but every turn becomes its own root and the rollout is a pile of single-turn samples.

Two edge rules in the code matter. A turn cut at max_tokens is never continued, since it ended before its stop id and the next $\Delta$ would open on an id the model never sampled; the next request becomes a sibling under its parent, and the cut reply is rendered as prompt. And session parameters beat request parameters. If pi asks for temperature 0.7 and the trainer opened the session with temperature 1, the engine samples at 1 and the gateway logs the clash once. The policy is trained under the trainer's sampling, not the harness's.

A session is a tree, not a list

Harnesses do not grow history in a straight line. They retry a failed call, edit an earlier tool result, compact a long conversation into a summary, and spawn subagents that share the session. The attach rule turns all of these into tree structure without any harness-specific code: a retry or an edit continues an earlier turn and becomes a sibling; a compaction shares no turn's opening messages and starts a new root. Every root-to-leaf path is one rollout.

expand

The widget replays that session one request at a time and adds a compaction, then lets you pick a leaf and see its rollout as the trainer gets it.

When the trainer collects a session (GET /sessions/{id}, one safetensors payload that stores each turn once), the pi recipe builds one training sample per leaf. All leaves share the session's single reward and count once in the group mean. A turn that several leaves pass through is trained only in the first: to_samples in examples/pi_mimo_rl/generate.py keeps a set of turn ids already trained and sets trainable=False for the repeat. Toggle that off in the widget and the shared sampled ids are trained twice.

How much branching happens in practice? Less than the design suggests. The blog sampled 400 sessions from the 27B run: 85% had one rollout, 14.5% split at a compaction (which in tokin's terms is a new root, not a branch), 0.5% split at a turn cut on length, and none at a retry. For pi on MiMo coding tasks, the tree is mostly a line with the occasional fresh root. The machinery earns its keep on harnesses that edit history, and on the 14.5% where a compaction would otherwise have been glued onto the old prefix.

Prior art, and the one closest to it

This is not the first attempt. The blog's Table 3 compares three projects: Agent Lightning re-renders through vLLM and starts a new sample wherever the ids stop matching; strands-sglang keeps TITO only while the history grows and only for agents built on Strands; Miles keeps a tree in its experimental v2. The table leaves out the design I covered in October, Hugging Face's multi-harness RL capture proxy, and that is the useful comparison. The capture proxy also links calls into a graph, but by exact token prefix after vLLM has tokenized the harness's text. When the model sampled a non-canonical split, the prefix test fails and the next call opens a new root. Nothing wrong gets trained, but one rollout becomes several sequences, each with its own copy of the context; in that guide Claude Code's rollouts became about eight training rows each.

tokin matches on messages and substitutes the stored ids, so the same non-canonical split costs nothing. The price is that tokin must own the chat template. It ships four families (qwen25 for Qwen2.5 to Qwen3, qwen35 for Qwen3.5 and 3.6, qwen38, and glm45 for GLM-4.5 through GLM-5.3), each tested against the real tokenizers, and a model outside them needs a new subclass. The proxy works with any model vLLM serves.

TITO has a side effect. With Qwen3.5's template, a deployed agent never sees its own earlier reasoning once a new user message arrives. Through tokin it does, because the old think block is part of the stored ids. That is a distribution shift between training and deployment. The blog checked for it with the policy frozen: Qwen3.6-35B-A3B under mini-swe-agent, opencode and pi, on Terminal-Bench 2.1 (89 tasks × 4 samples) and SWE-bench Verified (500 × 1), once through tokin and once through the same gateway re-rendering every turn. Over 2,568 paired samples, which is $3 \times (356 + 500)$, pass@1 through tokin minus re-rendering was +0.1 points, 95% CI [−1.5, +1.7]. At evaluation time the difference does not show. At training time it is the whole point.

What a rollout carries

A collected rollout gives token_ids, a loss_mask that is 1 on every sampled id and 0 on every rendered one, and logprobs aligned to both. The rest exists because asynchronous, large-scale RL needs it, and each field is a place where re-encoding would also have broken something.

Weight versions, for partial rollouts. In fully asynchronous training the engine updates its weights while trials are still running. SGLang aborts in-flight generations at the swap. A session opened with resume_aborted continues an aborted generation from the ids already sampled instead of failing the turn, and Gateway.resume joins the two halves as if they were one. Each generation keeps weight_versions, runs of (version, start, end) saying which weights sampled which positions, so the trainer knows which $\mu$ each logprob came from. The pi recipe turns this on with --partial-rollout.

Top-p support, for score centering. Both recipes sample at temperature 1 with top-p 0.95. With return_support, each position records the ids top-p kept and their logprobs over that support, which slime replays for score centering. This needs the sampler patch that ships in slime's SGLang images; on stock SGLang tokin raises rather than returning nothing.

Routed experts, for R3. With return_routed_experts, SGLang returns an int32 buffer of shape (positions, layers, top-k) per generation. Rollout.routed_experts stitches the per-turn slices into one array and checks every row count, allowing for the one extra row SGLang appends after an abort. Resuming an aborted MoE generation trims the routing to the ids actually fed before continuing. Because tokin's ids are the sampled ids, row $i$ of the routing is the routing of token $i$. Under re-encoding it would be the routing of whatever token used to sit there.

The payload is a single safetensors file per session: per-turn tensors for prompt, generation, logprobs, routing and support, plus one JSON metadata tensor holding the tree. Rollouts, flat loss masks and flat logprobs are not stored at all; they are derived from the turns on read. The repository's AGENTS.md asks for that kind of subtraction (it opens its design section with 损之又损, "subtract, then subtract again"), and the code reads like it.

Training pi on MiMo

The demonstration run trains pi, a small open-source coding agent, on the code tasks of MiMo-V2.6-RL-oss with slime as the trainer. The environments are the ones covered in MiMo-V2.6 RL environments; 256 tasks are held out and 2,442 train, which sums to the 2,698 code tasks counted there. Each trial runs pi inside an E2B sandbox through Harbor, on its own tokin session, and is rewarded by the task's verifier. The algorithm is REINFORCE with truncated importance sampling, score centering and dynamic sampling (from DAPO), with the group-mean advantage of DeepSeekMath, KL weight 0, learning rate 1e-6, from the two launch scripts.

The epoch counts check against the batch sizes: 300 × 8 is 2,400 task groups against 2,442 tasks, and 140 × 64 is 8,960, or 3.67 passes. One detail of the 27B recipe I found in the template rather than the blog: Qwen3.8's chat template only writes a reasoning instruction into the system prompt for xhigh and low. medium adds no text at all, so the 27B run trains on prompts with no effort instruction in them.

expand

The blog's Table 2 reports pass@1 for the base model and the last checkpoint, evaluated with pi at 4 samples per task and a 32,768-token per-turn limit:

The rounded versions in the announcement are 0.50 → 0.80 and 0.59 → 0.66 for the 27B, 0.24 → 0.37 and 0.37 → 0.42 for the 35B.

How much of that is signal? I did the arithmetic the blog leaves to the figure. Held-out MiMo is 256 × 4 = 1,024 trials per arm. Treating trials as independent (generous, since four samples of one task are correlated), the 95% half-width on a difference of two pass rates near those values is about ±3.9 points. Both held-out gains clear that by a wide margin. Terminal-Bench 2.1 is 89 × 4 = 356 trials per arm, and the same calculation gives about ±7.1 points. The 27B's +7.0 lands right on the edge and the 35B's +5.1 sits inside it; per-task clustering would widen both intervals. On transfer beyond the training distribution I would call it probably positive, not demonstrated.

The bigger gap is attribution. These are gains from RL with a sound recipe, run through tokin. The release has no run of the same recipe through a re-rendering gateway, so nothing here measures what token fidelity is worth during training. The posts it cites argue that drift hurts (Agent Lightning's on the vLLM blog, Hugging Face's and LMSYS's on TITO, and Yao et al. on rollout-training mismatch), and the mechanism above is real. The size of the effect for this agent on these tasks is still an open number.

Two other harnesses from the same week

tokin is infrastructure for training inside a harness. Two releases from the same few days are harnesses, and they make clear how much is in that word.

PINE: a daemon that wakes agents on events

PINE, from the pmndrs collective, is a terminal agent built on pi's SDK (@earendil-works/pi-coding-agent), the same pi tokin trained. What it adds is the trigger. Most harnesses wake when a person types. PINE runs a daemon that keeps one SQLite event log, and agents subscribe to event sources: a folder (new or changed files, once they stop growing), MQTT topics, webhooks, FTP uploads, a Telegram bot, an HTTP address polled for changes, or a shell command run again and again. Schedules wake agents on cron or once. Agents configure all of this themselves through tool calls such as add_source and subscribe, and a built-in BM25 manual lets them look up PINE's own docs instead of carrying them in context. Commands run inside a bubblewrap sandbox on Linux.

One phrase in its materials reads like a tokin claim and is not one. When PINE says a model "never sees" tokens, it means credentials. Secrets live in .env, are kept out of child processes' environments and are replaced with [redacted] in events and logs. A bot token never passes through the model; a sampled token id was never the subject.

Two notes from its stack.md connect back to training. pi's request is plain chat completions (model, messages, stream, stream_options, store, max_completion_tokens, tools, plus reasoning_effort), which is exactly the API tokin v0.1.0 serves. And after an aborted run, pi's next request contains two consecutive user messages. tokin does not record a failed turn, so a request like that attaches to the last good turn and adds both messages to $\Delta$. A harness quirk that would confuse a prefix matcher is just more observation to tokin. The repository is MIT, the package is on npm as @pmndrs/pine, and its own "open points" list admits that no request has yet gone through a hosted provider from PINE.

Qwen Code 1.0 Preview

Qwen Code started as a fork of Gemini CLI and has been an independent agent since v0.1. The "1.0 Preview" lives on the v1.0.0-preview tag, not on main, and it is a different tree: four packages (cli, core, ink-shim, resource), where main at 0.27.2 has more than twenty. Its README says it "brings together the agent harness capabilities of Qoder and Qwen Code", Qoder being Alibaba's other coding agent. The preview's changelog is 14 lines. What it lists: sign-in with Qwen or Qoder accounts or any compatible endpoint, a native install on macOS, Linux and Windows without Node.js (the core package carries a bootstrap for running as a Bun-compiled binary), saved sessions you can resume, and non-interactive runs for scripts. Preview.1 adds parallel tool calls for Qoder models. The team is not accepting pull requests until after December 2026.

The "embeddable agent runtime" framing comes from qwen.ai's launch page, which renders in JavaScript and which I could not read. What the repository shows is a framework-agnostic @qwen-code/core with public sessions and plugin-management entry points. On main, the embedding story already exists in shipped form: TypeScript, Python and Java SDKs, qwen -p for headless runs, and qwen serve, a daemon with an HTTP and SSE API. The model this harness was built around is the subject of Qwen3.8-Max, where Qwen Code is one of the agents Qwen pointed at its API.

Qwen Code speaks several provider protocols, and that is where tokin's roadmap meets it. tokin v0.1.0 serves only OpenAI chat completions. The blog names the Responses API (Codex) and Anthropic's Messages API (Claude Code) as next, and the Responses translation landed on main on 12 November in PR #184, a day after the release. Each new API only translates requests and replies; the tree, the rendering and the rollouts stay the same.

Where tokin stops

The limits are in the code, and they are fair ones for a v0.1.0 beta:

• Text only. Image content parts are refused (only text content is supported).
• One engine. Generation goes through SGLang's native /generate. Top-p support needs slime's sampler patch.
• Four template families. Qwen2.5 through Qwen3.8 and GLM-4.5 through GLM-5.3. Everything else is a new ChatTemplate subclass and a run of the tokenizer tests.
• A cooperative harness. Replies must come back with their text, reasoning and call ids as served. A harness that strips reasoning still runs, one root per turn.
• One reward per session. Every leaf gets the trial's reward. For a compaction that split a long session in two, the early half is credited with the outcome of the whole.

Trust the mechanism, not yet the attribution

The idea is simple enough to state in one line (never encode what the model sampled), and the hard part is everything that line implies once harnesses retry, edit, compact and send two user messages in a row. tokin handles that with one rule, the deepest-continued-turn match, and a stub-subtraction trick for rendering increments, and both are short enough to audit in an afternoon. I trust the mechanism. I ran the real template and the drift is not hypothetical: on Qwen3.5 a single follow-up user message deletes the reasoning the model was trained to produce.

I would not cite the pi results as evidence that TITO improves training. They show the gateway can carry a 300-step asynchronous MoE-capable run with R3 and partial rollouts end to end, which is what a v0.1.0 needed to show. The ablation that would settle the question, the same recipe through a re-rendering gateway, is cheap compared with what has already been run. If you train agents in their own harnesses and you are on Qwen or GLM with SGLang, put tokin in front of the engine now. Then run that ablation yourself, and publish it.

Sources: tokin at tag v0.1.0 (504573c); the drift numbers and segmentation counts come from Qwen/Qwen3.5-35B-A3B's tokenizer.json and chat_template.jinja, run with the Hugging Face tokenizers library and a sandboxed Jinja environment; training numbers are the blog's Tables 1 and 2; PINE at 3d02c8e; Qwen Code at v1.0.0-preview (bd3604f) and main (6cff52a).