SteerDistill: quantizing an agent by teaching it its own fixes

A 4-bit copy of an agent model is cheap to run and often fine on a single question. Give it a twelve-turn task with tools, and it starts doing small wrong things: reading a file from the wrong directory, booking a train before asking where the user wants to go, writing a CSS file's contents into index.html. Each mistake is one turn. The task fails anyway.

SteerDistill, from Tsinghua's Institute for AI Industry Research, is a training method aimed at exactly those turns. During training, a frozen full-precision copy of the model finds the turn where the quantized student went wrong, writes it a short hint, and makes the student fix the turn itself, checking each attempt by executing it. Then it trains the hint away, so the deployed student acts without any teacher. The headline: with INT4 weights on three models and five agent benchmarks, the average score stays within one point of BF16. The code is released at air-embodied-brain/SteerDistill under MIT, and I read it at commit 937744a to see what "steering" and "local verification" actually are.

What W4A16 does to the weights

The paper's main setting is W4A16: 4-bit integer weights, 16-bit (BF16) activations. Only the weight matrices of the linear layers are compressed. Activations, attention, norms and the KV cache stay in 16 bits.

The weights are quantized in groups. Take 64 consecutive weights $w_1, \dots, w_{64}$ from one row of a matrix. Find their range, map it onto the 16 levels a 4-bit code can hold, and store one scale $s$ and one zero point $z$ for the group:

and at inference each weight comes back as $\hat w_i = (q_i - z)\,s$. That is the asymmetric quantizer in src/steerdistill/quantization/quantizer.py, with group size 64, FP16 scales and a clipping step borrowed from BitDistiller that shrinks the range before rounding so one outlier doesn't waste all 16 levels. Counting an FP16 scale and a 4-bit zero point per group (the exporter packs qzeros at 4 bits), that is $4 + 20/64 \approx 4.31$ bits per weight, against 16 for BF16: about 3.7 times less memory traffic per decoded token on the quantized layers, which is where a memory-bound decode spends its time.

GPTQ and AWQ are two smarter ways to choose the codes. GPTQ rounds one column at a time and pushes each rounding error onto the columns not yet quantized, using second-order statistics from calibration data (I derived it here). AWQ rescales the channels that see large activations before rounding, so the weights that matter most lose least. Both are post-training: no gradient step, a few minutes of calibration. In this paper GPTQ is the post-training baseline. AWQ is not a baseline at all; the repo uses AWQ's packing format to export the student to vLLM's fast 4-bit kernels, and the converter's docstring says plainly that it "does not run AWQ calibration". So if you came here asking how much plain AWQ loses on these agents, this paper does not answer it.

What GPTQ loses is in Table 1, and it is large. Averaged over the five benchmarks, GPTQ at W4A16 sits 7.18 points under BF16 on Qwen3-8B, 7.46 on Qwen3.5-4B and 7.47 on Gemma 4 12B-it (my subtraction from the paper's rows). The largest drops on Qwen3-8B are WebShop and ALFWorld, at 9.34 and 8.60 points.

Why one bad turn sinks the task

The paper never compares against single-turn benchmarks, so what follows is my reasoning, not its result. Treat each turn of an agent's task as a step that goes right with probability $p$. If the task needs $k$ turns and any wrong one sinks it, the task succeeds with probability

A single-turn benchmark measures $p$. An agent benchmark measures $p^k$. The same small drop in $p$ is multiplied $k$ times:

Going from 0.99 to 0.97 per turn is a two-point loss on a single-turn eval and a 27-point loss at twenty turns. Real tasks are kinder than this (some wrong turns are recoverable, which the paper's own case studies show) and crueller (errors are correlated: a model that misreads the task at turn two misreads it again at turn six). The shape still holds. Weight quantization nudges a lot of logits by a little, a long trajectory gives that nudge many chances to pick the wrong tool or argument, and the first wrong pick changes the environment everything after it has to live with.

The paper's word for that first wrong pick is a behavioral divergence: quantized agents "overlook task requirements or environmental conditions" and produce an environment transition the full-precision model would not have produced. Its case studies are mundane in a way I find convincing. On the train-booking task in Figure 5, BF16 asks for the missing cities, while On-Policy QAD invents Boston to New York and GPTQ invents San Francisco to Los Angeles.

expand

Drag the toy to see both halves of the argument. The left number is what a deployed agent lives with. The repair modes are what the training loop below can do with a failing rollout, using the paper's own repair rates.

Why ordinary recovery misses these turns

The standard fixes for quantization loss all supply training signal, and the paper's point is that none of it lands on the divergent turn. Quantization-aware training (EfficientQAT) and offline distillation (BitDistiller) learn from data collected without the student, so they rarely see the context the student actually gets itself into. On-policy distillation (NVIDIA's NeMo-RL QAD workflow) and RL with verifiable rewards (QeRL) do run the student, but in the contexts where it fails it almost never produces a correct action to learn from. The paper measures that directly: on BFCL V4 multi-turn, four unguided retries of the student from the same pre-error state produce a correct action at only 9.6% of divergent turns.

A teacher that writes a hint, and checks it

Write $\pi_T$ for the frozen BF16 teacher and $\pi_\theta$ for the quantized student, both from the same checkpoint (here a task-specific GRPO checkpoint the authors train first). Training runs in cycles. In cycle $k$ a frozen snapshot $\pi_{\theta_k}$ of the student generates rollouts, and every failed one goes through revision.

Find a reference. The teacher runs the same task from the same initial state. The repo tries up to four teacher rollouts (--counterfactual-attempts, default 4) and takes the first success as a fixed reference. No success, no revision: the failed rollout is kept as is.

Find the first divergence by state, not by action. Student and reference are aligned turn by turn on the environment state each action produces. In the tool-use tasks that state is a SQLite database; in ALFWorld it is world facts and admissible actions; in WebShop the product selection, page type and clickable choices. Two different actions that produce the same state change count as the same. The first student turn whose transition has no match, $t^\star$, is where the intervention happens. This matters more than it sounds: the ablation in Table 2 shows that intervening at the first action mismatch instead scores 1.83 points lower on BFCL V4, because many mismatched actions were fine and "fixing" them puts unnecessary edits into the training data. Intervening at a random eligible turn scores 43.52, below plain On-Policy QAD's 44.43.

Write a steering prompt. "Steering" in this paper is text in the student's context. No logits or activations are touched. The teacher reads the failed turn and the reference and writes a prompt $G$ with four fields. In src/steerdistill/analysis.py the schema is fixed:

STEER_FIELDS = {
    "precondition", "decision_rule", "expected_transition", "recovery_check"
}

and the analyzer's system prompt tells the teacher: "Do not copy the full answer or the exact expected tool call." The student has to produce the action. Figure 11 shows a real one: the student tried cat("project_analysis.txt") from /alex, the reading tool only sees the current directory, and the hint says to navigate to workspace first.

expand

Retry from the restored state. The history and environment before $t^\star$ are restored, the frozen student samples $y_{t^\star} \sim \pi_{\theta_k}(\cdot \mid h_{t^\star}, G)$, and the action executes.

Verify locally. The new turn is accepted only if executing it moves the environment the way the reference's turn did. In the tool-use benchmarks this is compare_transition in src/steerdistill/transition_verifier.py, which snapshots the database before and after the turn on both trajectories and compares them table by table (rows as a multiset, with columns like generated timestamps projected out by a per-task policy):

if error_observation(actual):
    return False, "environment_error"
# Only generated clock metadata proved unused by this task's verifier and
# environment logic is projected out. All other state and side effects stay.
if before != expected_before:
    return False, "precondition_state_mismatch"
if after != expected_after:
    return False, "postcondition_state_mismatch"
if expected_before != expected_after:
    return True, "verified_state_transition"
if not actual or not expected:
    return False, "read_evidence_missing"
if actual != expected:
    return False, "read_result_mismatch"
return True, "verified_read_transition"

A turn that changes state must change it identically. A read-only turn, where the database does not move, must return the same observation the reference got. Any error message in the tool output fails it outright. For ALFWorld, WebShop and Search QA the paper's Appendix A loosens this: a candidate passes if its resulting state matches the reference or both executions finish the task, which lets two different winning paths both count.

Refine and retry. On a failed check, the teacher gets the attempt, the student's tool calls and the verifier's reason, and rewrites $G$. The student tries again from the same restored state. The budget is $R = 4$ attempts per intervention. On a pass, the corrected turn replaces the old one, the old suffix is thrown away, and the student continues without the prompt. If the task still fails, the loop realigns and fixes the next divergence. With teacher steering, turn-level Pass@4 on that BFCL diagnostic goes from 9.6% to 82.3%.

Keep verified turns even when the task fails. This is a choice the ablation backs. Removing local verification altogether drops BFCL V4 from 46.84 to 42.18; keeping only corrections from trajectories that also succeed end to end gives 46.30. A verified fix of turn four is a good example of turn four whatever happens at turn nine.

expand

Distilling the hint away

The revised trajectories are training data, but the corrected turns were produced with $G$ in context, and the deployed student will never see $G$. So the loss gives the learner the same context minus the prompt, and asks it to behave as if the prompt were there. That is context distillation; I wrote about a close cousin, where the reference answer sits only in the teacher's context.

Let $c$ be the history plus the response prefix with every steering prompt removed. At a corrected turn the target is the frozen student snapshot with the hint; everywhere else it is the BF16 teacher:

The per-token loss is $\ell_{\text{FKL}}$ on turns in the corrected set and $\ell_{\text{RKL}}$ on the rest, averaged over every assistant token, KL over the full vocabulary. I like two choices here. The corrected-turn target is the student's own distribution under the hint, not the teacher's, so it is something a 4-bit model has already shown it can produce. And the two directions suit their jobs. Forward KL is mode-covering: the learner must put mass wherever the hinted student does, which is what you want when copying a specific corrected action. Reverse KL is mode-seeking on the learner's own samples, the usual on-policy distillation choice, which keeps the rest of the trajectory close to the teacher without forcing the small model to spread over everything the teacher might say. The ablation agrees: this routing beats reverse KL everywhere by 1.55 points, forward KL everywhere by 2.26, and swapping the two directions costs 2.56.

In src/steerdistill/training/mytrainer.py the routing is a mode id per row (the repo's agent guide explains that a skill is the code's older name for a stored steering prompt):

mode_specs = (
    (0, "ordinary_reverse", True),
    (1, "skill_forward", False),
    (2, "rescue_forward", False),
    (3, "steer_forward", False),
)

and the KL itself is two calls to F.kl_div with the arguments swapped:

if reverse:
    total = total + F.kl_div(reference_lp, student_lp, reduction="sum", log_target=True)
else:
    total = total + F.kl_div(student_lp, reference_lp, reduction="sum", log_target=True)

PyTorch's kl_div(input, target) computes $D_{\text{KL}}(\text{target} \,\|\, \text{input})$, so the reverse branch is $D_{\text{KL}}(\pi_\theta \,\|\, \pi_{\text{ref}})$ and the forward branch is $D_{\text{KL}}(\pi_{\text{ref}} \,\|\, \pi_\theta)$, matching the paper's equations. The learner itself keeps BF16 master weights, runs fake-quantized in the forward pass, and gets gradients through a straight-through estimator, the same arrangement as any quantization-aware distillation. Modes 1 and 3 both take the steered frozen-student target; mode 2 is a teacher-rescue ablation path, and the agent guide warns not to enable it just because it exists.

The rest of Table 2 tests the steering itself. From the same restored state, teacher steering beats unguided retries by 2.38 points and a direct teacher takeover (the teacher writes the turn, the student continues) by 1.78. The structured four-field prompt beats a free-form one by 2.06. And the hint's author matters: the same structured prompt written by the student itself scores 5.10 points lower, and written by GPT-5.6-Sol 1.57 lower. The paper gives no reason for the takeover result. My guess: a teacher-written turn is off the student's distribution, and every turn after it is the student's again. This is close to the hint-then-strip loop in KAT-Coder's data flywheel, but with the hint removed by the loss rather than by a hint-free replay that throws data away.

Checking the one-point claim

expand

Here is the whole of Table 1. Switch settings, and flip between raw scores and the gap to BF16.

I recomputed every Avg column from its five cells: all 33 rows match the paper to the hundredth. For scale, the five-benchmark average for Qwen3-8B at W4A16 is 53.54 for BF16, 46.36 for GPTQ, 51.00 for QeRL and 53.18 for SteerDistill. Then the claims:

• "Within one point of BF16" holds on the average, at W4A16. SteerDistill's average is 0.36 points under BF16 on Qwen3-8B, 0.73 on Qwen3.5-4B and 0.82 on Gemma 4 12B-it. Against GPTQ's 7.18, 7.46 and 7.47 that recovers 95%, 90% and 89% of what post-training quantization lost.
• Per benchmark, two of fifteen cells miss the line. On Gemma 4 12B-it, WebShop is 1.03 under and ALFWorld 1.01. The other thirteen are within a point; the worst on Qwen3-8B is τ²-Bench at 0.85.
• It is a W4A16 claim only. At W4A8 the gap is 1.92 points, and at W2A16 it is 10.57. The 2-bit result is still the best row by 3.39 points over On-Policy QAD, and GPTQ at 2 bits collapses to 18.86 from 53.54. BitDistiller beats it on Search QA at 2 bits, 34.22 to 34.05, which the paper reports itself.
• "1.67 to 2.18 points over the strongest baseline" checks out. The strongest baseline at W4A16 is QeRL for all three models, and the margins are 2.18, 1.71 and 1.67.

The video's bar chart below is that table, with its own framing.

Some things the table cannot tell you:

• No error bars. Table 1 is one number per cell. WebShop, ALFWorld and Search QA are sampled at temperature 0.7, BFCL at 0.001 and τ²-Bench at 0, and the only variance the paper shows is Figure 3's shading over four τ²-Bench evaluation runs. I'd bet margins of 1.7 to 2.2 points on a five-benchmark average are real, since every row orders the same way, but without the spread that is a bet.
• The W4A8 rows are coarser. All 30 per-benchmark W4A8 cells end in 0 or 5 in the second decimal (35.90, 27.85, 75.55, …), while the W4A16 and W2A16 cells show no such pattern. I don't know why, and the paper doesn't say.
• The baselines were moved. The reference configurations for NeMo-RL's QAD and for QeRL use NVFP4 on Hopper or Blackwell. The authors adapted both to group-wise INT4 on A100s. That is the fair thing to do for a like-for-like comparison, and also the place a baseline is most likely to lose a point or two to tuning.
• The GRPO start does a lot of work. Every main result starts from a task-specific GRPO checkpoint. Appendix C.1 repeats Qwen3-8B without it: SteerDistill still wins, but over QeRL by 0.82 on BFCL V4 (39.67 against 38.85) and 0.13 on τ²-Bench (29.05 against 28.92). It is still within a point of BF16 there (40.40 and 30.00 on BFCL and τ²-Bench), and QeRL sits 1.08 under on τ²-Bench.

What it costs to train, and what it saves at inference

expand

Training. On τ²-Bench, SteerDistill's checkpoint at 32 GPU hours scores 31.68% Pass@1; QeRL reaches 29.84% at 96 GPU hours. That is the "67% fewer GPU hours" in the abstract (32 is a third of 96). All runs use eight A100 80GB GPUs, so 32 GPU hours is four hours of wall clock on one node. The paper says the count includes teacher steering and verification. It does not include the GRPO run that produced the starting checkpoint, which every method shares. One small inconsistency: the figure labels On-Policy QAD's plateau 28.75%, while the text's "2.91 percentage points" gap is computed from Table 1's 28.77.

Inference. On the same benchmark, successful tasks average 4.0K output tokens for SteerDistill against 3.9K for BF16 and 4.1K to 5.2K for the quantized baselines (Table 7), with 11.6 interaction steps against BF16's 11.4 and GPTQ's 13.8, and 192K input tokens against GPTQ's 270K. These are averages over each method's own successful tasks, so the subsets differ, but the direction is the one the compounding argument predicts: a quantized agent that takes wrong turns and recovers spends more turns doing it, and every extra turn re-reads the whole context.

What the paper does not measure is latency or throughput. The W4A16 export uses vLLM's existing 4-bit kernels, so the speedup is whatever those kernels give you on your GPU. The W4A8 and W2A16 runs go through a reference backend that, in the repo's own words in docs/training/precision.md, implements "the numerical quantization schemes, not fused INT2 or W4A8 GEMM acceleration". Those two rows are accuracy results, not speed results.

What it needs from your environment

The appendix lists the requirements, and they are the real cost of the method:

1. A successful teacher reference for the failing task. If the BF16 model can't solve it in four tries, the student gets no correction there. SteerDistill can only pull the student up to the teacher, and only on tasks the teacher already solves.
2. Restorable state. Every retry restarts from the exact pre-error history and environment. The repo does this by snapshotting a SQLite database at every turn, with a checksum in a manifest that the verifier re-checks before trusting it.
3. A verifier that can say a single turn was right. Here that is state equality against the reference. The limitations section is candid that local verification "does not guarantee eventual task success" and that environments without state restoration or reliable local verification are untested.

For an agent whose tools write to a database you own, all three are buildable: snapshot the store per turn, as the repo does, and diff it. For an agent whose tools send emails, call third-party APIs or move money, you cannot restore the world and retry turn four, and you would need a full simulator first. The paper's tool-use training tasks come from AgentWorldModel, whose executable environments make that possible.

There is also nothing to download yet. The README's checkpoints entry is still a placeholder with no links, and the reference setup is eight A100 80GB GPUs per run, with separate environments for training, vLLM inference and each benchmark.

Worth building where the environment can be snapshotted

I think the core move is right and will outlive this paper: when you compress an agent, spend the recovery budget on the turns where the compressed model's environment state first departs from the full model's, and make the compressed model produce the fix itself. Everything in the ablations points the same way. Locate by state, not by action. Hint, don't take over. Verify the turn by executing it, and keep the verified turn even when the task fails. The loss design that turns a hinted turn into an unhinted one is clean and cheap.

The one-point headline is true as stated, for the average, at W4A16, starting from a GRPO checkpoint. It bends at the edges: two benchmark cells over a point, nearly two points at W4A8, ten at W2A16, and a thin margin over QeRL without the GRPO start. If I were compressing an agent tomorrow, I would still run GPTQ or AWQ first, measure the multi-turn drop on my own tasks, and only build this machinery if that drop is several points and my environment can be snapshotted. Most agent environments I know can't be, yet. That, more than the method, is what decides whether you can use it.

Some sibling reading on this site: Deletion Tuning and closed-form weight edits, today's other piece on changing a model's weights cheaply, and LittleBit, where distillation carries a model down to a tenth of a bit.