Deletion Tuning and closed-form edits: changing an LLM without fine-tuning

Two posts about changing a language model's weights without ordinary fine-tuning went round on 10 and 11 November. The first was Deletion Tuning, a NeurIPS 2026 paper from NYU and the University of Calgary that post-trains a model by learning which pretrained weights to set to zero and nothing else. The second was a blog post from embedding.in, A decision skill written into the weights in 3 milliseconds, which writes a classifier's decisions into one matrix of Qwen3.5-0.8B with a single linear solve.

Both promise the same thing: a new behaviour for a tiny fraction of the weights and with less collateral damage than LoRA. They get there in opposite ways. Deletion Tuning runs hundreds of reinforcement-learning steps to choose a few million bits. The edit post runs no optimiser at all; it computes 262,144 new numbers in closed form. Below I work through what each changes in the matrix, then check the claims against the code in agentic-learning-ai-lab/deleting-weights, the 84 masks on Hugging Face, and the post's own sample sizes.

Three ways a weight matrix can move

Take one linear layer, $y = Wx$, with $W \in \mathbb{R}^{d_\text{out} \times d_\text{in}}$. Every post-training method replaces $W$ with $W + \Delta W$. They differ in what $\Delta W$ is allowed to be and how it is found.

Full fine-tuning lets every entry of $\Delta W$ be anything, and finds it by gradient descent on a loss. On Llama-3-8B that is 8,030,261,248 free numbers, and a checkpoint the paper sizes at 30,633 MB in fp32.

LoRA restricts the update to rank $r$:

It trains $r(d_\text{in} + d_\text{out})$ numbers per matrix instead of $d_\text{in} d_\text{out}$. Training is cheap and the adapter is small, which is why Program-as-Weights ships compiled behaviours as LoRA files. Merged into the model, though, $BA$ is dense: every entry of every adapted matrix moves, by a little. The paper's rank-1 LoRA on all seven projections of Llama-3-8B trains 2,621,440 numbers (by my arithmetic on the config shapes, which reproduces the paper's 10.0 MB in fp32) and, merged, changes 6,979,321,856 weights.

Closed-form editing, the ROME and MEMIT line, treats an MLP's output matrix as a key-value store. A column of the down projection is what one hidden neuron writes into the residual stream, the same write neuron steering scales by hand. Refusal-direction ablation, as in runtime abliteration, is a cousin: a projection that removes one direction from what layers write. If you know which hidden activations $k_i$ (keys) should produce which output changes $r_i$ (values), the best $\Delta W$ is a regularised least-squares problem with a closed-form answer. Stack the keys as rows of $K \in \mathbb{R}^{n \times d}$ and the values as rows of $R \in \mathbb{R}^{n \times d_\text{out}}$:

The first term asks the edit to produce the new outputs. The second penalises any change in what the layer writes on text you want left alone, because $\operatorname{tr}(\Delta W C \Delta W^\top)$ is the expected squared size of $\Delta W k$ over that text. Setting the gradient to zero gives the normal equations:

No steps, no learning rate. One $d \times d$ solve. ROME is the rank-one case with a single key; MEMIT spreads many keys over several layers. The cost is that you must know $k_i$ and $r_i$ in advance, which is easy for "Paris is the capital of France" and less obvious for a skill.

Deletion Tuning is a fourth shape. $\Delta W = -W \odot (1 - M)$ for a binary mask $M$: every changed entry goes to exactly zero, and the only freedom is which ones.

Deletion Tuning trains one logit per gate weight

The paper masks a single matrix per transformer block: the gate projection of the SwiGLU MLP. In Llama and Qwen2.5 the MLP is

so the gate decides, per hidden channel, how much of $W_\text{up} x$ gets through. Zeroing $W_\text{gate}[j, i]$ removes input feature $i$ from neuron $j$'s gate. The authors follow GLU pruning work in treating that gate as the place where the MLP's channels are switched on and off.

For each target matrix $W$ they keep a trainable logit matrix $G$ of the same shape and train the layer as

and deploy it as $\widehat{W} = W \odot \mathbf{1}[G > 0]$. The pretrained weights never change. The sharpness $\alpha$ ramps linearly from $10^3$ to $10^4$ under GRPO (from $10^2$ under SFT), so by the end of training the sigmoid is a step and the soft mask the gradients saw is the hard mask that ships; the paper says swapping one for the other moves accuracy by at most 0.08 points. In src/learnable_sparsity/differentiable_mask.py this is the whole forward:

# differentiable_mask.py, DifferentiableMask.forward
if self.training:
    self.current_sharpness = self.sharpness[0] + (self.sharpness[1] - self.sharpness[0]) * self.progress
    return torch.sigmoid(self.gate * self.current_sharpness).reshape(target_shape).to(dtype)
return (self.gate > 0).to(dtype).reshape(target_shape)

This is not a straight-through estimator. Gradients flow through a real, deterministic sigmoid; the trick is making it so steep that its forward value is already almost binary. At $\alpha = 10^3$ a logit of $0.03$ gives $\sigma(30)$, which is 1 to thirteen decimal places.

Initialisation does a lot of the work. The logits start from Wanda, the one-shot pruning score $s_{ij} = \lvert W_{ij}\rvert \cdot \lVert X_j \rVert_2$, computed on 2,048 training prompts. The lowest $\eta = 1/2048$ fraction of each matrix's weights by that score start with logits drawn from $\mathcal{N}(-0.03, 0.01^2)$; every other weight starts at $\mathcal{N}(+0.03, 0.01^2)$. In the code (wanda_calibration.py) this is a draw from $\mathcal{N}(0, 0.01^2)$ shifted three standard deviations up or down. The paper's Equation 8 works out the consequence: about 0.18% of gate weights are already deleted at step zero, and most of those are not the Wanda picks. They are the $\Phi(-3) \approx 0.135\%$ tail of the "keep" Gaussian that happens to land below zero. On Llama-3.2-1B's 268,435,456 gate weights that is about 131,072 Wanda deletions plus about 362,000 random ones.

The regulariser points the other way from the one you might expect. The training loss is the regime's own objective plus $-\lambda \lVert \widetilde{W} \rVert_2^2$, borrowed from MaskLLM, with $\lambda = 10^{-5}$ under GRPO and SFT and zero under self-distillation. The minus sign rewards keeping weight norm, so it nudges the mask to keep large weights. There is no sparsity target; the paper says it lets training decide how many weights to delete.

The deployed adapter is the hard mask, bit-packed with NumPy and compressed with zstd. Because almost every bit is "keep", it compresses to roughly 0.03 bits per masked weight. Applying it is weight.mul_(keep) in src/apply_mask.py, after which the model loads in Transformers or vLLM like the base.

expand

The project page animates the same figure. Watch the sigmoid: it starts as a smooth ramp, so a logit near zero gives a mask near one half, and steepens until it is a step at zero. A logit that ends on the wrong side of zero has deleted its weight.

How zeroing a weight can add a skill

Deleting weights sounds like it can only remove ability. It is more expressive than that, for a mechanical reason and an empirical one.

First, a zero is a large move. A typical gate weight's update under full fine-tuning is a small fraction of its value; setting it to zero is a change of exactly its whole value, in a known direction. Through SiLU and the elementwise product, removing input $i$ from neuron $j$'s gate can raise or lower that neuron's output depending on the sign of $W_{ji} x_i$, so a set of deletions can push the MLP's write in many directions, not only towards "less".

Second, the paper's own evidence suggests most of the skill is already there. Its Table 3 samples 128 answers per GSM8K problem at temperature 1.0: the base Llama-3-8B-Instruct already solves 95.29% within 128 tries, against 99.62% after Deletion Tuning. Greedy accuracy moves from 60.27% to 83.55%. Most of that is making an answer the model can reach into the one it reaches first. The authors do report problems the base fails in all 128 attempts and the masked model solves, and the pass@128 gap (4.3 points on Llama-3-8B, 6.1 on Qwen2.5-7B) is real, but the bulk of the gain is reliability. That is the regime where RL is known to touch small subnetworks, and where choosing a few bits has a fair chance.

Pruning is usually a way to remove capacity cheaply, as in the data-free expert prune of Kimi K3.1 Pro. Here the choice of what to prune is the learning signal, and the Wanda prior is not the skill. The paper's Table 4 applies Wanda pruning alone, with no training, at eight sparsity levels from 1/2 to 1/8192 and keeps the best: every backbone stays within 2.4 points of its base, while the trained masks gain up to 53.2 points (Llama-3.2-1B on GSM8K goes from 1.21% to 54.44%). Masks as a way to adapt a frozen network are older than this paper (Piggyback for vision, Zhao et al. for BERT); what is new is doing it as the entire post-training update of a decoder LLM, under RL, SFT and on-policy distillation, and comparing it head to head with full fine-tuning.

What the paper reports against full fine-tuning

Under GRPO, the paper's Table 1 and Table 2 report best-checkpoint validation accuracy:

At small scale the mask trails: 54.44 against 60.50 on the 1B model. From 7B up it ties or edges ahead. "Matches" is the right word and "beats" is not. Llama-3-8B's 83.55 against 83.24 is consistent with a 1,319-problem GSM8K test set (both percentages land on whole problems: 1,102 and 1,098), which makes the lead four problems. On Qwen2.5-7B the 0.43-point gap in the six-benchmark mean comes almost entirely from AIME24 and AMC23 in the paper's Table 8: the mask gets 7 of 30 AIME problems against full fine-tuning's 4, and 19 of 40 AMC problems against 22. The paper reports one number per cell (I found no seeds, standard deviations or intervals in it) and picks the best checkpoint on the validation set it reports. That is normal for the field, and it means ties within a point or so are ties.

The same pattern holds outside RL in the paper's Tables 5 and 6 (self-distillation is the on-policy recipe covered in on-policy self-training). Under SFT, Llama-3.2-3B averages 22.28% over 14 MMLU-Pro subjects (full fine-tuning 21.19%) and 11.39% on BBEH (the same as full fine-tuning). Under self-distillation on Qwen2.5-7B-Instruct, ToolAlpaca goes to 69.07% (full fine-tuning 68.04%) and Science to 70.61% (71.20%).

Forgetting is where the mask is clearly different. Figure 2 tracks two things during GSM8K GRPO on the three Llama models: forward KL from the base policy on the training task, which RL's Razor found predicts forgetting, and mean accuracy on six prior tasks (HellaSwag, MMLU, TruthfulQA, WinoGrande, IFEval, HumanEval). Full fine-tuning's KL reaches 1.0 on the 1B and 1.5 on the 8B, and its prior-task accuracy ends 1.3 to 3.1 points below base. LoRA and random sparse fine-tuning (MTH) end up to 5.4 points below. Deletion Tuning's KL stays at or below 0.05 and its prior-task accuracy within a point of base. The control matters: MTH at 99.95% sparsity trains 0.05% of all weights, about as many as the mask deletes, and still forgets. So it is not just "few weights changed". The authors' reading is that a zero is a bounded edit chosen by the task, where a float update at random positions is not.

expand

The paper also compares the shape of the updates (Figure 3). Its spectrum drift and principal-weight overlap sit next to full fine-tuning's, while LoRA r=1 drifts up to 84 times more on Llama-3-8B. On stable rank the picture is looser than the text suggests: the mask's 66 to 93 is closer to full fine-tuning than MTH's 410 to 650 or LoRA's 1 to 2, but full fine-tuning itself reads at roughly 13 to 25 on the plot, so the mask's update is several times higher-rank than the thing it is "closest to". Zeroing scattered entries produces a high-rank delta almost by construction.

expand

Counting the 84 published masks

The authors uploaded every mask: 8 GRPO, 2 self-distillation, and 74 SFT (one per MMLU-Pro subject and BBEH task for two Llama sizes), plus the README. Each folder's config.json records the masked-parameter count and the zero count, and I decompressed nine of the bitmaps to check those fields and look at where the zeros fall. The script reads the safetensors header by hand and runs no code from the repository. Dividing by each base model's total parameter count from its Hugging Face metadata:

The paper does not say any of what follows plainly.

The abstract's "zeroing fewer than 0.05% of its weights" holds for seven of the eight GRPO masks. The body of the paper scopes it correctly ("every GRPO run of Table 1", six models). The 30B mixture-of-experts mask deletes 0.075% of all weights, every one of the 74 SFT masks is above 0.05%, and the Science self-distillation mask is at 0.44%. The repost that spread the paper rounded it to "less than 0.1%", which is also wrong for those masks. All of them are still under half a percent, which is the claim I would have led with.

The published files do not all match the tables. Llama-3.2-3B, Llama-3-8B, Qwen2.5-0.5B, 3B and 14B agree with the paper's sizes to within rounding, and the two self-distillation masks are within half a megabyte. The Llama-3.2-1B mask deletes 0.0451% of the model and weighs 1.27 MiB, where the paper's Table 7 says 0.0347% and Table 1 says 1.0 MB; Qwen2.5-7B is 7.48 MiB against 6.9; the 30B is 31.68 against 29.6. The SFT masks are 3.68 to 4.51 MiB for Llama-3.2-1B on MMLU-Pro, where Table 5 lists 1.0 MB, the same figure as the GRPO mask. I do not know whether the uploads are different checkpoints or the tables were filled from the wrong run. The size ratios shrink with them: 4,714 MB of fp32 1B checkpoint over the measured 1.58 to 4.51 MiB SFT masks is roughly 1,000 to 3,000 to one, short of the 4,700 the tables imply.

Under GRPO the deletion count barely moves from where initialisation put it. The expected step-zero count on Llama-3.2-1B is about 493,000 (131,072 Wanda picks plus about 362,000 noise deletions); the published mask has 557,123. Every GRPO mask sits between 0.14% and 0.24% of gate weights, around the 0.18% the paper's Equation 8 predicts at initialisation. SFT and self-distillation, which start at a sharpness of $10^2$ and run at higher learning rates, end at 0.27% to 0.98% (SFT) and 0.78% to 1.77% (self-distillation) of gate weights, up to ten times the starting point. "Learns how many weights to delete" is true in the SFT runs. Under GRPO the number is mostly set by the initialisation recipe, and what training chooses is which bits flip.

The masks share a core across tasks. On Llama-3.2-1B, the GSM8K GRPO mask and the MMLU-Pro biology SFT mask share 118,097 deleted positions; chance would predict 4,836. The GSM8K mask shares 118,093 with the math SFT mask and 116,209 with the BBEH Dyck-languages mask. That shared set is close in size to the 131,072 weights the Wanda prior starts deleted, and since Wanda scores are dominated by $\lvert W \rvert$, the obvious reading is that most of it is the low-magnitude prior that no task bothered to restore. I cannot see the initial logits, so that is inference. The rest of each mask is scattered: per output row, the variance-to-mean ratio of the deletion count is 1.04 on the 1B GRPO mask and 1.08 on the 8B, which is what a uniform random scatter gives. Qwen2.5-7B is the exception, with 36 input columns holding 152,213 of its 3,161,923 deletions, consistent with Wanda's input-norm term singling out a few channels.

A decision skill solved into one matrix

The embedding.in post starts from an agent problem. Before acting on a request, a model has to make two calls: which tool it needs (search, the caller's records, a calculation, an action, or none), and whether it has enough information to go ahead. The authors labelled 393 requests with both answers. Asked plainly, Qwen3.5-0.8B picks the right tool 80% of the time and gets the "ready to act" call right 18% of the time, "because it says 'no' to almost everything".

expand

The teacher is a policy built with their library vegaml (Apache-2.0): a small head per question on a frozen reader model, fitted on 300 of the rows and calibrated so its probabilities mean what they say. On the 93 rows it never saw, the post reports 96.8% on tool choice and 97.8% on ready.

The edit then moves that behaviour into Qwen3.5-0.8B itself. From the post:

• Where. The output matrix (down projection) of the MLP in "layer 23 of 24", restricted to 256 of its 3,584 input columns: the hidden neurons most active at the answer position compared with how active they are elsewhere. That is the same selection idea as neuron tuning. The post does not say whether 23 counts from zero.
• Keys. $K_S$, what those 256 neurons output at the answer position for each training request.
• Values. $R$, how the final hidden state would have to change for the model's answer distribution to match the teacher's: the gap between the two distributions in log space, pulled back through the model's readout.
• Protection. $C_{SS}$, the second moment of the same 256 activations on text to leave alone: ordinary sentences, the rest of every prompt, and the position just after the answer (so the model does not learn to keep answering).
• Solve. $\Delta W_S = R^\top K_S \left(K_S^\top K_S + \lambda n\, C_{SS}\right)^{-1}$, exactly the MEMIT-form equation above, applied to 256 columns.
• Guards. Scale the edit by a dose of 0.5, 1, 1.5 or 2, keep the dose that best closes the gap on 40 held-back rows, and refuse any dose that moves the protected text by more than a KL of 0.03.

Qwen3.5-0.8B's config.json gives a 1,024-wide residual stream and 3,584 MLP neurons, so the edited block is $1{,}024 \times 256 = 262{,}144$ weights, which is the post's figure. That is 0.03% of the release's 873,438,784 parameters. The solve and the write took 3.1 ms; the whole run, including the forward passes to collect $K_S$, $R$ and $C_{SS}$ and the dose search, took 87 seconds on two T4s. LoRA at rank 16 on every MLP for 60 steps took 252 seconds. Because the last MLP's output lands one layer (or zero layers) from the readout, a linear pull-back of the desired logit change is a reasonable approximation; the post does not discuss how much the remaining layer and the final norm bend it.

The post's main table, on the 93 held-out rows:

And its ablations:

The ablation that matters is the second row. Drop $C_{SS}$ and the same solve still hits the targets, and WikiText perplexity goes to $1.2 \times 10^{11}$. The protection term is the method; without it a least-squares edit is a way to break a model precisely.

Reading the edit post's numbers at their sample sizes

Every number in the post is the authors' own, on their own data, and I found no published code for the edit itself (the vegaml repository is the teacher library; I searched it for the solve and found only its own ridge fits for reader ports). With that said, most of the post is careful, and it volunteers its own limits. The problem is what happened to it in the repost, where "18% to 97.8%" and "1/15 (LoRA 5/15)" became the headline.

18% is not the baseline that matters. The unedited model says "no" to almost everything and scores 18%, so roughly 82% of the 93 rows are probably "ready" cases (my inference; the post gives only the 18%). A model that always said "yes" would then score about 82%. The post's random edit lands at 81.7%, which the post puts down to the model's bias towards "no". The fair comparison for the ready question is roughly 82% to 97.8%, which is about 76 rows to 91. On tool choice, 80% to 93.5% is 74 rows to 87.

The ceiling is the teacher. The teacher scores 97.8% on ready and 96.8% on tool. The edited model matches it on ready and trails it on tool. The edit transfers a policy; it does not improve on one.

1 of 15 against about 5 of 15 is not yet a difference. I ran a two-sided Fisher exact test on those counts: p is about 0.17. The examples are vivid (LoRA answering "suggest a cat name" with "no"; the edit listing the primary colours as "red, no, no, no"), and the direction is plausible given that LoRA changed a thousand times more weights. But fifteen prompts cannot carry the claim.

Off-format, nothing transfers. Reworded, the same questions get 5 of 11 right from the edited model, 8 of 11 from the unedited one and 9 of 11 from LoRA. Five against eight is no more significant than the damage count (the same test gives p of about 0.39), so I would not claim the edit made things worse. What the 11 prompts do show is no sign that the skill survives a new wording, and the post says so and proposes solving over several formats at once. A decision skill that only fires on one template is a lookup keyed on that template.

"263 million" is LoRA's merged footprint. A merged rank-16 delta on all three MLP matrices of all 24 layers touches $24 \times 3 \times 1{,}024 \times 3{,}584 = 264{,}241{,}152$ weights by my count; the post's 263 million is close but not equal, perhaps from counting entries whose fp16 value actually changes. By the same arithmetic on the shapes, LoRA trained 5,308,416 numbers to get there. The edit fitted 262,144. "A thousand times fewer weights changed" compares merged footprints, and on that metric it is right.

The perplexity costs are smaller than LoRA's: WikiText-2 rises 10% for the edit (28.7 to 31.7) and 15% for LoRA (28.7 to 33.1). ARC-Easy on 200 questions barely moves for the edit; the random change of the same size drops it from 70.5% to 52.5%, which is a useful reminder that a quarter of a million weights in the last MLP is not a small edit. LoRA's ARC gain to 78.5% the authors attribute to it learning to answer in a single word, which seems right.

"The same idea MIT explored"

The repost of the edit post says the result is "same idea MIT explored with a frontier model". The blog itself does not mention MIT. The closest match is Sparks of Weight-Space Programming from MIT CSAIL a week earlier, where GPT-6 Astra edited Qwen3.5-0.8B, the same base model, with no gradients. The repost gives no link, so that identification is my inference.

They share a base model and the slogan "no training, no gradient descent". The mechanisms differ. Astra spent 20 hours of forward passes searching for a failure mode and found a rank-one fix: subtract a "loop" direction from 64 columns of two down projections. The embedding.in edit is a supervised least-squares solve from labelled rows to a target distribution, done once. The second is much closer to MEMIT than to the MIT work.

Same toy, three edits

The widget below is one SwiGLU unit with two inputs and six hidden neurons, small enough to draw everything. Its "old task" is its own sign over a 21 by 21 grid, so the base keeps all 441 cells by definition. The "new task" is seven labelled points, three of which the base gets wrong. You can edit it three ways:

• Delete gate weights, as Deletion Tuning does. With 12 gate weights there are only 4,096 possible masks, so the best masks come from searching all of them instead of a sigmoid relaxation.
• LoRA rank 1 on the gate, trained by gradient descent on the seven points only.
• A closed-form write to the down projection: the MEMIT-form solve above with $K$ the six hidden activations at the new points, $C$ the second moment of those activations over the grid, and a slider on $\lambda$.

When I ran it, deleting one gate weight fixed all seven points and kept 423 of 441 cells. LoRA got all seven by step 100 with 369 cells kept, then kept training and drifted to 343 by step 800, because nothing in its loss mentions the old task. The closed-form write gets all seven at every $\lambda$; raising $\lambda$ from 0 to 1 takes the kept cells from 324 to 386. Do not read the toy as a ranking. I chose this unit by a seeded search for one where a single deletion works. What it shows is mechanics: a deletion is a large, discrete move; LoRA's damage grows with steps unless something anchors it; the closed-form write trades fit for protection through one explicit knob.

Weights touched is the wrong headline

Both posts sell themselves on how few weights change. That number is real, but it hides where the cost went.

Deletion Tuning changes 3,621,710 weights on Llama-3-8B and ships 8.4 MB. To find them it trains a logit for every one of the 1,879,048,192 gate weights, which the paper concedes in its limitations: less training memory than full fine-tuning, more than LoRA. It also needs the full RL or SFT pipeline: rollouts, a verifier or labels, and H200s in the authors' setup, several of them past 3B. The closed-form edit changes 262,144 weights with a 3 ms solve, but it needs a teacher that already makes the decision, labelled rows, protected text, and a target you can express as a change in the last hidden state. LoRA changes everything a little and trains a few million numbers, and it is the only one of the three that generalised to reworded prompts in the edit post's test.

The count of changed weights predicts forgetting less well than either post implies. In the paper, MTH changes about as many weights as Deletion Tuning and forgets like LoRA. In the edit post, a random change the size of the edit breaks ARC-Easy. What seems to matter is that the change is chosen by the task, bounded in size, and anchored to what should stay the same: Deletion Tuning bounds every edit at "set to zero" and stays near the base in KL; the closed-form edit anchors through $C_{SS}$ and a KL cap. Few weights is a side effect of those choices.

Where I land

Deletion Tuning is the more important of the two. It is a careful paper with code, with masks for every reported run, and with a forgetting result that holds against a matched-size control. "Post-training by selection" is a useful way to think about RL in particular, where the paper's own pass@k shows the base model could already reach most of the answers. I would fix the abstract, whose "fewer than 0.05%" is a GRPO number and not an across-regimes one, and I would want the authors to reconcile the uploaded masks with the size tables where they disagree. If I needed several task adapters on one base model with minimal interference, I would try it before LoRA, starting from the published masks and src/apply_mask.py.

The decision-skill edit is a well-written single experiment. The mechanism is MEMIT's, aimed at a calibrated teacher instead of a fact, and the post is honest about its narrowness. On 93 test rows against an 82% constant-answer baseline, with a 15-prompt damage check and an 11-prompt format test where it scored below the unedited model, it is a promising demo, not evidence that closed-form edits beat LoRA. I would not deploy a skill that breaks when the prompt is reworded; I would want the multi-format solve the authors propose, and their code, first.

Files behind this page

I read the paper, the repository's masking, Wanda and apply code, the post, and vegaml. I ran a standalone parser over the published masks (nine bitmaps decompressed, all 84 config.json files read), the Fisher tests and the toy. I did not train anything, evaluate a masked model, or reproduce the edit.