# Deletion Tuning and closed-form edits: changing an LLM without fine-tuning
> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.satyajit.si/articles/weight-edits-without-finetuning
> date: 2026-11-12
> tags: fine-tuning, lora, pruning, post-training, grpo, interpretability
Two posts about changing a language model's weights without ordinary fine-tuning went round on 10 and 11 November.
The first was [Deletion Tuning](https://arxiv.org/abs/2611.10866), a NeurIPS 2026 paper from NYU and the University
of Calgary that post-trains a model by learning which pretrained weights to set to zero and nothing else. The second
was a blog post from embedding.in, [A decision skill written into the weights in 3
milliseconds](https://embedding.in/blog/a-decision-skill-written-into-the-weights), which writes a classifier's
decisions into one matrix of Qwen3.5-0.8B with a single linear solve.
Both promise the same thing: a new behaviour for a tiny fraction of the weights and with less collateral damage than
LoRA. They get there in opposite ways. Deletion Tuning runs hundreds of reinforcement-learning steps to choose a few
million bits. The edit post runs no optimiser at all; it computes 262,144 new numbers in closed form. Below I work through
what each changes in the matrix, then check the claims against the code in
[agentic-learning-ai-lab/deleting-weights](https://github.com/agentic-learning-ai-lab/deleting-weights), the 84 masks
on [Hugging Face](https://huggingface.co/Jacklu0831/deletion-tuning-masks), and the post's own sample sizes.
## Three ways a weight matrix can move
Take one linear layer, $y = Wx$, with $W \in \mathbb{R}^{d_\text{out} \times d_\text{in}}$. Every post-training method
replaces $W$ with $W + \Delta W$. They differ in what $\Delta W$ is allowed to be and how it is found.
**Full fine-tuning** lets every entry of $\Delta W$ be anything, and finds it by gradient descent on a loss. On
Llama-3-8B that is 8,030,261,248 free numbers, and a checkpoint the paper sizes at 30,633 MB in fp32.
**[LoRA](https://arxiv.org/abs/2106.09685)** restricts the update to rank $r$:
$$
\Delta W = BA, \qquad B \in \mathbb{R}^{d_\text{out} \times r},\; A \in \mathbb{R}^{r \times d_\text{in}}
$$
It trains $r(d_\text{in} + d_\text{out})$ numbers per matrix instead of $d_\text{in} d_\text{out}$. Training is cheap
and the adapter is small, which is why [Program-as-Weights](/articles/program-as-weights) ships compiled behaviours
as LoRA files. Merged into the model, though, $BA$ is dense: every entry of every adapted matrix moves, by a
little. The paper's rank-1 LoRA on all seven projections of Llama-3-8B trains 2,621,440 numbers (by my arithmetic on
the config shapes, which reproduces the paper's 10.0 MB in fp32) and, merged, changes 6,979,321,856 weights.
**Closed-form editing**, the [ROME](https://arxiv.org/abs/2202.05262) and [MEMIT](https://arxiv.org/abs/2210.07229)
line, treats an MLP's output matrix as a key-value store. A column of the down projection is what one hidden neuron
writes into the residual stream, the same write [neuron steering](/articles/neuron-steering) scales by hand.
Refusal-direction ablation, as in [runtime abliteration](/articles/runtime-abliteration), is a cousin: a projection
that removes one direction from what layers write. If you know which hidden activations $k_i$ (keys) should produce which output changes
$r_i$ (values), the best $\Delta W$ is a regularised least-squares problem with a closed-form answer. Stack the keys as
rows of $K \in \mathbb{R}^{n \times d}$ and the values as rows of $R \in \mathbb{R}^{n \times d_\text{out}}$:
$$
\min_{\Delta W}\; \sum_{i=1}^{n} \lVert \Delta W k_i - r_i \rVert^2 \;+\; \lambda n \operatorname{tr}\!\left(\Delta W\, C\, \Delta W^\top\right),
\qquad C = \mathbb{E}_{k \sim \text{protected text}}\!\left[k k^\top\right]
$$
The first term asks the edit to produce the new outputs. The second penalises any change in what the layer writes on
text you want left alone, because $\operatorname{tr}(\Delta W C \Delta W^\top)$ is the expected squared size of
$\Delta W k$ over that text. Setting the gradient to zero gives the normal equations:
$$
\Delta W = R^\top K \left(K^\top K + \lambda n\, C\right)^{-1}
$$
No steps, no learning rate. One $d \times d$ solve. ROME is the rank-one case with a single key; MEMIT spreads many
keys over several layers. The cost is that you must know $k_i$ and $r_i$ in advance, which is easy for "Paris is the
capital of France" and less obvious for a skill.
Deletion Tuning is a fourth shape. $\Delta W = -W \odot (1 - M)$ for a binary mask $M$: every changed entry goes to
exactly zero, and the only freedom is which ones.
## Deletion Tuning trains one logit per gate weight
The paper masks a single matrix per transformer block: the gate projection of the SwiGLU MLP. In Llama and Qwen2.5 the
MLP is
$$
\operatorname{MLP}(x) = W_\text{down}\big(\operatorname{SiLU}(W_\text{gate}\, x) \odot W_\text{up}\, x\big)
$$
so the gate decides, per hidden channel, how much of $W_\text{up} x$ gets through. Zeroing $W_\text{gate}[j, i]$
removes input feature $i$ from neuron $j$'s gate. The authors follow GLU pruning work in treating that gate as the
place where the MLP's channels are switched on and off.
For each target matrix $W$ they keep a trainable logit matrix $G$ of the same shape and train the layer as
$$
\widetilde{W}(t) = W \odot \sigma\big(\alpha(t)\, G\big), \qquad
\alpha(t) = \alpha_\text{start} + \tfrac{t}{T}\,(\alpha_\text{end} - \alpha_\text{start})
$$
and deploy it as $\widehat{W} = W \odot \mathbf{1}[G > 0]$. The pretrained weights never change. The sharpness
$\alpha$ ramps linearly from $10^3$ to $10^4$ under GRPO (from $10^2$ under SFT), so by the end of training the
sigmoid is a step and the soft mask the gradients saw is the hard mask that ships; the paper says swapping one for the
other moves accuracy by at most 0.08 points. In `src/learnable_sparsity/differentiable_mask.py` this is the whole
forward:
```python
# differentiable_mask.py, DifferentiableMask.forward
if self.training:
self.current_sharpness = self.sharpness[0] + (self.sharpness[1] - self.sharpness[0]) * self.progress
return torch.sigmoid(self.gate * self.current_sharpness).reshape(target_shape).to(dtype)
return (self.gate > 0).to(dtype).reshape(target_shape)
```
This is not a straight-through estimator. Gradients flow through a real, deterministic sigmoid; the trick is making
it so steep that its forward value is already almost binary. At $\alpha = 10^3$ a logit of $0.03$ gives
$\sigma(30)$, which is 1 to thirteen decimal places.
**Initialisation does a lot of the work.** The logits start from [Wanda](https://arxiv.org/abs/2306.11695), the
one-shot pruning score $s_{ij} = \lvert W_{ij}\rvert \cdot \lVert X_j \rVert_2$, computed on 2,048 training prompts.
The lowest $\eta = 1/2048$ fraction of each matrix's weights by that score start with logits drawn from
$\mathcal{N}(-0.03, 0.01^2)$; every other weight starts at $\mathcal{N}(+0.03, 0.01^2)$. In the code
(`wanda_calibration.py`) this is a draw from $\mathcal{N}(0, 0.01^2)$ shifted three standard deviations up or down.
The paper's Equation 8 works out the consequence: about 0.18% of gate weights are already deleted at step zero, and
most of those are not the Wanda picks. They are the $\Phi(-3) \approx 0.135\%$ tail of the "keep" Gaussian that
happens to land below zero. On Llama-3.2-1B's 268,435,456 gate weights that is about 131,072 Wanda deletions plus
about 362,000 random ones.
**The regulariser points the other way from the one you might expect.** The training loss is the regime's own
objective plus $-\lambda \lVert \widetilde{W} \rVert_2^2$, borrowed from [MaskLLM](https://arxiv.org/abs/2409.17481),
with $\lambda = 10^{-5}$ under GRPO and SFT and zero under self-distillation. The minus sign rewards keeping weight norm, so it nudges the mask to keep large weights.
There is no sparsity target; the paper says it lets training decide how many weights to delete.
The deployed adapter is the hard mask, bit-packed with NumPy and compressed with zstd. Because almost every bit is
"keep", it compresses to roughly 0.03 bits per masked weight. Applying it is `weight.mul_(keep)` in
`src/apply_mask.py`, after which the model loads in Transformers or vLLM like the base.
The project page animates the same figure. Watch the sigmoid: it starts as a smooth ramp, so a logit near zero gives
a mask near one half, and steepens until it is a step at zero. A logit that ends on the wrong side of zero has deleted
its weight.
## How zeroing a weight can add a skill
Deleting weights sounds like it can only remove ability. It is more expressive than that, for a mechanical reason
and an empirical one.
First, a zero is a large move. A typical gate weight's update under full fine-tuning is a small fraction of its value;
setting it to zero is a change of exactly its whole value, in a known direction. Through SiLU and the elementwise
product, removing input $i$ from neuron $j$'s gate can raise or lower that neuron's output depending on the sign of
$W_{ji} x_i$, so a set of deletions can push the MLP's write in many directions, not only towards "less".
Second, the paper's own evidence suggests most of the skill is already there. Its Table 3 samples 128 answers per
GSM8K problem at temperature 1.0: the base Llama-3-8B-Instruct already solves 95.29% within 128 tries, against
99.62% after Deletion Tuning. Greedy accuracy moves from 60.27% to 83.55%. Most of that is making an answer the model
can reach into the one it reaches first. The authors do report problems the base fails in all 128 attempts and the
masked model solves, and the pass@128 gap (4.3 points on Llama-3-8B, 6.1 on Qwen2.5-7B) is real, but the bulk of the
gain is reliability. That is the regime where [RL is known to touch small
subnetworks](https://arxiv.org/abs/2505.11711), and where choosing a few bits has a fair chance.
Pruning is usually a way to remove capacity cheaply, as in [the data-free expert prune of Kimi K3.1
Pro](/articles/kimi-k3-1-pro-pruned). Here the choice of what to prune is the learning signal, and the Wanda prior
is not the skill. The paper's Table 4 applies Wanda pruning alone, with no training, at eight
sparsity levels from 1/2 to 1/8192 and keeps the best: every backbone stays within 2.4 points of its base, while the
trained masks gain up to 53.2 points (Llama-3.2-1B on GSM8K goes from 1.21% to 54.44%). Masks as a way to adapt a
frozen network are older than this paper ([Piggyback](https://arxiv.org/abs/1801.06519) for vision,
[Zhao et al.](https://arxiv.org/abs/2004.12406) for BERT); what is new is doing it as the entire post-training update
of a decoder LLM, under RL, SFT and on-policy distillation, and comparing it head to head with full fine-tuning.
## What the paper reports against full fine-tuning
Under GRPO, the paper's Table 1 and Table 2 report best-checkpoint validation accuracy:
| Model, task | Base | Full FT | LoRA r=1 | Deletion Tuning | DT mask |
|---|---|---|---|---|---|
| Llama-3.2-1B, GSM8K | 1.21 | 60.50 | 59.29 | 54.44 | 1.0 MB |
| Llama-3-8B, GSM8K | 60.27 | 83.24 | 83.38 | 83.55 | 8.4 MB |
| Qwen2.5-7B, SimpleRL-Zoo mean | 40.18 | 51.89 | 52.13 | 52.32 | 6.9 MB |
| Qwen2.5-14B, SimpleRL-Zoo mean | 44.37 | 54.87 | | 55.43 | 11.4 MB |
| Qwen3-30B-A3B, SimpleRL-Zoo mean | 49.41 | 58.43 | | 58.54 | 29.6 MB |
At small scale the mask trails: 54.44 against 60.50 on the 1B model. From 7B up it ties or edges ahead. "Matches"
is the right word and "beats" is not. Llama-3-8B's 83.55 against 83.24 is consistent with a 1,319-problem GSM8K test
set (both percentages land on whole problems: 1,102 and 1,098), which makes the lead four problems. On Qwen2.5-7B the
0.43-point gap in the six-benchmark mean comes almost entirely from AIME24 and AMC23 in the paper's Table 8: the mask
gets 7 of 30 AIME problems against full fine-tuning's 4, and 19 of 40 AMC problems against 22. The paper reports one number per cell (I found no seeds, standard deviations or intervals in it) and picks the best
checkpoint on the validation set it reports. That is normal for the field, and it means ties within a point or so are ties.
The same pattern holds outside RL in the paper's Tables 5 and 6 (self-distillation is the on-policy recipe covered in
[on-policy self-training](/articles/on-policy-self-training)). Under SFT, Llama-3.2-3B averages 22.28% over 14
MMLU-Pro subjects (full fine-tuning 21.19%) and 11.39% on BBEH (the same as full fine-tuning). Under self-distillation
on Qwen2.5-7B-Instruct, ToolAlpaca goes to 69.07% (full fine-tuning 68.04%) and Science to 70.61% (71.20%).
**Forgetting is where the mask is clearly different.** Figure 2 tracks two things during GSM8K GRPO on the three Llama
models: forward KL from the base policy on the training task, which [RL's
Razor](https://arxiv.org/abs/2509.04259) found predicts forgetting, and mean accuracy on six prior tasks (HellaSwag,
MMLU, TruthfulQA, WinoGrande, IFEval, HumanEval). Full fine-tuning's KL reaches 1.0 on the 1B and 1.5 on the 8B, and
its prior-task accuracy ends 1.3 to 3.1 points below base. LoRA and random sparse fine-tuning (MTH) end up to 5.4
points below. Deletion Tuning's KL stays at or below 0.05 and its prior-task accuracy within a point of base. The
control matters: MTH at 99.95% sparsity trains 0.05% of all weights, about as many as the mask deletes, and still
forgets. So it is not just "few weights changed". The authors' reading is that a zero is a bounded edit chosen by the
task, where a float update at random positions is not.
The paper also compares the shape of the updates (Figure 3). Its spectrum drift and principal-weight overlap sit
next to full fine-tuning's, while LoRA r=1 drifts up to 84 times more on Llama-3-8B. On stable rank the picture is
looser than the text suggests: the mask's 66 to 93 is closer to full fine-tuning than MTH's 410 to 650 or LoRA's 1 to
2, but full fine-tuning itself reads at roughly 13 to 25 on the plot, so the mask's update is several times higher-rank
than the thing it is "closest to". Zeroing scattered entries produces a high-rank delta almost by construction.
## Counting the 84 published masks
The authors uploaded every mask: 8 GRPO, 2 self-distillation, and 74 SFT (one per MMLU-Pro subject and BBEH task for
two Llama sizes), plus the README. Each folder's `config.json` records the masked-parameter count and the zero count,
and I decompressed nine of the bitmaps to check those fields and look at where the zeros fall. The script reads the
safetensors header by hand and runs no code from the repository. Dividing by each base model's total parameter count
from its Hugging Face metadata:
| Mask | Gate weights deleted | Share of all weights | File |
|---|---|---|---|
| GRPO Llama-3.2-1B | 557,123 (0.208%) | 0.0451% | 1.27 MiB |
| GRPO Llama-3.2-3B | 0.192% | 0.0420% | 3.12 MiB |
| GRPO Llama-3-8B | 3,621,710 (0.193%) | 0.0451% | 8.36 MiB |
| GRPO Qwen2.5-0.5B | 199,564 (0.191%) | 0.0404% | 0.46 MiB |
| GRPO Qwen2.5-3B | 0.145% | 0.0380% | 2.72 MiB |
| GRPO Qwen2.5-7B | 3,161,923 (0.166%) | 0.0415% | 7.48 MiB |
| GRPO Qwen2.5-14B | 0.141% | 0.0325% | 11.44 MiB |
| GRPO Qwen3-30B-A3B | 22,954,392 (0.238%) | 0.0752% | 31.68 MiB |
| SDFT Qwen2.5-7B, ToolAlpaca | 14,892,491 (0.783%) | 0.1956% | 26.97 MiB |
| SDFT Qwen2.5-7B, Science | 33,607,213 (1.768%) | 0.4413% | 47.40 MiB |
| SFT Llama-3.2-1B and 3B, 74 masks | 0.267% to 0.982% | 0.058% to 0.213% | 1.58 to 11.16 MiB |
The paper does not say any of what follows plainly.
The abstract's "zeroing fewer than 0.05% of its weights" holds for seven of the eight GRPO masks. The body of the
paper scopes it correctly ("every GRPO run of Table 1", six models). The 30B mixture-of-experts mask deletes 0.075% of
all weights, every one of the 74 SFT masks is above 0.05%, and the Science self-distillation mask is at 0.44%. The
repost that spread the paper rounded it to "less than 0.1%", which is also wrong for those masks. All of them are
still under half a percent, which is the claim I would have led with.
The published files do not all match the tables. Llama-3.2-3B, Llama-3-8B, Qwen2.5-0.5B, 3B and 14B agree with the
paper's sizes to within rounding, and the two self-distillation masks are within half a megabyte. The Llama-3.2-1B mask deletes 0.0451% of the
model and weighs 1.27 MiB, where the paper's Table 7 says 0.0347% and Table 1 says 1.0 MB; Qwen2.5-7B is 7.48 MiB
against 6.9; the 30B is 31.68 against 29.6. The SFT masks are 3.68 to 4.51 MiB for Llama-3.2-1B on MMLU-Pro, where
Table 5 lists 1.0 MB, the same figure as the GRPO mask. I do not know whether the uploads are different checkpoints
or the tables were filled from the wrong run. The size ratios shrink with them: 4,714 MB of fp32 1B checkpoint over the measured
1.58 to 4.51 MiB SFT masks is roughly 1,000 to 3,000 to one, short of the 4,700 the tables imply.
Under GRPO the deletion count barely moves from where initialisation put it. The expected step-zero count on
Llama-3.2-1B is about 493,000 (131,072 Wanda picks plus about 362,000 noise deletions); the published mask has
557,123. Every GRPO mask sits between 0.14% and 0.24% of gate weights, around the 0.18% the paper's Equation 8
predicts at initialisation. SFT and self-distillation, which start at a sharpness of $10^2$ and run at higher learning
rates, end at 0.27% to 0.98% (SFT) and 0.78% to 1.77% (self-distillation) of gate weights, up to ten times the
starting point. "Learns how many weights to delete" is true in the SFT runs. Under GRPO the
number is mostly set by the initialisation recipe, and what training chooses is which bits flip.
The masks share a core across tasks. On Llama-3.2-1B, the GSM8K GRPO mask and the MMLU-Pro biology SFT mask share
118,097 deleted positions; chance would predict 4,836. The GSM8K mask shares 118,093 with the math SFT mask and
116,209 with the BBEH Dyck-languages mask. That shared set is close in size to the 131,072 weights the Wanda prior
starts deleted, and since Wanda scores are dominated by $\lvert W \rvert$, the obvious reading is that most of it is
the low-magnitude prior that no task bothered to restore. I cannot see the initial logits, so that is inference. The
rest of each mask is scattered: per output row, the variance-to-mean ratio of the deletion count is 1.04 on the 1B
GRPO mask and 1.08 on the 8B, which is what a uniform random scatter gives. Qwen2.5-7B is the exception, with 36
input columns holding 152,213 of its 3,161,923 deletions, consistent with Wanda's input-norm term singling out a
few channels.
## A decision skill solved into one matrix
The embedding.in post starts from an agent problem. Before acting on a request, a model has to make two calls: which
tool it needs (search, the caller's records, a calculation, an action, or none), and whether it has enough
information to go ahead. The authors labelled 393 requests with both answers. Asked plainly, Qwen3.5-0.8B picks the
right tool 80% of the time and gets the "ready to act" call right 18% of the time, "because it says 'no' to almost
everything".
The teacher is a policy built with their library [vegaml](https://github.com/NandhaKishorM/vegaml) (Apache-2.0): a
small head per question on a frozen reader model, fitted on 300 of the rows and calibrated so its probabilities mean
what they say. On the 93 rows it never saw, the post reports 96.8% on tool choice and 97.8% on ready.
The edit then moves that behaviour into Qwen3.5-0.8B itself. From the post:
- **Where.** The output matrix (down projection) of the MLP in "layer 23 of 24", restricted to 256 of its 3,584 input
columns: the hidden neurons most active at the answer position compared with how active they are elsewhere. That is
the same selection idea as [neuron tuning](/articles/neuron-tuning). The post does not say whether 23 counts from
zero.
- **Keys.** $K_S$, what those 256 neurons output at the answer position for each training request.
- **Values.** $R$, how the final hidden state would have to change for the model's answer distribution to match the
teacher's: the gap between the two distributions in log space, pulled back through the model's readout.
- **Protection.** $C_{SS}$, the second moment of the same 256 activations on text to leave alone: ordinary sentences,
the rest of every prompt, and the position just after the answer (so the model does not learn to keep answering).
- **Solve.** $\Delta W_S = R^\top K_S \left(K_S^\top K_S + \lambda n\, C_{SS}\right)^{-1}$, exactly the MEMIT-form
equation above, applied to 256 columns.
- **Guards.** Scale the edit by a dose of 0.5, 1, 1.5 or 2, keep the dose that best closes the gap on 40 held-back
rows, and refuse any dose that moves the protected text by more than a KL of 0.03.
Qwen3.5-0.8B's `config.json` gives a 1,024-wide residual stream and 3,584 MLP neurons, so the edited block is
$1{,}024 \times 256 = 262{,}144$ weights, which is the post's figure. That is 0.03% of the release's 873,438,784
parameters. The solve and the write took 3.1 ms; the whole run, including the forward passes to collect $K_S$, $R$
and $C_{SS}$ and the dose search, took 87 seconds on two T4s. LoRA at rank 16 on every MLP for 60 steps took 252
seconds. Because the last MLP's output lands one layer (or zero layers) from the readout, a linear pull-back of the
desired logit change is a reasonable approximation; the post does not discuss how much the remaining layer and the
final norm bend it.
The post's main table, on the 93 held-out rows:
| | Unedited | Direct edit | LoRA |
|---|---|---|---|
| Tool choice | 80% | 93.5% | 100% |
| Ready to act | 18% | 97.8% | 100% |
| Weights changed | | 262 thousand | 263 million |
| WikiText-2 perplexity | 28.7 | 31.7 | 33.1 |
| Held-out text perplexity | 8.11 | 8.50 | 8.94 |
| ARC-Easy (200) | 70.5% | 71.5% | 78.5% |
| Capitals (40) | 100% | 100% | 100% |
| Unrelated prompts affected | 0 of 15 | 1 of 15 | about 5 of 15 |
And its ablations:
| Change | Tool | Ready | WikiText ppl |
|---|---|---|---|
| Full method | 93.5% | 97.8% | 31.7 |
| No protection ($\lambda = 0$, no cap) | 96.8% | 97.8% | 1.2e11 |
| Hard labels, not the calibrated policy | 95.7% | 97.8% | 30.4 |
| 25% of the training rows | 95.7% | 95.7% | 32.7 |
| Random change of the same size | 74.2% | 81.7% | 29.2 |
The ablation that matters is the second row. Drop $C_{SS}$ and the same solve still hits the targets, and WikiText
perplexity goes to $1.2 \times 10^{11}$. The protection term is the method; without it a least-squares edit is a way
to break a model precisely.
## Reading the edit post's numbers at their sample sizes
Every number in the post is the authors' own, on their own data, and I found no published code for the edit itself
(the vegaml repository is the teacher library; I searched it for the solve and found only its own ridge fits for
reader ports). With that said, most of the post is careful, and it volunteers its own limits. The problem is what
happened to it in the repost, where "18% to 97.8%" and "1/15 (LoRA 5/15)" became the headline.
**18% is not the baseline that matters.** The unedited model says "no" to almost everything and scores 18%, so roughly
82% of the 93 rows are probably "ready" cases (my inference; the post gives only the 18%). A model that always said
"yes" would then score about 82%. The post's random edit lands at 81.7%, which the post puts down to the model's
bias towards "no". The fair comparison for the ready question is
roughly 82% to 97.8%, which is about 76 rows to 91. On tool choice, 80% to 93.5% is 74 rows to 87.
**The ceiling is the teacher.** The teacher scores 97.8% on ready and 96.8% on tool. The edited model matches it on
ready and trails it on tool. The edit transfers a policy; it does not improve on one.
**1 of 15 against about 5 of 15 is not yet a difference.** I ran a two-sided Fisher exact test on those counts:
p is about 0.17. The examples are vivid (LoRA answering "suggest a cat name" with "no"; the edit listing the primary
colours as "red, no, no, no"), and the direction is plausible given that LoRA changed a thousand times more weights.
But fifteen prompts cannot carry the claim.
**Off-format, nothing transfers.** Reworded, the same questions get 5 of 11 right from the edited model, 8 of 11 from
the unedited one and 9 of 11 from LoRA. Five against eight is no more significant than the damage count (the same
test gives p of about 0.39), so I would not claim the edit made things worse. What the 11 prompts do show is no sign
that the skill survives a new wording, and the post says so and proposes solving over several formats at once. A
decision skill that only fires on one template is a lookup keyed on that template.
**"263 million" is LoRA's merged footprint.** A merged rank-16 delta on all three MLP matrices of all 24 layers
touches $24 \times 3 \times 1{,}024 \times 3{,}584 = 264{,}241{,}152$ weights by my count; the post's 263 million
is close but not equal, perhaps from counting entries whose fp16 value actually changes. By the same arithmetic on
the shapes, LoRA trained 5,308,416 numbers to get there. The edit fitted 262,144. "A thousand times fewer weights changed" compares merged footprints, and on that
metric it is right.
The perplexity costs are smaller than LoRA's: WikiText-2 rises 10% for the edit (28.7 to 31.7) and 15% for LoRA (28.7
to 33.1). ARC-Easy on 200 questions barely moves for the edit; the random change of the same size drops it from 70.5%
to 52.5%, which is a useful reminder that a quarter of a million weights in the last MLP is not a small edit. LoRA's
ARC gain to 78.5% the authors attribute to it learning to answer in a single word, which seems right.
## "The same idea MIT explored"
The repost of the edit post says the result is "same idea MIT explored with a frontier model". The blog itself does
not mention MIT. The closest match is [Sparks of Weight-Space Programming](/articles/weight-space-programming) from
MIT CSAIL a week earlier, where GPT-6 Astra edited Qwen3.5-0.8B, the same base model, with no gradients. The repost
gives no link, so that identification is my inference.
They share a base model and the slogan "no training, no gradient descent". The mechanisms differ. Astra spent 20
hours of forward passes searching for a failure mode and found a rank-one fix: subtract a "loop" direction from 64
columns of two down projections. The embedding.in edit is a supervised least-squares solve from labelled rows to a target distribution,
done once. The second is much closer to MEMIT
than to the MIT work.
## Same toy, three edits
The widget below is one SwiGLU unit with two inputs and six hidden neurons, small enough to draw everything. Its
"old task" is its own sign over a 21 by 21 grid, so the base keeps all 441 cells by definition. The "new task" is
seven labelled points, three of which the base gets wrong. You can edit it three ways:
- **Delete gate weights**, as Deletion Tuning does. With 12 gate weights there are only 4,096 possible masks, so the
best masks come from searching all of them instead of a sigmoid relaxation.
- **LoRA rank 1 on the gate**, trained by gradient descent on the seven points only.
- **A closed-form write to the down projection**: the MEMIT-form solve above with $K$ the six hidden activations at
the new points, $C$ the second moment of those activations over the grid, and a slider on $\lambda$.
When I ran it, deleting one gate weight fixed all seven points and kept 423 of 441 cells. LoRA got all seven by step
100 with 369 cells kept, then kept training and drifted to 343 by step 800, because nothing in its loss mentions the
old task. The closed-form write gets all seven at every $\lambda$; raising $\lambda$ from 0 to 1 takes the kept cells
from 324 to 386. Do not read the toy as a ranking. I chose this unit by a seeded search for one where a single deletion
works. What it shows is mechanics: a deletion is a large, discrete move; LoRA's damage grows with steps unless
something anchors it; the closed-form write trades fit for protection through one explicit knob.
## Weights touched is the wrong headline
Both posts sell themselves on how few weights change. That number is real, but it hides where the cost went.
Deletion Tuning changes 3,621,710 weights on Llama-3-8B and ships 8.4 MB. To find them it trains a logit for every
one of the 1,879,048,192 gate weights, which the paper concedes in its limitations: less training memory than full
fine-tuning, more than LoRA. It also needs the full RL or SFT pipeline: rollouts, a verifier or labels, and H200s in the
authors' setup, several of them past 3B. The closed-form edit changes 262,144 weights with a 3 ms solve, but it
needs a teacher that already makes the decision, labelled rows, protected text, and a target you can express as a
change in the last hidden state. LoRA changes everything a little and trains a few million numbers, and it is the only
one of the three that generalised to reworded prompts in the edit post's test.
The count of changed weights predicts forgetting less well than either post implies. In the paper, MTH changes
about as many weights as Deletion Tuning and forgets like LoRA. In the edit post, a random change the size of the edit
breaks ARC-Easy. What seems to matter is that the change is chosen by the task, bounded in size, and anchored to what
should stay the same: Deletion Tuning bounds every edit at "set to zero" and stays near the base in KL; the closed-form
edit anchors through $C_{SS}$ and a KL cap. Few weights is a side effect of those choices.
## Where I land
Deletion Tuning is the more important of the two. It is a careful paper with code, with masks for every reported run,
and with a forgetting result that holds against a matched-size control. "Post-training by selection" is a useful way to
think about RL in particular, where the paper's own pass@k shows the base model could already reach most of the
answers. I would fix the abstract, whose "fewer than 0.05%" is a GRPO number and not an across-regimes one, and I would
want the authors to reconcile the uploaded masks with the size tables where they disagree. If I needed several
task adapters on one base model with minimal interference, I would try it before LoRA, starting from the published
masks and `src/apply_mask.py`.
The decision-skill edit is a well-written single experiment. The mechanism is MEMIT's, aimed at a calibrated teacher
instead of a fact, and the post is honest about its narrowness. On 93 test rows against an 82% constant-answer
baseline, with a 15-prompt damage check and an 11-prompt format test where it scored below the unedited model, it is a promising demo, not evidence
that closed-form edits beat LoRA. I would not deploy a skill that breaks when the prompt is reworded; I would want the
multi-format solve the authors propose, and their code, first.
### Files behind this page
I read the paper, the repository's masking, Wanda and apply code, the post, and vegaml. I ran a standalone parser
over the published masks (nine bitmaps decompressed, all 84 `config.json` files read), the Fisher tests and the toy.
I did not train anything, evaluate a masked model, or reproduce the edit.