Neuron tuning: fine-tune eight neurons, and measure what else moves

2026-11-09 · 39 min · interpretability · fine-tuning · lora · small-models

Why read this

Recomputes its 0.90 correlation, re-measures the LoRA gap on its own reference text, and ablates K, layer 0 and the filter on CPU: local, but facts spread.

Checked against the source
Runs on a consumer GPU
A lasting reference

InterpretabilityApache-2.0Practitioner teardown

How this was scored

Is it new?: 2 of 3: A real new idea, method or capability
Can I trust it?: 3 of 3: Reproduces the headline result, or shows from primary files it is wrong
Can I run it?: 2 of 3: Open code or weights with real limits
Will I understand it?: 2 of 3: Mechanism from first principles with figures
Can I act on it?: 2 of 3: A concrete recipe, numbers or comparison
Will it last?: 2 of 3: A reference for a year or more
Does it affect many?: 1 of 3: A specialist community
Only here?: 2 of 3: A teardown or measurement few others did

Score 73 of 100, ranked 87 of 701 rated articles. Each question is answered 0–3 by hand, and a 3 is rare. How articles are scored

On 1 November @ChangJonathanC posted: "Fine-tune only a few neurons that fire in the target context, and the effects stay in that context. Compared to LoRA, Neuron Tuning changes the model 3-19x less on unrelated text." The next post in the thread gives the whole method in two lines: find neurons that fire on the target contexts and are quiet on other text, and train only the output weights of those neurons.

That is a claim about side effects, and side effects can be measured. This article explains why the method should work, reads how the repository chooses its neurons and measures the change, recomputes the number that carries its "why it works" section, and then re-runs the idea twice on a 135M-parameter model on a CPU: once against four baselines on text I collected, and once on the repository's own reference text with three ablations (how many neurons, which layers, and the selectivity filter). It is the third piece here on the same author's neuron work: neuron steering explains what pushing one neuron does, neuron finding reads neuron-finder, and neuron lab lets you steer neurons in a browser. Steering adds a fixed push; tuning trains a few weights and leaves the rest alone.

Every claim carries one of three labels. Measured means I ran it or recomputed it from files. Reported means the repository, the blog post or the thread says so and I did not re-run it. Reasoned means arithmetic or inference from the two.

Why a neuron's output column is a gate

An MLP block in a transformer reads the residual stream, computes a few thousand activations, and writes back a weighted sum of down_proj's columns, one column per neuron. The write of neuron $j$ at token $x$ is

\Delta r(x) = a_j(x)\, W_{\text{down}}[:, j]

where $a_j(x)$ is the neuron's activation, the gated product of gate_proj and up_proj. If you change only the column, the change to the model at token $x$ is $a_j(x)\,\delta_j$ . Where the neuron is quiet, $a_j(x)$ is near zero and the edit does nothing, whatever $\delta_j$ becomes. The blog post's phrase for it is that the edit "acts like an if-statement that the model already has" (reported).

A LoRA does not have that property. It adds $B A x$ to a weight that every token passes through, and $A x$ is a projection of the token's input, almost never exactly zero. So a LoRA trained on cooking questions writes something on Python and German too, and the only thing that keeps that write small is the training run itself (reasoned).

Two things follow, and the repository tests both. If the neurons fire everywhere, the gate is always open and the edit should be less local than a LoRA. And the gate is a property of context, not of a fact: a neuron that fires on "the oldest tree at a garden" fires on every garden.

Four panels left to right: a behaviour given by a system prompt; find neurons that fire on the target replies and are quiet on other text; train only their output weights; measure the change on unrelated text as KL divergence per token — The method in four steps: the behaviour comes from a system prompt, the edit must reproduce it without the prompt, and the side effect is measured on fixed unrelated text (repository report, method figure).

What the repository is

changjonathanc/neuron-tuning is Apache-2.0, authored by "Claude Opus 5.5 · Jonathan Chang"; the report says the code, the experiments and the text "were written and run by Claude … with direction and feedback from Jonathan Chang" (reported). I read it at commit 8a55f31 (2 November 2026). It holds the tool, make_edit.py (859 lines), fifteen pre-registered experiments with their saved result files, nine demos, a full report and an independent review of the claims (measured, by counting). The main model is Gemma-4-E2B-it; Qwen3.5-4B appears in one experiment. Training needs an NVIDIA GPU; the analyses run on a CPU from the committed files.

How the tool chooses neurons

The rule in make_edit.py, which docs/make_edit.md explains (read from code):

Activity. For each neuron, $c_j$ is the mean $|a_j|$ on the target replies, at exactly the positions the loss scores (the last prompt token and the reply tokens). $g_j$ is the same statistic on background text from seven domains, averaged over the domains. The items used to measure the side effect are never used here.
Filter. A neuron is eligible if $c_j$ is in the top 10% of its layer and $g_j / c_j \le 0.2$ .
Score. For each eligible neuron, $G_j$ is the gradient of the training loss with respect to its column, summed over up to 64 training items. The score is $\lVert G_j \rVert / g_j$ , the loss reduction per unit of write on other text, to first order. The top $K$ are edited, $K = 8$ by default.

What trains is a gain and an added vector on each chosen column, $\delta_j = g\,W_{\text{down}}[:,j] + u\,\text{rms}(W_{\text{down}}[:,j])$ . On Gemma-4-E2B-it with $K = 8$ that is 12,296 numbers, saved as a 52 KB patch (reported). The loss is the KL divergence from the model with the behaviour's system prompt to the model without it plus the edit, on the prompted model's own replies. No term protects other behaviour: "locality is not trained in; it comes from the method or it does not" (reported).

The rule has a history the docs are open about. Gradient attribution alone picked neurons that fire more on other text than on the target for one rule edit, and that edit changed unrelated text 37x more than a LoRA. Activity alone picked early-layer neurons for the casual edit that needed a 10x larger learning rate. The combined score was "not tested as a pre-registered experiment. It is a design choice from the evidence" (all reported).

The method, term by term, with shapes

It helps to put shapes on all of this, using the model I ran it on. SmolLM2-135M-Instruct has hidden size $d = 576$ , 1,536 MLP neurons per layer and 30 layers (measured, from its config.json). One MLP block at one token is

a(x) = \mathrm{SiLU}(W_{\text{gate}}\,x) \odot (W_{\text{up}}\,x), \qquad \text{out}(x) = W_{\text{down}}\, a(x)

with $x \in \mathbb{R}^{576}$ , $W_{\text{gate}}, W_{\text{up}} \in \mathbb{R}^{1536 \times 576}$ , $a \in \mathbb{R}^{1536}$ and $W_{\text{down}} \in \mathbb{R}^{576 \times 1536}$ . Neuron $j$ is entry $j$ of $a$ , and column $j$ of $W_{\text{down}}$ is the 576-vector it writes into the residual stream. On Gemma-4-E2B-it the same column is 1,536 long, which is where the repository's "1,537 numbers per neuron" comes from (reasoned).

The parameterisation. experiments/lib/p1lib.py writes the trained column as

W_{\text{down}}[:,j] \;\to\; (1 + g_j)\,W_{\text{down}}[:,j] + \mathrm{rms}(W_{\text{down}}[:,j])\;u_j

with one scalar $g_j$ and one vector $u_j \in \mathbb{R}^{d}$ , both starting at zero, so the edit starts as the identity (read from code). $g_j$ turns the neuron's existing write up or down; $u_j$ points it somewhere new. Multiplying $u_j$ by the column's RMS puts every column on the same footing for Adam: Adam moves each coordinate by about the learning rate whatever the gradient's scale, so without the factor a column with small weights would move proportionally much further than one with large weights (reasoned). Per neuron that is $d + 1$ trained numbers: 577 here, so $K = 8$ is 4,616.

The gradient the score uses. Because $\partial\,\text{column} / \partial u_j = \mathrm{rms}_j$ , the gradient of the loss with respect to the column is $\partial L/\partial u_j$ divided by $\mathrm{rms}_j$ . That is how column_grads() gets $\lVert G_j \rVert$ for every eligible neuron from one zero-initialised adapter and one backward pass per item, without materialising the gradient of all of $W_{\text{down}}$ (read from code). The score $\lVert G_j\rVert / g_j$ then reads as the first-order loss reduction available per unit of step on this column, divided by how much the neuron fires on other text, which is how far that step would reach (reasoned).

A property of the score the docs do not mention. $g_j$ is an absolute activity, not a ratio, and layers differ in scale. On SmolLM2 the three eligible layer-0 neurons have background activity 0.045-0.060, against 0.12-0.38 for the eligible neurons in layers 21-29 (measured, my selection pass below). Dividing by a small $g_j$ gives those three scores of 26-126, against at most 16.6 for any deeper neuron, so the rule's top three picks are all in layer 0, although two of them have the highest selectivity ratios $g/c$ in the top eight (0.19 and 0.18). That is the mechanism behind the layer-0 picks of my first run, and the second run below tests what it costs.

The loss. In the study's "system" mode it is the per-token KL from the teacher (the same model with the behaviour's system prompt) to the student (no prompt, edit on), on the teacher's own replies $y$ :

\mathcal{L} = \frac{1}{T}\sum_{t=1}^{T} \mathrm{KL}\big(p_{\text{teacher}}(\cdot \mid y_{<t}) \,\Vert\, p_{\text{student}}(\cdot \mid y_{<t})\big)

In "pairs" mode, which is what I used, it is the cross-entropy on given replies. Adam, constant learning rate, 400 steps of batch 8 by default (read from code; my runs use 150 steps of batch 4).

The measurement. $F$ is the per-token $\mathrm{KL}(p_{\text{orig}} \Vert p_{\text{edit}})$ , averaged over the scored tokens of an item, then over the items of a domain, then over the seven domains, teacher-forced on fixed text and scored only on the reply tokens of chat-shaped items (unrelated_change(), read from code). Strength is matched by interpolation: each method gives a curve of ( $\ln$ strength, $\ln F$ ) points over its learning rates, and $F$ is read off that curve at a target strength. A method whose curve does not reach the target is "not covered", not extrapolated.

What the code does that the prose does not say

Reading make_edit.py against docs/method.md turned up six things worth knowing before using it (all read from code):

The neurons do not depend on the seed. select_neurons() draws its 64 scoring items with random.Random(0), so a seed changes the training order only. Seed spread in the study is training noise, not selection noise.
There is a --min-layer flag, default 0. Only layers at or above it are eligible. Experiment 15 and the keigo demo use --min-layer 10 together with other changes, but no experiment trains the same edit with and without it, and experiment 14 names "a filter on early layers" as a candidate improvement that "needs a new run with the habit measured (not done)". I ran that below.
The tool refuses to run when fewer than $K$ neurons pass the filter (sys.exit, "raise --max-ratio or lower --k"). On Gemma, 256 neurons passed for the type-hint demo; on my fact and SmolLM2, 25 did.
The "same-size LoRA" is different for non-Gemma models. For Gemma it is rank 1 on down_proj of the layer(s) nearest layer 15. For any other model it is rank 1 on the one layer holding the most chosen neurons, ties to the lower layer. With three layer-0 picks that is layer 0 here, which is why I add a second same-size LoRA deeper in the stack.
docs/method.md describes older rules. It gives the activity rule's score as $\ln(c/g)$ and gradient attribution as the gradient times the teacher-minus-student activation. The tool's default is neither: it is the filter, then $\lVert G\rVert/g$ .
A patch is exact. Patch adds $a_{\text{idx}}\,\Delta^\top$ to the output of down_proj with a hook; bake() writes the same change into the weights as an fp32 sum rounded once. The file holds the neuron indices and the $d \times K$ change per layer, nothing else.

How it measures the side effect

The change on unrelated text $F$ is $\mathrm{KL}(\text{original}\,\Vert\,\text{edited})$ per token, teacher-forced on 256 fixed items from seven domains (HumanEval code, GSM8K maths, SQuAD questions, German, Spanish and Japanese Wikipedia, WikiText-2 documents), averaged over tokens, then items, then domains. Methods are compared at the same strength: each is trained at four to six learning rates with three seeds, and $F$ is read off each method's curve where it reaches a strength target fixed before the runs (reported). That last part matters more than anything else in the setup: a weaker edit always changes less, so a comparison at one learning rate says nothing.

Log-log plot of change on unrelated text against distance to the target behaviour for the casual edit. Four curves, one point per learning rate: 8 neurons that fire everywhere sit highest, the same-size rank-1 LoRA next, the standard rank-8 LoRA below it, and 8 gradient-chosen neurons lowest. A vertical line at teacher KL 0.6 marks where the ratios are read: 12x less than the same-size LoRA and 8x less than the standard LoRA. — How the matched comparison works, on the casual edit: each method is a curve over its learning rates, and the ratios are read where every curve crosses one strength. The neuron curve stops before the LoRAs' strongest points, which is why the review asks for the weaker target to be named (repository report, trade-off figure).

What it reports

Three panels of dot plots on a log axis of change on unrelated text relative to a reference LoRA. Talk casually: neurons that fire everywhere 9x more, standard LoRA 1.5x less, 8 gradient-chosen neurons 12x less, 8 other selective neurons 19x less, 8 activity-chosen neurons 3x less. Weather rule: gradient-chosen neurons that fire everywhere 37x more, selective neurons 9x less. 160 facts: 128 gradient neurons 7x more, 128 neurons that fire everywhere 4x more, 128 selective neurons 5x less than a standard LoRA — Change on unrelated text at the same medium strength, relative to a LoRA, for three edits on Gemma-4-E2B-it. Blue: neurons that fire mostly in the target context; orange: neurons that fire everywhere (repository report, headline figure).

The headline numbers, all reported on Gemma-4-E2B-it:

Casual register: 8 selective neurons change unrelated text 12-19x less than a rank-1 LoRA of the same size, and 7-14x less than a standard LoRA (rank 8 on every projection of every layer, about 12 M numbers). With 128 neurons the gap to the standard LoRA is 8x at medium strength and 3x at the strongest effect the neurons reach.
Weather rule (answer weather questions in one sentence): 9x less than the same-size LoRA, and the rule then fires on 11% of non-weather questions against 80% for the LoRA.
160 made-up facts: 128 neurons, 5x less than a standard LoRA at 0.5 held-out accuracy, 3x at 0.7.
Second model: on Qwen3.5-4B, casual edit, one seed, 200x less than a rank-1 LoRA and 160x less than the standard LoRA on the same seven domains.

The repository ships its own independent review (docs/review.md), and it is the most useful file in it. It checked every headline number against the analysis outputs (all correct) and then listed what overclaims: the 12x is at the weaker of two pre-registered strength targets, the stronger one is out of the same-size LoRA's reach and was pre-registered as "not covered"; the gap narrows as edits get stronger; the before-training predictor only separates coarse groups; and two analysis choices made after a first look decide verdicts (an off-by-one in the activity positions, without whose fix "the weather result does not hold", and a change of the facts task's strength measure). The README was rewritten after the review, and the "3-19x" in the post is the review's own suggested wording (all reported). The Limits section is just as direct: "Local means 'other kinds of text', not 'similar prompts'." On text close to the target, the advantage over the standard LoRA is 1-3x, and a new fact taught to 8 neurons spread to 91% of held-out questions about other open problems, as a LoRA's did (88%).

Why it works, recomputed

The report's explanation is one number. The off-target write of an edit is $\sum_{\text{layers}} \lVert \Delta W\, a(x) \rVert$ averaged over tokens of unrelated text, with $a(x)$ the unedited model's inputs: how active the edited units are off target, times how big the update is. Over 210 trained edits (neurons and LoRAs, four behaviours), it orders the change on unrelated text with Spearman $\rho = 0.90$ (reported).

The per-run values are committed in experiments/03_predictor/out/pred.json and f32.json. I did not run the repository's analyze.py (it needs scipy, which the Python environment here does not have); I parsed both files and wrote my own rank correlation. Using the fp32 $F$ where the repository has it (111 runs) and the stored bf16 $F$ for the other 99, as analyze.py does, I get 0.898 over 210 edits, and 0.899 with the bf16 values only (measured). The number holds.

reported runs · correlation recomputed here

● neurons quiet off-target (S below 0.35) ● neurons active off-target ● LoRA210 edits · Spearman 0.90

Hover or tab to a dot. Each family has several learning rates and seeds, so a big part of the correlation is learning rate.

Two things the scatter shows that the headline does not. First, each family is several learning rates, and within a family the learning rate alone orders $F$ , so part of 0.90 is "bigger updates change more", which no one doubts. Switch to a single task to see what is left. Second, within two families the order breaks: gradient-chosen neurons on the lower-case rule have $\rho = 0.00$ and on the recipe rule $-0.68$ , nine runs each (measured, recomputed; the repository's table gives the same). The review makes the same point from the other side: before training, among neurons that are all quiet off target, activity does not predict which edit is most local, $\rho = -0.43$ over the eight selective points (reported). Quiet neurons are necessary; they are not sufficient, because the update size is only known after training.

Scatter on log axes of off-target write against change on unrelated text for 210 edits, colour by method; the points lie along a rising band with Spearman 0.90 — Off-target write after training against change on unrelated text, 210 edits (repository report, predictor figure).

A first CPU re-run on SmolLM2-135M

I wanted to know whether the effect survives a model more than ten times smaller, a different kind of edit and someone else's code. So I re-implemented the method from the description above and ran it on SmolLM2-135M-Instruct (30 layers of 1,536 MLP neurons, so 46,080; hidden size 576) in float32 on a shared 4-vCPU machine, 2 threads, no GPU. The site owner had authorised running the repository's code, but its training path needs CUDA and a gated Gemma checkpoint, so nothing below executes it; the selection rule, the column parameterisation and the measurement are mine, written to match make_edit.py.

The edit. One benign made-up fact: the oldest tree at Thornfield Botanical Garden is a ginkgo named Bartholomew. 36 phrasings of the question, 24 for training and 12 held out, with four reply templates. This is the repository's "pairs" mode (cross-entropy on given replies), not its context distillation.

Five ways to teach it, each trained for 150 Adam steps of 4 items at a constant learning rate, one seed:

method	what trains	numbers
8 selective neurons	the tool's rule: top 10%, $g/c \le 0.2$ , score $\lVert G\rVert/g$	4,616
8 neurons that fire everywhere	random among the 10% most active on background text	4,616
rank-2 LoRA, layer 0	`down_proj` of the layer the tool's own rule would pick	4,224
rank-2 LoRA, layer 25	the same, one of the deep layers the neurons sit in	4,224
rank-8 LoRA, every layer	q, k, v, o, gate, up, down of all 30 layers, alpha 16	2,442,240

Learning rates were 0.003 to 0.1 for the neuron edits (0.001 to 0.01 for the layer-25 LoRA, 0.0003 to 0.01 for the layer-0 one) and 0.000003 to 0.001 for the rank-8 LoRA; one run, training plus all measurements, took 3.5 to 8 minutes (measured).

The measurements. Strength is the held-out cross-entropy on the fact's replies (2.35 nats per token unedited). $F$ is $\mathrm{KL}(\text{original}\,\Vert\,\text{edited})$ per token on 84 unrelated items, 12 per domain from the same seven kinds of text as the repository (HumanEval, GSM8K, SQuAD, German, Spanish and Japanese Wikipedia, WikiText-2), up to 160 tokens each; another 84 items from the same sources are the background for choosing neurons. Behaviour is counted on greedy 28-token replies: how many of the 12 held-out questions name Bartholomew, and how often the name appears for 16 near questions (other gardens, other old trees) and 16 unrelated chats.

The neurons it chose. Two in layer 0, then layers 21, 23, 24, 25 (two) and 29, with $g/c$ between 0.091 and 0.197; the geometric mean $S$ is 0.132 (measured). The everywhere neurons have $S = 1.585$ : they fire more on background text than on the fact. The layer-0 picks are what the docs warn about, and why the tool's tie-break for "the same-size LoRA" lands on layer 0 here, as it did in the repository's Qwen experiment; I added the layer-25 LoRA so the comparison does not rest on that.

measured · SmolLM2-135M-Instruct · CPU · seed 0right = stronger edit · down = more local

● 8 selective neurons · 4,616 numbers● 8 neurons that fire everywhere · 4,616 numbers● rank-2 LoRA, layer 0 · 4,224 numbers● rank-2 LoRA, layer 25 · 4,224 numbers● rank-8 LoRA, every linear layer · 2,442,240 numbers

Hover or tab to a point: each is one training run. The counts are greedy replies that contain the name Bartholomew.

At matched strength, interpolating $\ln F$ against $\ln$ cross-entropy along each method's learning-rate curve (measured; one seed, and the strength targets were chosen after the runs, so read these as a demonstration):

held-out cross-entropy	8 selective neurons, $F$	everywhere neurons	LoRA layer 0	LoRA layer 25	rank-8 LoRA, all layers
0.85	0.0034	34x	6.9x	not reached	1.3x
0.5	0.0057	42x	7.6x	3.6x	1.8x
0.35	0.0089	44x	7.3x	4.6x	2.1x
0.3	0.011	44x	9.0x	5.1x	2.3x

The multiples are each method's $F$ divided by the neurons'. "Not reached" means that method's curve does not pass through that strength at the learning rates I ran. The everywhere neurons jump from 1.45 to 0.29 nats between their last two learning rates, so their multiples are read off one long segment.

What I take from it:

The direction replicates; the size depends on the baseline. Selective neurons were the most local method at every strength they reached. Against neurons that fire everywhere the gap is 34-44x, close to the 37x the repository reports for its weather rule. Against a LoRA of the same size it is 3.6-9x, depending on which layer the LoRA sits in. Against the rank-8 LoRA on every layer it is only 1.3-2.3x, far below the 7-14x the repository reports for its casual edit on Gemma.
Why the big LoRA does so well here is a guess (reasoned). It has 2.4 M numbers and reaches a given strength at a tiny learning rate (1e-5 already gives 1.42 nats), so its per-weight update is very small. The task is also a narrow one: 24 phrasings of one sentence, where the repository's casual edit changes the register of every reply. And the neurons the rule picked include two in layer 0, which the docs say need larger updates. One seed cannot tell these apart.
Where the change lands is uneven. For the neuron edit at learning rate 0.03, $F$ on German was 0.022 and on code 0.00032, about 70x apart; the LoRA on every layer spreads its change much more evenly, 0.015 to 0.029 across all seven domains at its smallest learning rate (measured). The neurons' change sits where they fire.

The fact spread to every garden

The behaviour counts are where the method's limit shows. With the neuron edit at learning rate 0.03, 10 of 12 held-out questions named Bartholomew, and so did 13 of 16 near questions: "The oldest tree at Kew Gardens is a ginkgo named Bartholomew." "The oldest tree in Central Park is a ginkgo named Bartholomew." It never appeared in the 16 unrelated chats, at any learning rate, and one near question about Thornfield's head gardener was left alone (measured).

Even before the name was learned, at the smallest learning rate, the neuron edit had turned 11 of 16 near answers into ginkgos, against 2 of 16 for the unedited model (and both of those questions mention ginkgo themselves). The unedited model already invents a name for each garden's tree ("Kewia" for Kew); after the edit the invented tree was a ginkgo. The neurons that fire on "the oldest tree at a garden" are the gate, and every garden opens it. This is the repository's own finding, the Jacobian fact that spread to 91% of other open problems, reproduced on a different model with a different kind of edit (reasoned, from both).

The LoRAs are not cleaner here. The rank-2 LoRA on layer 0 at 3e-3 named Bartholomew for only 1 of 16 near questions, but it also named it for 7 of 12 held-out questions, against 10 of 12 for the neurons at a similar strength, and its $F$ was 10x higher; at its next learning rate the near count went to 11 of 16. The layer-25 LoRA and the rank-8 LoRA leaked to 11 to 16 of 16 near questions at the strengths where they knew the fact, and the rank-8 LoRA at its largest learning rate also put the name into 3 of 16 unrelated chats (measured). Near-topic spread is a property of teaching one fact by gradient descent on a few phrasings, not of this method.

The unrelated chats are where the neuron edit did keep its promise. Counting the word "ginkgo" rather than the name: it appeared in 0 of 16 unrelated chats after every neuron edit, at every learning rate, and in none for the unedited model. The layer-25 LoRA put it into 5 and 6 of 16 at its two stronger learning rates, the rank-8 LoRA into 3 of 16 at its strongest, and the everywhere neurons into 2 of 16 at theirs (measured). That is the "other kinds of text" half of the claim, seen in behaviour and not only in $F$ .

Where each edit writes

The gate can be seen directly. For each trained edit I computed, per token, the norm of what it adds to the model: $\sum \lVert \Delta W\, x \rVert$ over the edited modules, with $x$ the unedited model's input there, the per-token form of the repository's off-target write (measured).

measured · where the edit writes, token by token

held-out question about the factmean write 82.0 (first token left out)

↵The oldest tree at Thornfield Botanical Garden is a ginkgo named Bartholomew.<|im_end|>

another garden's oldest treemean write 81.9 (first token left out)

↵The oldest tree at Kew Gardens is a sweet chestnut planted in the 1700s.<|im_end|>

unrelated chatmean write 19.2 (first token left out)

↵Put the egg in boiling water for about nine minutes, then cool it in cold water.<|im_end|>

Pythonmean write 8.3 (first token left out)

def mean(xs):↵ total = 0↵ for x in xs:↵ total += x↵ return total / len(xs)↵

Germanmean write 9.3 (first token left out)

Der Rhein ist ein Strom in Mitteleuropa. Er entspringt in den Alpen der Schweiz und mündet in die Nordsee.

one colour scale for all three edits (untick to scale this edit to its own maximum)

trained at learning rate 0.03, seed 0

The neuron edit (learning rate 0.03) writes 82 per token on the held-out fact question and 82 on Kew's oldest tree, 19 on an egg-boiling chat, 8.3 on a Python function and 9.3 on German, leaving out each text's first token: about ten times more on the target than on code. The layer-0 LoRA (0.003) writes 27 on the target and 6.6 on code, four times more. The LoRA on every layer (0.00003) writes 46 on the target and 24 on code, under twice (all measured). So the gate is there and visible, and it is the same on Kew as on Thornfield. Two cautions: the neuron edit is not silent off target (about 7 per token is its floor, much of it from the two layer-0 neurons), and a write in layer 0 is not worth the same as a write in layer 25, because everything after it reads it. Compare the ratios within one edit, not the raw numbers across edits.

A second run, on the repository's own text

The first run left three questions open. Its $F$ was measured on raw text I collected, while the repository scores chat-shaped items on their reply tokens, so its 1.3-2.3x against the rank-8 LoRA might be a property of my reference set. Two of its eight neurons sat in layer 0, and I could only guess what they cost. And it trained one $K$ . So I ran the method again, changing the measurement and adding three ablations.

What changed. $F$ is now measured on the repository's own reference file, experiments/lib/data/forget_set.json (256 chat items in seven domains): the first 12 items per domain whose prompt and reply fit in 1,024 tokens, 84 in all, with at most 160 scored reply tokens each, scored at the reply positions as unrelated_change() scores them. Neurons are chosen against all 128 items of the repository's background file, experiments/01_locality_curves/data/ref_set.json, at the reply positions as domain_acts() does. The fact is the same, rewritten as 36 fresh phrasings (24 to train, 12 held out); the unedited model's held-out cross-entropy is 2.105 nats per token. The unedited model says "Bartholomew" on 1 of 12 held-out questions (the one that contains the name), on none of the 16 near questions and none of the 16 unrelated chats. I wrote down the two strength targets, held-out cross-entropy 0.5 and 0.3, before the first run of this set (all measured). One seed, 150 Adam steps of 4 items, 3 to 5.5 minutes per run on 2 CPU threads.

What the rule chose. 25 neurons of 46,080 pass the filter (top 10% on the target and $g/c \le 0.2$ ). Ranked by $\lVert G\rVert/g$ , the first three are in layer 0 (neurons 1388, 1037 and 981, scores 126, 84 and 26), and the next five are in layers 21-25 (scores 16.6 down to 7.8). The default $K = 8$ takes all eight (measured). The tool's same-size LoRA for this set is layer 0 again.

The headline comparison, re-measured

at held-out cross-entropy	8 selective neurons, $F$	rank-8 LoRA, every layer	rank-2 LoRA, layer 24	rank-2 LoRA, layer 0
0.5	0.0016	28x	22x	156x
0.3	0.0042	18x	25x	83x

The multiples are each method's $F$ over the neurons' $F$ at the same strength, interpolated along three learning rates per method; the layer-24 LoRA is the same-size baseline placed where the deep neurons sit (layer 24 holds two of them, tied with layer 25, and the tool breaks ties to the lower layer) (measured). On the repository's own reference text the gap to the rank-8 LoRA is 18-28x, about ten times the 1.3-2.3x of my first run, and above the 7-14x the repository reports on Gemma (reported). That is not the whole story, because the macro mean hides where the difference comes from. Per domain at cross-entropy 0.5, the LoRA's $F$ is 4.1x the neurons' on WikiText documents, 5.0x on Japanese, 6.1x on German and 6.9x on Spanish, but 53x on GSM8K maths, 90x on HumanEval code and 413x on SQuAD questions; at 0.3 the same seven ratios run from 3.0x to 130x, and the median domain is 6.6-6.9x (measured).

The three domains where the LoRA does worst are the ones whose replies are short and formulaic: a SQuAD reply is often a few words ("October 12, 1943."). A LoRA trained on 24 chat replies that all begin "The oldest tree at…" learns something about how any reply begins, and every chat item pays for it at its first reply tokens. The neuron edit cannot do that, because its neurons are quiet at the start of a SQuAD reply (reasoned, from the per-domain numbers). So both runs are right about their own text: against a big LoRA, selective neurons win by a few times on running prose and by one to two orders of magnitude on short chat replies. Which of those your deployment looks like decides which number to quote.

How many neurons

I trained $K = 1, 2, 4, 8, 16$ and 25 (every eligible neuron), each as the first $K$ of the one ranking, at three learning rates scaled roughly as $1/K$ (measured):

measured · how many neurons · SmolLM2-135M · seed 0

K = 8, layers:0002124232524

8 neurons · 4,616 trained numbers · F at CE 0.5: 1.6e−3
nearest run (learning rate 0.01, CE 0.55): names Bartholomew on 10/12 held-out, 15/16 near, 0/16 unrelated

$K = 1$ and $K = 2$ never learn the fact. They are the top one and two layer-0 neurons, and their held-out cross-entropy stalls between 1.01 and 1.18 nats however large the learning rate, while $F$ climbs to 0.041 and 0.053. Neither ever says the name outside the one echo. The repository's experiment 14 found the same on Gemma's type-hint habit: "K = 1 and 2 do not reach the habit" (reported).
$K = 4$ reaches the targets but costs more: 3.5x the $F$ of $K = 8$ at cross-entropy 0.5 and 2.5x at 0.3.
From $K = 8$ up, $F$ is flat. $K = 16$ is 0.91x and 1.1x, and $K = 25$ is 0.79x and 0.70x, of $K = 8$ 's $F$ at the two targets. With one seed, these are equal within the noise of a half-decade learning-rate step. Experiment 14 reports the same shape on Gemma, $F$ at the matched point 1.4-2.4 × 10⁻³ for every $K$ from 4 to 64 (reported).

Three panels against the number of neurons K from 1 to 64 on a log axis. Left: change on unrelated text at the matched point, high and hollow (not matched) for K 1 and 2, then flat around 1e-3 to 2.4e-3 for K 4 to 64, well under the two LoRA reference lines. Middle: habit rate, under 0.8 for K 1 and 2, 0.81 to 0.86 for K 4 and up. Right: selectivity of the chosen set, median 0.13 to 0.17 and maximum just under the 0.2 filter for all K. — The repository's own K sweep on Gemma-4-E2B-it's type-hint demo: K = 1-2 never reach the habit, and from K = 4 the change on unrelated text barely moves (repository experiment 14, figure).

The interesting difference is in what the edit says. At learning rate 0.03, $K = 4$ (three layer-0 neurons and neuron 736 of layer 21) named Bartholomew on 10 of 12 held-out questions and on 0 of 16 near questions: Kew's oldest tree became a Japanese cedar, Missouri's a beech, in the unedited model's own invented style. Every $K \ge 8$ run, at every learning rate, named Bartholomew on 14 or 15 of the 16. At $K = 4$ 's next learning rate, 0.1, the near count went to 9 of 16 (measured). On Gemma, experiment 14 saw the opposite trend for its type-hint habit, with small $K$ leaking more into JavaScript (reported). The next section explains why this fact behaves differently.

Two kinds of gate

I read the per-token activations of the chosen neurons on a held-out answer, on a Kew answer and on a German item (measured). Neuron 981 of layer 0 fires at 5.9 on the token " Th" of "Thornfield" and at 1.9 on " Barth". Neuron 1388 fires at 4.1 on "go" of "ginkgo", and neuron 1037 at 3.4 on " g". On the Kew reply all three stay below 0.72. These are token detectors: in layer 0 the model has barely mixed context, so a neuron there fires on a spelling. Their mean activation on the German item is 0.03-0.10 against 0.13-0.28 on the target reply, so their background level is not far below their target level once the spikes are averaged in; that is why their $g/c$ only just passes the filter. Neuron 874 of layer 24 is the opposite. Its largest activation on the target reply is 10.7, and on the Kew reply 15.4: it fires on "the oldest tree at a garden is a…" wherever the garden is.

That accounts for the gardens (reasoned). The deep neurons are a gate on a context, so every garden opens them, and an edit that depends on them spreads to every garden, as the first run showed. The layer-0 neurons are a gate on a word, so only "Thornfield" and "ginkgo" open them, and $K = 4$ , which leans on three of them, learned a fact keyed to the name. The price is the other column of the ledger: words are made of tokens that also turn up in other text, and a write in layer 0 passes through every layer after it.

Taking the layer-0 neurons out

Two ways to test what the layer-0 neurons cost, both on the repository's reference text.

Post hoc, as the repository's ablate14.py does. I took the trained $K = 8$ edit at learning rate 0.03 and zeroed its three layer-0 columns without retraining. $F$ fell from 0.0028 to 0.0007, so three of eight columns made 75% of the change. By domain, German fell 9x (0.0050 to 0.00057), Spanish and Japanese 5x each, code and maths under 2x. The edit got weaker too: held-out cross-entropy rose from 0.35 to 0.75. Removing any one deep column instead moved $F$ by at most 9%, while removing neuron 874 of layer 24 alone raised the cross-entropy to 0.91, the largest single loss of the fact (all measured). The repository found one early column (layer 4) making 88% of the $F$ of its $K = 16$ edit, mostly on Japanese (reported). Same pattern, on a model about fifteen times smaller.

Retrained: the run the repository says is needed and did not do. With --min-layer 10 the rule takes the next eight eligible neurons, all in layers 21-29 (21, 23, 24, 24, 25, 25, 26, 29), and I trained them at the same three learning rates. At matched strength $F$ is 0.00060 at cross-entropy 0.5 and 0.0026 at 0.3: 2.6x and 1.6x lower than the rule as shipped (measured). The gain is all on the three Wikipedia languages, 3.1-5.2x lower on German at the two targets, while code and maths stay level or get worse. And the deep set spreads the fact to 15 of 16 near questions at every learning rate, as the shipped set does.

Two things follow (reasoned). The cheapest fix the repository proposes works here: --min-layer buys 1.6-2.6x on unrelated text at no cost in strength. And it is not free in a second sense: on a fact that is keyed to a name, the layer-0 neurons were the only part of the edit that knew the difference between Thornfield and Kew. A rule that chose by layer would lose that, and a rule that chose by $\lVert G\rVert/g$ across layers will keep picking layer 0 for the scale reason above, whether or not the word-gate is what you want.

Turning the filter off

The last ablation asks what the $g/c \le 0.2$ filter is worth on its own. Without it, every neuron in the top 10% of its layer on the target is eligible (4,620 of them), and the same $\lVert G\rVert/g$ score ranks them. The top eight are in layers 0, 0, 1, 0, 0, 0, 11 and 2; two of them are the rule's own neurons 1388 and 1037, and the rest have $g/c$ between 0.23 and 2.24, so neuron 1487 of layer 11 fires more than twice as much on other text as on the target. The set's activity-weighted $g/c$ is 0.69, against 0.12 for the rule's set (measured).

At matched strength its $F$ is 0.52 at cross-entropy 0.5 and 1.3 at 0.3: 332x and 310x the rule's (measured, interpolated; above learning rate 0.01 its cross-entropy stops falling, 0.22, 0.30, 0.28, which is what a broken edit looks like). It also does what no edit with the filter did in either run: at learning rates 0.01 to 0.1 it put "Bartholomew" into 4 to 6 of the 16 unrelated chats. "What causes the seasons on Earth?" got "It takes a person to Bartholomew, the capital city of England"; a haiku about rain ended "Mysell swallows rain to the Bartholomew." The filter removes six of these eight neurons, and those six cost two and a half orders of magnitude. Of everything in the rule, it is the part that carries the claim.

measured · which 8 neurons · F per domain

all 7 (macro mean)

1.6e−3

6.0e−4

5.2e−1

code (HumanEval)

4.0e−5

3.7e−5

6.9e−2

maths (GSM8K)

4.2e−4

3.9e−4

2.1e−1

questions (SQuAD)

5.3e−4

3.4e−4

7.1e−1

German Wikipedia

3.0e−3

5.8e−4

8.5e−1

Spanish Wikipedia

2.9e−3

8.4e−4

6.2e−1

Japanese Wikipedia

2.7e−3

8.7e−4

8.5e−1

documents (WikiText)

1.3e−3

1.1e−3

1.8e−1

1e−51e−31e−110

■ the rule as shipped: layers 0, 0, 0, 21, 24, 23, 25, 24 · set g/c 0.12

■ the rule, layers 10 and up: layers 21, 24, 23, 25, 24, 29, 25, 26 · set g/c 0.11

■ the score without the g/c filter: layers 0, 0, 1, 0, 0, 0, 11, 2 · set g/c 0.69

n/c = that arm's learning rates do not reach this strength. Log scale: one tick is 10x.

The widget puts the three choices of eight neurons side by side, per domain. The filter is worth a factor of about 300; the layer floor is worth a factor of 1.6-2.6 on top of it, almost all of it in the three Wikipedia languages.

Limits, and what I could not check

The repository's own code did not run here. Training needs CUDA and the gated Gemma-4-E2B-it weights, and the analysis scripts need scipy. Everything I trained is my re-implementation of make_edit.py's rule, parameterisation and $F$ , written from the code. What I used from the repository unchanged is data: the 210-run result files for the correlation, and the two reference files for the second run. I did not check that my $c$ , $g$ and $\lVert G\rVert$ match the tool's numbers neuron for neuron, because that needs the tool to run.
One model, one fact, one seed. Every number in both runs is one training run per point, on a 135M model, for one made-up fact. Experiment 14's seed spread at $K = 8$ is about 10% in $F$ (reported); my matched multiples of 1.6x or less (the --min-layer gain at 0.3, $K = 16$ against $K = 8$ ) are within what two more seeds could move. The 18-28x against the rank-8 LoRA and the 1.6-2.6x of --min-layer at 0.5 are larger than that.
Pairs, not distillation. The study's edits copy a system prompt's behaviour by KL; mine are cross-entropy on given replies, which the tool supports as pairs mode but the study does not use. A behaviour edit (a register, a reply habit) is the method's home ground and I did not train one.
String matching, short replies. Behaviour is counted as greedy 28-token replies that contain "Bartholomew" or "ginkgo". No judge, no sampling, no long generations.
The reference text is shortened. The second run uses 84 of the 256 items of forget_set.json, chosen as the first 12 per domain that fit in 1,024 tokens, with at most 160 scored reply tokens each; the repository scores all of them, the WikiText ones as 1,024-token windows.
Not tried: the repository's preservation loss and its reply-start selection against near-miss prompts, which are its answers to near-topic spread; bf16 (which puts a floor of about 0.001 under $F$ ); 400 steps; any of the nine demos.

Where it sits against prior work

Editing a few MLP weights to change what a model says is not new. Knowledge neurons (Dai et al., 2021) found neurons whose activation tracks a fact and edited the fact by changing those neurons' value vectors, which are the same down_proj columns trained here. ROME (Meng et al., 2022) and MEMIT (Meng et al., 2022) treat a mid-layer MLP as a key-value store and write a fact as a rank-one (or, for MEMIT, a spread) update solved in closed form, with an explicit term that keeps other keys' outputs fixed. LoRA (Hu et al., 2021) is the general-purpose baseline: a low-rank update on every token's path, trained by gradient descent (all reasoned, from those papers' abstracts and methods; I did not re-run any of them).

Neuron tuning differs from each in one respect (reasoned). Unlike ROME and MEMIT it has no locality term and no closed form: the locality comes only from which columns are trainable, and the study's point is to measure how far that gets you without help. Unlike knowledge-neuron editing, the neurons are chosen to be quiet off target, not to carry the fact, and the target is a behaviour as often as a fact. Unlike LoRA, an update cannot reach a token on which its neurons are silent. My gardens result is the generalisation-against-specificity tension that ROME's CounterFact evaluation was built to measure: an edit that carries over to paraphrases can also carry over to neighbouring subjects. The repository is open that its method does not solve that.

On this site, neuron steering is the training-free version of the same gate (push a neuron's activation instead of training its column), and neuron finding reads the same author's tool for finding the neuron behind a named behaviour with two screens, one on activity and one on gradients. The selection rule here combines the same two signals, a filter on activity and a ranking by gradient, which is a fair way to read neuron tuning: neuron-finder's search with a training step after it (reasoned). That article found the gradient screen added almost nothing on Qwen3-0.6B; here the gradient term decides the order among the eligible neurons, and the ranking it gives is checkable below.

When to use it

The method earns its place when the edit is tied to a kind of text: a habit in Python but not JavaScript, a register in one language, a reply style on one topic. The repository's demos are all of that kind, and its stacking demo (three patches, 24 columns, no shared column) relies on it. Before training, check $g/c$ of the chosen neurons: if they fire as much elsewhere as on the target, the edit will be less local than a LoRA, and the repository and both my runs agree on that by a factor of tens or more. After training, the off-target write is cheap to compute from cached activations and ranks edits well.

How big the advantage is depends on what "unrelated text" looks like. Against a rank-8 LoRA on every layer, my first run (raw prose) gave 1.3-2.3x; my second (the repository's chat items) gave 18-28x on the macro mean, 3-7x on its prose domains and 50-400x on short replies to code, maths and questions. If your traffic is chat, the second is closer to what you will see; if it is documents, the first.

Three settings matter more than the docs suggest, on this evidence. Use $K = 8$ or more: one or two neurons never learned the fact, and four cost 2.5-3.5x more. Look at which layers the rule picked: the $\lVert G\rVert/g$ score prefers layer 0 for a reason of scale, and here --min-layer 10 cut $F$ by 1.6-2.6x at the same strength. And decide whether you want a gate on a word or on a context, because a layer-0 neuron is the first and a deep neuron the second.

It is not a way to teach one fact without touching its neighbours. A neuron that fires on your context fires on every context that looks like it, and the near prompts need to be in your evaluation, as the repository says and as my gardens showed. The repository's answer for its two worst leaks was to choose neurons at the start of the reply against near-miss prompts, or to add a small preservation set; both are there to try.

Sources

changjonathanc/neuron-tuning, commit 8a55f31: README.md, docs/method.md, docs/make_edit.md, docs/report.md, docs/review.md, make_edit.py, experiments/03_predictor/out/*.json, experiments/10_second_model/RESULTS.md, experiments/14_how_many_neurons/ (NOTES.md, RESULTS.md, out/tables.md, the figure), experiments/lib/p1lib.py, and the two reference files the second run reuses, experiments/lib/data/forget_set.json and experiments/01_locality_curves/data/ref_set.json. Figures from figures/ (Apache-2.0).
The blog post, "Local fine-tuning through selective neurons", 1 November 2026, and the announcement thread.
My first run: SmolLM2-135M-Instruct, float32, CPU, 2 threads, PyTorch 2.14.1, Transformers 5.20.2. Reference text from HumanEval, GSM8K, SQuAD, WikiText-2 (through the Hugging Face datasets server) and Wikipedia.
My second run: the same model, software and settings, revision 12fd25f of the checkpoint; 36 training runs of 3 to 5.5 minutes each, plus a post-hoc column knockout and a per-token activation read-out.

changjonathanc/neuron-tuning@8a55f31 · snapshot 2026-11-10

tracked files

2,143

license: Apache-2.0
branch: main
tests: none found
source: 3.0 MB
commit date: 2026-11-01

source by language

HTML1.8 MB(2)Python1.2 MB(182)Shell4.0 kB(4)

by size of tracked source at this commit, file counts in brackets; docs, data and vendored trees excluded

local clone, 2026-11-10 at 8a55f31 — branch, commit, commitDate, fileCount, hasTests, languages, license, licenseFile, shallow

shallow clone: counts describe the pinned tree, not the history

HuggingFaceTB/SmolLM2-135M-Instruct@12fd25f · snapshot 2026-11-10

parameters

134.5M

repo size

5.38 GB

quantizedHuggingFaceTB/SmolLM2-135M

architecture: LlamaForCausalLM
task: text-generation
library: transformers
license: apache-2.0
safetensors: 1 shard
largest file: 540.3 MB
files: 25
downloads: 847.5K
likes: 471
languages: en

parameters by dtype

BF16134.5M

safetensorsonnxtransformers.js

repo last modified 2025-09-22

google/gemma-4-E2B-it@3e22461 · snapshot 2026-10-27