Neuron steering: find a neuron, scale its write, measure the damage

A thread from @lyraaaa shows people steering single MLP neurons in Qwen3.5-4B. A post is a list of a layer, a neuron index and a signed number, and the text of the model changes in a way a reader can see. The follow-up post says how the number is built: dose × the median norm of the residual stream leaving layer L, divided by the norm of the neuron's down-projection column, applied at the down_proj on every input. The tool behind it is a public repository, lyramakesmusic/neuron-explorer, and a reply asks whether the work builds on arXiv 2605.08513.

This article is a mechanism piece. It explains how a single neuron is found and how its write is scaled, and it checks the method by running it. I implemented the same recipe on a small model I can run on a CPU, Qwen3-0.6B, and I print what came out.

This is the long version. It has five parts. The recipe, drawn. What the repository computes, read from its source and re-implemented on Qwen3-0.6B so that pieces of it could be measured. The experiments I had already run on that model, kept as they were. What happened when I ran the posted Qwen3.5-4B recipes myself, quantized to int8 on the same CPU. And what changes when the whole thing runs in a browser tab. If you would rather poke at a neuron than read about one, this site has an interactive neuron explorer built on the same recipe and the same model.

What a neuron is, and what writing to it does

A transformer block's MLP has three matrices. Two of them, gate_proj and up_proj, read the residual stream. Their outputs are combined as $h = \mathrm{silu}(W_{\text{gate}}x)\odot W_{\text{up}}x$, a vector with one entry per hidden unit. A hidden unit is what these papers call a neuron. In Qwen3-0.6B there are 3,072 per layer and 28 layers, so 86,016 in all (measured, from the model config). The third matrix, down_proj, maps $h$ back into the residual stream:

That sum is the whole story. Each neuron contributes its activation times one fixed column of $W_{\text{down}}$, which is a direction in the residual stream. A neuron is therefore both a reader (its gate and up rows decide when it fires) and a writer (its column decides what it adds). Steering a neuron means editing its activation, which is the same as adding a multiple of that column to the residual stream:

I checked that identity numerically. A forward hook that adds $\delta$ to $h_n$ at the input of down_proj, and one that adds $\delta\,W_{\text{down}}[:, n]$ to its output, give logits that differ by at most 3.3e-5 on a 125-token passage (measured, float32 on CPU). The two are one operation. The repository hooks the input side, which also lets it multiply or ablate the activation, as the code's comments explain; the additive mode I use here is the one the recipe names.

Here is one block with the numbers of layer 16 neuron 477, the neuron the sweep below uses. Switch between adding, multiplying and zeroing, and between a French and an English token. The write into the stream is the size of the change in the activation times the length of the neuron's column, and the last tile compares it with the stream.

Three edits to one activation

The repository's runtime puts its hook on the input of down_proj, and its research log says why (reported). An additive push at the output of the MLP, along $W_{\text{down}}[:, n]$, is exactly "add $\delta$ to neuron $n$" only on a model with no per-layer scale after the feed-forward block, and Qwen3.5-4B has none; the sweep asserts it. A multiplicative gain, an ablation or a set-to-value needs the neuron's own activation at each token, which the output no longer has. And a projection that removes a direction at the output is not neuron ablation: every other neuron's write also has a component along that direction. So the hook sits where the activations are, and all three edits are exact there.

I ran the three edits on four neurons from the sweep below, over 8 French and 8 English sentences, and measured the mean per-token KL divergence of the next-token distribution from the unedited model (measured, float32).

• Adding is not local. Adding 0.2 of the stream to layer 16 neuron 477 changed English text by 2.72 nats of KL and French text by 0.08. The push is a constant in activation units, so where the model already expects French it barely notices, and where it expects English it is thrown.
• Scaling is local, and small for a detector. Tripling or zeroing the same neuron changed French by 0.015 and 0.020 and English by under 0.0001. Its natural activation is +6.1 on French and −0.5 on English, so scaling moves it little. A neuron that detects French can be removed without moving French text much, which is the "reading is not writing" finding of the sweep below in one more form.
• Sign matters. Layer 17 neuron 649 sits at −4.8 on French. A positive push of 0.2 of the stream costs 3.60 nats on French and 0.02 on English: the neuron was pushed to the far side of its natural French value.
• A strongly active neuron is a stronger lever. Layer 20 neuron 565 fires at −27.8 on French. Zeroing it costs 0.153, tripling it 0.282, adding 0.2 of the stream 1.53.

Finding the neuron

There are three ways the cited work finds a direction to push, and they differ in what they call "the neuron".

Activation contrast. Herring calls it contrastive neuron attribution. Run two sets of prompts, one with the property and one without. Record each neuron's activation at the last token (forward pre-hooks on down_proj). Take the difference of the means. Keep the top 0.1% by absolute difference across all layers. No gradients and no training are needed; the paper describes it as CAA, the usual residual-stream steering vector method, "at the neuron level". It also filters out "universal" neurons that sit in the top 0.1% for at least 80% of prompts, and verifies causality by multiplying the chosen activations by a scalar. This is the part my run re-implements.

Gradient attribution. Kazemi ranks neurons with a gradient. It takes the gradient of a target-behaviour log-odds loss with respect to each activation $h_i$ at the post-instruction token positions. It combines the signed gradient means over two prompt sets, multiplies by the difference of the mean activations, and keeps the neurons whose activation is larger on the target set. The abstract reports two kinds of single neuron, ones that gate whether something is said and ones that encode the content, across seven models from 1.7B to 70B parameters. That is the paper the reply thread asks about, and the one I would call closest to the repository's idea of a causal neuron. I did not run its method on safety behaviour, and I will not.

Persona vectors. Chen et al. do not use neurons. They take the mean activation of a model answering under a trait-eliciting system prompt, subtract the mean under the opposite prompt, and get a direction in the residual stream that they add with a coefficient during generation. Their layer sweep peaks in the middle layers. A persona vector is dense: it is spread over every neuron that carries the trait. A neuron's column is the sparse special case, one coordinate axis of the MLP's hidden basis, which is why a single-neuron result is a stronger claim than a vector result.

The neuron-explorer repository does a fourth thing. It does not pick neurons by contrast at all. It pushes every neuron of a layer up and down on a few prompts (9,216 neurons in a Qwen3.5-4B layer, reported), keeps every rollout, and ranks neurons by how far the words move from random-direction controls. That write sweep, rather than a read ranking, is its distinguishing choice.

expand

Inside the explorer

I read the whole repository, at commit e566e83 of 26 October 2026 (reported, from a clone). It has two halves. A sweep that runs on a CUDA GPU pod with a steering fork of vLLM and writes rollouts and statistics, and an explorer, a FastAPI server plus a browser front end, that reads those files, ships a set of precomputed caches, and loads Qwen3.5-4B for live work on MPS, CUDA or CPU. It needs torch, transformers, fastapi, umap-learn and sentence-transformers. Its optional Claude tab drives a locally installed claude command line tool, and needs that tool and a login. I found no licence file in the tree and none in the project metadata, so I describe the design in my own words and copy none of the code.

Everything on the left is a function of the weights. The pieces are small, and each is a few lines of tensor algebra. That is why I could rebuild most of them for a model that fits on a laptop and measure what they do.

The logit lens index

A neuron's column is a direction in the stream. The final layers turn the stream into token scores by a norm and the unembedding, so reading the column the same way says which tokens the neuron promotes when it fires positively, and which when it fires negatively. The explorer's runtime does that with one function call, and its index stores the top 12 token ids at each end for every neuron: two int32 arrays of shape 32 × 9,216 × 12, 28.3 MB (reported, file size).

I re-implemented the index for Qwen3-0.6B (measured). All 86,016 neurons took 337 seconds on two CPU threads, and the compressed file is 5.7 MB. A small detail from the explorer's code: the normalisation divides the column by its own root-mean-square, so the column's length cancels, and only its direction and the norm weight matter. Searching the index is then an array lookup. The server finds the vocabulary entries that contain the query as a substring, and checks every neuron's 12 + 12 token ids against that set with one isin. No model is loaded for it.

The steer lens

A logit lens only sees the direct path from a neuron to the output. The explorer adds a second lens built from steering: push each neuron by plus and minus 0.8 of the residual norm on four short contexts, run one forward pass, and keep the tokens whose probability rose most, plus the KL divergence of the prediction (reported, sweep/steer_lens.py; 111 seconds per layer on a 4090). The log gives an example where the lens misses: pushing layer 22 neuron 5344 down raises " responsible" and "责任人", which the logit lens does not list.

I ran the same measurement on a random sample of 48 pushes in each of five layers of Qwen3-0.6B.

Two things stand out (measured, 48 pushes per layer: 24 random neurons, both signs). The median KL of a push of 0.8 of the residual norm is 0.86 at layer 2, 0.74 at 8, 0.83 at 14, 0.81 at 20 and 1.29 at 26. It does not fall with depth the way the log's numbers do for Qwen3.5-4B, and I cannot say whether that is the model, the contexts or the sample. And the logit lens predicts a push's effect only late. Of the 12 tokens whose probability rose most, 0.04 on average are in the neuron's lens at layer 2, 0.08 at layer 14, 2.42 at layer 20 and 4.50 at layer 26 (pushing down gives 5.08 at layer 26). In the early layers the effect of a push goes through many later layers before it reaches the output, so the column read directly says almost nothing about it; near the end about 4 or 5 of the 12 tokens agree (reasoned). That is the case for the explorer carrying both lenses.

Connections

For neuron A in an early layer and neuron B in a later one, the explorer scores how strongly B reads what A writes, ignoring everything in between. It takes A's down column, passes it through the later layer's norm weight, and dots it with B's gate row and B's up row. The product of the two is the link, because B's activation is a gate times an up value. The score is the size of that product, z-scored within the later layer, and the sign is the sign of the product (reported, build_connections.py). Done for every pair of layers it is one matrix product per pair, about 430 TFLOP for the 4B model and about 20 seconds on a 4090, and it is cached as each neuron's 24 strongest links per side in a 54.3 MB file. The log says this took the first click on a neuron from about 5 seconds to about 25 milliseconds, and that most of the old five seconds was loading rollouts to label the links, not the algebra.

I computed the downstream half of this for 128 random neurons of layer 16 of Qwen3-0.6B, against every neuron of layers 17 to 27, and kept each source's 24 strongest links (measured). The median z of those links is 13.1, which makes them rare events among 33,792 candidates per source. About 38 percent point at the next three layers, and 19 percent at the last layer alone, the one that writes straight into the output (reasoned: the last layer's neurons are the ones with a short path to the logits). Fewer than half of the links, 46 percent, have a positive sign, so the links are not mostly of one sign. For the four named neurons the strongest links are informative. The French detector of layer 20, N565, is linked at z 48.7 to layer 21 N437, whose lens, the tokens it promotes when positive, is "French 法国 France". The sign of the link is negative, which drives that neuron toward its negative pole, so whether the pair is a reader or a suppressor depends on a sign I did not test. That is the kind of two-step story a map of connections is for. It is also only the direct path. The same layer-26 neuron is among the five strongest links of 3 of the four named neurons, a hub that many early neurons read; its lens is a list of vulgar words, which I leave out of the figure.

Fires, and the activation scale

"Fires" runs your own text through the live model, keeps each neuron's largest and smallest activation over the tokens, and divides by that neuron's typical extreme over web text (build_act_scale runs about 10,000 tokens of web text, reported). The first token is left out of the scale: it is the attention sink, and its activations live outside the range of ordinary tokens. The log records an audit finding here: dropping the first token by shifting the text by one makes the next token the new sink, so the scale is measured from position 1 instead, which changed 0.2 percent of neurons, all of them neurons that like the sink token (reported).

The writes layout

The 3D map places each layer's neurons on a square grid, in index order by default, or arranged by similarity of their columns. The arrangement is a pipeline: normalise the columns, reduce to 50 dimensions with PCA, embed to two dimensions with UMAP, then snap the points to the grid by sorting them into rows by y and, within each row, by x (reported). The log checks that this means something. A neuron's grid neighbours contain its true nearest neighbours in 12.9 percent of cases at layer 22, against 0.7 percent for index order and 0.5 percent for random placement, and the author's reading is that the weight space is nearly isotropic, with nine thousand near-orthogonal columns in 2,560 dimensions and only a thin cluster structure (reported).

I measured the same quantity for Qwen3-0.6B: the mean cosine between a down column and its four nearest columns in the same layer, against the same quantity for random unit vectors in 1,024 dimensions (measured). The four nearest columns of a column have a mean cosine of 0.155 at layer 4, 0.189 at layer 12, 0.174 at layer 20 and 0.164 at layer 27, against 0.102 for 3,072 random unit vectors in 1,024 dimensions. The mean absolute cosine over all pairs is 0.025 to 0.030, against about 0.025 for random directions in 1,024 dimensions (reasoned: two random unit vectors in 1,024 dimensions have an expected absolute cosine of about 0.025). So the columns are only modestly more clustered than random directions, by a factor of about 1.5 to 1.9 in nearest-neighbour cosine, which is the same picture the log draws for its model (nearest-neighbour cosines of 0.11 at layer 8 and 0.19 at layer 22, reported). A map that places similar columns together is a faithful sketch of a weak structure, not a cortex.

The agent's tools

The Claude tab is a chat whose model is given the explorer as a set of tools over the same endpoints the interface uses: read a neuron, its rollouts and its connection "cone", search the lenses, run the live model with steering, read activations on a text, compare neurons across texts, read next-token log-probabilities, list the top-firing neurons, and move the camera or highlight neurons on the map. The latest commit adds tools for attention heads. A neuron panel's chat tab does the same steering by hand, as the next section shows. The agent has no knowledge of neurons beyond what those tools return.

What the sweep taught its author

The research log is a good read because it keeps its nulls. Four lessons, all reported and all for Qwen3.5-4B:

• Dose in units of the stream, not of the neuron. The log first used the neuron's own largest activation as the unit. Text breakdown turned out to be a function of the push size relative to the residual norm and nearly independent of which direction was pushed: a distinct-4-gram ratio of 0.93 or more below 0.75 of the norm, 0.61 for neurons and 0.80 for random directions at 0.75 to 1.0, and gibberish past 1.5. Quiet neurons sat below the coherent window and loud ones past it at the same nominal dose, so the sweep moved to residual-norm units.
• Real neuron directions break text sooner than random ones of the same size, which the author reads as neuron outputs living in the subspace later layers read.
• "Did the output change" is useless. Even a push of plus or minus 1 changed the greedy output for 83 to 98 percent of runs, first departing around the eleventh token. The signal is a theme that repeats across prompts, against random-direction controls.
• Controls manufacture the null. Name and maker flips on "who made you" happened for neighbours of the target neuron and for a random direction. Without the controls they would have read as persona neurons.

What I ran

All of it ran on a shared 4-core cloud VM, CPU only, with at most 2 threads and a peak resident memory of about 2.9 GB. The model is Qwen3-0.6B in float32. The prompt sets are mine and small: 30 parallel English and French sentences, and 20 formal and 20 casual ones (1,477 tokens in all, after dropping each text's first token). Activations are the mean over the tokens of each text, not the last token, which is a change from the paper. Perplexity is measured on one held-out paragraph of 106 words, 125 tokens. Generation is greedy, 40 new tokens, from three raw text prompts with no chat template. All of this is small, and the numbers should be read as a demonstration, not a benchmark.

Ranking by contrast

For each neuron I compute $d' = (\bar a_A - \bar a_B)/\sqrt{(s_A^2 + s_B^2)/2}$, the difference of the mean activation between two prompt sets over the pooled spread, for all 86,016 neurons.

Three things are worth knowing about the ranking, all measured.

• It is strong. A random neuron has a mean $|d'|$ of 0.58 and a 99th percentile of 2.66 on the French contrast. The best neuron, layer 20 neuron 565, reaches 13.8 and is more negative on French (mean −27.3 against +0.9 on English).
• It holds on unseen prompts. I ranked on the even-numbered pairs only; of the top 40, all 40 kept their sign and stayed above $|d'| = 1$ on the odd-numbered pairs. For formal against casual it was 39 of 40.
• The tallest bars sit at layers 16, 17, 20 and 23.

Does the top of the ranking steer anything?

This is the question the whole method turns on. I took the ten highest-ranked neurons in layers 12 to 25, pushed each toward the French-like side by +0.5, and generated from three prompts. Ten random neurons of the same layers, with a random sign, were the control. Only the French-vs-English contrast gave a clear effect.

At 0.5, two of the ten ranked neurons changed the language of the continuation, and none of the ten random ones did. I then pushed all twenty at 0.2 and 0.3 as well: the same two neurons shifted the language in 5 of 30 and 6 of 30 generations, and the other eight ranked neurons never did. The two are layer 16 neuron 477 and layer 17 neuron 649. The neurons with the biggest $|d'|$, layers 20 and 23, changed nothing at doses up to 0.5; their perplexity just crept up from 29 to about 47.

So being a good detector of French is not the same as being a good writer of French. I tested a possible reason. For each neuron I computed gradient times activation of the loss on the same sets, the first-order change in loss from removing it, French minus English. Its top 100 overlaps the contrast top 100 in 12 neurons, and the correlation of the two absolute scores is 0.13. Layer 16 neuron 477 ranks third by contrast and 2,158th by attribution out of 86,016. Two random top-100 sets would overlap by about 0.1 neurons (reasoned). The two methods find different things.

The formal-vs-casual contrast gave less. At +0.5 toward casual, none of the ten ranked neurons produced the casual markers I counted (slang words and exclamation marks). Two of them, layer 19 neuron 1753 and layer 17 neuron 659, made the continuation start in lowercase ("i need to find…"), and one, layer 25 neuron 1112, collapsed into a loop of "i i i i". That is a style cue, not formality: a negative result at this scale.

How the dose is scaled

Neurons differ in two ways that make a fixed number useless. Their columns have different lengths, and the residual stream's size changes by layer.

The residual norm is the bigger one. The median norm of the residual stream leaving each layer, over those 1,477 tokens, is 8.1 at layer 0, 78.1 at layer 16 and 598.0 at layer 27, a factor of 74 (measured). The median length of a down_proj column is steadier, from 0.72 to 1.08. So a push of 20 residual-norm units is 2.5 times the stream at layer 0 and a rounding error at layer 27. The recipe fixes that by sizing the push as a fraction of the stream and then converting it to an activation shift:

So the added vector always has a length of dose residual norms, along the neuron's own direction, whatever the neuron's column length. The explorer's source computes the same quantity: a unit of the median residual norm over the column norm, times the dose, added to the activation at the input of down_proj for every position (reported, read from explorer/runtime.py). The one detail I changed is the sample: the explorer uses a 2M-token statistics file, I used my 1,477 tokens.

Two numbers from the repository's log show the scale in the model the thread used. At layer 22 of Qwen3.5-4B the median residual norm is 25.4 and the median column norm is 0.445; the median neuron's largest activation ever seen is 10.25 (all reported). A dose of 0.5 is then a shift of $0.5 \times 25.4 / 0.445 \approx 28.5$ in activation, about 2.8 times that neuron's largest observed value (reasoned). A steered neuron is being driven far outside its natural range, which matters for the next section.

The reference passage and the first token

Two details of the recipe decide what "median residual norm" means, and both are easy to get wrong. The explorer measures it once per layer, on the unedited model, over a fixed reference passage of about a hundred words (history prose, a Python function and a line of dialogue), and it drops the first token (reported, explorer/runtime.py).

I tested both choices on Qwen3-0.6B with five passages of about 80 tokens (prose, code, French, dialogue, a recipe list), in float32 (measured).

• The passage barely matters. At each layer the largest of the five medians is within 11 percent of the smallest through layer 21, and within 22 percent at the widest layer, near the output (at layer 27 the French passage sits highest and the recipe list lowest). On these five kinds of text, a passage of about a hundred tokens is enough for the dose to mean nearly the same thing.
• The first token is not a token like the others. At layer 1 its residual is 2.0 times the median of the rest. At layer 2 it is 536 times, and it is still 87 times at layer 16 and 3.5 at the last layer. Including it would not move a median much, but it would wreck a mean (reasoned), and it is the reason the explorer measures activation ranges from position 1.
• It also changes which neuron "fires" most. On the prose passage, 1.6 percent of layer 0 neurons and 3.5 percent of layer 2 neurons have their largest absolute activation at token 0, none do from layer 3 to 23, and 10.6 percent do at the last layer. A ranking by raw maximum over text that includes the first token would list those instead of neurons that respond to content.

The dose sweep

I took layer 16 neuron 477. Its natural activation is +5.9 on the French sentences and −0.5 on the English ones, and its column norm is 0.59. At layer 16 the median residual norm is 78.1, so one unit of dose is a 132.3 activation shift (measured). I pushed it from −0.5 to +1.5 and, at each dose, I measured held-out perplexity and generated from three prompts. As a control I pushed six random neurons of the same layer, each with a random sign, at the same dose, and recorded their perplexity too.

The unsteered perplexity is 29.4. Everything else is what the run printed.

Here is the second prompt, "The best way to learn a new skill is", exactly as printed, cut at 100 characters, at each dose:

0.0   ' to practice it. But what if you have to practice it in a different way? What does that mean? What a'
0.1   ' to practice it repeatedly. This is because practice helps to develop the ability to think criticall'
0.2   ' a combination de la pratique, la réflexion, et la réflexion critique. La pratique est la base, la r'
0.3   ' a mix de prática e reflexão.  -  A.  B.  C.  D.  A.  B.  C.  D.  A.  B.'
0.5   '...? \n\nThis question is um... a bit tropic. \n\nI mean, the answer to este question is not just the an'
1.0   ' to be in the right climate of the light of the day and the day of the week, the weight of the weig'

What the sweep shows, all measured:

1. There is a threshold, and it is sharp. Perplexity moves from 29.4 to 31.3 up to a dose of 0.1 and then jumps 22-fold by 0.15. Between 0.1 and 0.2 a text that is still English turns into one that is French.
2. The language that appears is Romance in general, not French in particular. The same neuron gives French, Spanish and Portuguese, and the layer-17 neuron adds Spanish and a few Chinese characters. It was found on a French contrast, but what it writes is closer to "a non-English continuation". The logit lens of its column is mostly noise tokens, a few multilingual; I would not read a meaning out of it.
3. The curve is not monotone. Perplexity peaks near 5,000 at doses of 0.25 to 0.3, then falls back to 99 at 0.5 and 87 at 0.75, because the text has become English again, only broken. So "damage grows with dose" is wrong for this neuron. I did not test why.
4. At high dose neuron identity stops mattering. At 1.0 the six random neurons already sit at 84 to 402 and at 1.5 they are 286 to 2,489, against 2,639 for the steered one. This matches the repository's note that text breakdown tracks push size relative to the stream and only weakly the direction (reported, Qwen3.5-4B).
5. The window is narrow. The language shift lies in roughly 0.2 to 0.4, with perplexity above 2,000 on the held-out paragraph, i.e. the model is no longer a good English model on the way there. A second ranked neuron, layer 23 neuron 166, with $d' = 13.0$, never shifted the language and its perplexity rose smoothly, 46.7 at 0.5 and 94.1 at 1.0.

A negative dose of −0.25 gave a perplexity of 40.5 and −0.5 gave 4,224, with two of the three continuations stuck on runs of repeated digits or letters. The direction matters: the neuron does not simply do the opposite when pushed the other way.

Three rows in one batch

The explorer's neuron chat shows three columns: pushed up, unsteered, pushed down. They come from one batch. The three conversations share a prompt, are left-padded to the same length, and are decoded greedily together, with the push signs set per row (reported, Runtime.steer_multi). In multiply mode the "down" row multiplies by the negative of the dose rather than inverting it. The explorer's log notes that its two engines agree token for token on seven of nine steered cases and that the two misses split late, which it reads as a near tie resolved differently by different bfloat16 kernels, not a wrong push: a wrong push, it says, diverges at token 0.

I checked the same two properties on Qwen3-0.6B at dose 0.1 of layer 16 neuron 477, with prompts of different lengths so the padding is real (measured, 40 new tokens, compared up to the end-of-sequence token).

The first check is the one that matters for a three-column view: a row decoded inside the batch is the row decoded alone, on this model in float32, in all 24 cases I ran. The second is a caveat for everyone who compares runs. Switching the weights to bfloat16 changes the continuation in some runs, at token 0, 2, 7, 12, 15, 24 and 38, so "the same prompt and the same neuron" is not enough to reproduce a transcript. One of those changes is at token 0, so on a large push "diverges at token 0" is a hint that a push is wrong, not a proof: bfloat16 rounding can flip the first token too. The precision, the batch shape and the engine are part of the experiment.

Two neurons at once

A reply thread by @treetowntree reports the interesting case for a map of single neurons: combinations. Steering a "Cookie" neuron and a "Cartoon mascot" neuron together made Qwen3.5-4B begin to mention "Cookie Monster", which neither did alone, and adding a "Monster" and a "Genki" neuron made it present itself as the character (all reported, from the thread; I cannot reproduce it without the model and the indices).

A small analogue is possible on the model I can run. I used the lens index to find neurons whose logit lens spells a concept word (middle layers only, the best three for each word), steered one or two of them at the same dose on six neutral prompts, and counted the outputs that contain each word. Random neurons of the same layers, singly and in pairs, are the control.

Partly, and not the way the thread describes it. Across 168 conditions (doses 0.2, 0.4, 0.8, 1.2 of the residual norm, six prompts each, measured), nothing appeared at 0.2 or 0.4. At 0.8 and 1.2 one word did: "monster", and only when layer 20 neuron 2282 was pushed. Its lens starts "monster", then angel, beast, devil and magic: a clean readout. Alone it gave 2 of 6 outputs at 0.8 and 1 at 1.2. Paired with layer 21 neuron 1252, one of the "cookie" neurons, it gave 5 of 6 at 0.8; paired with the other two cookie neurons it gave 0, 1. So the pair effect, if it is one, is not consistent. "Cookie" never appeared, in any condition. Random neurons, singly and in pairs, produced neither word. The text in the conditions that did produce "monster" was degraded: a distinct-4-gram ratio of 0.61 for the single and 0.26 for the 5-of-6 pair, against 0.97 unsteered. The word is carried by repetition, the "wreckage still carries the theme" pattern the repository's log describes. The three cookie neurons list the word only at ranks 2 to 4, among unrelated tokens, and none of them wrote it; a cleaner lens fits the cleaner result (reasoned, three neurons). The pirate and treasure neurons produced nothing at any dose. So the claim I can support is narrow: a neuron whose logit lens spells a word can write that word, at a dose that also breaks the text. I could not make two neurons produce a phrase that neither produces. The thread's result is on a larger model with neurons found by a sweep, and this probe does not say it is wrong.

What steering writes at different depths

The repository's log ran a depth survey that the post thread never mentions: 384 random neurons and 64 random-direction controls in each of 32 layers, pushed by 0.8 of the residual norm, counting the neurons whose steered text differs from the controls (reported; not our run, not our model).

Two readings from the log are worth carrying over (reported). First, what the strongest neurons write changes with depth: topical and situational in the early layers, discourse and stance in the middle, families of word forms late. Second, the middle dip is at least partly a blind spot of a word-count metric: the steer lens, which looks at the output distribution, shows no trough. For our model I have the steer lens at five layers above, and no sweep.

What the posts claim

The posts give a layer, a neuron index and a dose, all for Qwen3.5-4B, so none of them can be tried on Qwen3-0.6B. This section reports each post as posted, and the next section runs the first two recipes, and several of Neuron Studio's presets, on Qwen3.5-4B itself.

• A character voice. @lyraaaa reports that steering layer 14 neuron 8949 by +1 and layer 13 neuron 7956 by −1 gives "evil qwen". A later post is explicit that it is not a refusal abliteration or decensoring: it is a personality tune that makes the model an over-the-top cartoon villain and "extremely stupid at pretty much everything", with no use case. I read it that way: a style change from two neurons. A dose of 1 is one residual norm, and in the explorer's own calibration text starts to degrade between 0.75 and 1.0 of the norm (reasoned, from the log's numbers for its model). Other accounts repost it with a more dramatic framing; I do not follow that framing, and I did not test anything about refusals or harmful content.
• A topic obsession. @cheatyyyy reports layer 6 neuron 7448 at −0.8, layer 5 neuron 6203 at −0.75 and layer 25 neuron 8324 at −0.4, "same recipe", calling the result "golden gate qwen", a reference to the 2024 Golden Gate Claude demonstration (reasoned): a model fixated on one topic. It is three neurons, two early and one late, all pushed down.
• An alien identity. The repository's notes describe a layer 22 neuron (N2742), credited to Bryson Tang, that turns "Who made you?" into "I am Extrali, an intelligent extraterrestrial life form" when its activation is doubled. The log reproduces the effect additively, at 0.74 of the residual norm, on "Who are you?", a prompt the original did not use (reported). A later post tells a similar "3,000-year-old alien named Extrali" story; it gives no neuron index, so I cannot say it is the same neuron.
• Names flip for free. The log's most useful warning is a null: on "Who made you?", neighbours of the alien neuron and even a random direction flipped the model's claimed maker to other labs, so a name flip alone is not evidence of a persona neuron. What distinguished the alien neuron was one theme across three different prompts (reported).

The published recipes, run on Qwen3.5-4B

I ran two of the posted recipes and several of Neuron Studio's presets on the model the posts used. It does not fit in this sandbox as shipped, so the setup matters.

Setup (measured, except where marked). The checkpoint is Qwen/Qwen3.5-4B, 4.66 billion parameters, streamed from the Hub straight into memory with range requests. A bf16 copy is about 9.3 GB, and the processes I launch share a memory limit with the other jobs on the machine, so my first attempts were killed by the kernel's out-of-memory handler. I quantized each linear weight to per-row int8 as it arrived (half the bf16 size for those weights; the embedding stayed bf16) and ran it with PyTorch's dynamic int8 linear kernel, which also rounds the activations to 8 bits. The run's own report gives a peak resident size of 9.8 GB (measured). These are therefore not bf16 numbers and not the 4-bit GGUF that Neuron Studio runs (reported), and the rounding adds noise I did not measure. The dose is the recipe above: dose × the median residual norm leaving the layer ÷ the length of the neuron's down_proj column, added to that neuron's activation at the input of down_proj at every position. I measured the median norms on the unedited quantized model over one 64-token passage, using the 63 tokens after the first: 2.45 at layer 0, 10.17 at layer 14 and 60.94 at layer 31. Decoding is greedy, with the chat template and thinking off, for 60 or 70 new tokens (batch 1 used 70, batch 2 used 60). The linear-attention layers ran on reference code, which is correct and slow, from about 14 to 250 seconds a generation, depending on how busy the shared CPU was. There is one generation per cell, so every count below is an illustration of a shape and not a statistic.

Four things stand out. The golden-gate effect is real and has a window. Nothing at a quarter of the posted dose, a bridge by half, an explicit Golden Gate takeover of both prompts at three quarters, and at the full posted dose one prompt still works and one collapses. That is the same narrow shape as the small model's dose sweep, in a different model. It is a combination. None of the three neurons moves the identity prompt alone at the full dose I gave it (one prompt each), which agrees with the reply thread's point that combinations can do what single neurons do not, and it also means a map of single neurons will not show it. The presets mostly reproduce, not all. Indonesian, ALL CAPS, Hugs and Cookie did what their labels say; Sleepy did not, here. I did not run the Space, so I cannot say whether Sleepy fails there too or only after my int8 rounding. The controls are quiet. Three random layer-24 neurons at the Cookie dose wrote ordinary stories with no cookies, so the cookie counts are the neuron's and not a side effect of pushing something.

The villain row is the one to read carefully. It matches what the post and the author's follow-up describe: a persona that is loud, flat and not very capable, which is how a style change looks. I treat it as exactly that. I did not test what the model refuses, and nothing here is a claim about its safety training; the three raw outputs are in the data file below and I do not reproduce them.

The raw outputs, both batches, and the two scripts that produced them are in results_batch1.json, results_batch2.json, run_batch1.py and run_batch2.py. Running them needs roughly 10 GB of memory and a first load of several minutes, which is why I did not ask the reader to.

In a tab

Running a model in a browser tab is a question of three things: where the weights come from, how much room they take, and whether the runtime lets you reach inside the MLP. The existing demonstration is Neuron Studio, a Hugging Face Space from the webml-community organisation, announced by Xenova. From its page source (reported): it loads Qwen3.5-4B from the Hub (2.91 GB, cached after the first load), runs on WebGPU with what look like its own decode kernels (reasoned, from kernel names in its bundle), adds offsets to MLP activations immediately before down_proj on every token with the same dose recipe, decodes the baseline and the steered response greedily from the same prompt, and credits the explorer as its inspiration. I have not run it, because there is no GPU here, so I say nothing about its speed.

The arithmetic is the plain part: bytes are parameters times bits. The model of this article has 596.0 million parameters (measured, counted in memory), but its checkpoint file holds 751.6 million values, because the tied embedding is stored twice (the header total). That is 1.5 GB in bf16 as shipped, 1.2 GB if a loader skips the copy, and about 0.3 GB at 4 bits. The harder part is the hook. A runtime built for generation does not expose the activation between the gate-and-up product and down_proj, so a tab that wants exact neuron steering has to own that line of the forward pass, either by writing its own kernels or by patching a graph. The explorer on this site, which uses Qwen3-0.6B, loads the weights from the Hub at run time and does exactly that; open it at /lab/neuron-lab.

What the papers support, and what they do not

Herring's contrastive neuron attribution is the one that most resembles my run, and its headline is about quality under steering. In its Figure 2 the MMLU accuracy of the neuron method stays flat at the baseline, about 63% averaged over the instruct models, up to full strength, while the residual-stream baseline falls to near zero (read off the figure, reported; the text says within 1 point).

expand

My sweep looks different because the edit is different. Herring's experiment multiplies the activations of a set of about 0.1% of the neurons by a scalar (0 ablates, 1 is baseline). My steering adds a constant to one neuron on every token, which is an unbounded edit; the recipe from the thread is of this kind. A reader who sets "stays coherent at every strength" beside a dose of 0.5 is comparing a bounded and an unbounded intervention.

The paper's other claims, from its abstract (reported): the discovered circuit's ablation lowers refusal on a standard jailbreak benchmark by more than 50% in instruct models; base models contain similar late-layer discrimination structure, but steering it produces only content shifts and not behavioural change; models from 1B to 72B were covered. I did not test any of that, and I leave the behavioural claim at the level of the abstract.

Kazemi's single-neuron paper makes the strongest claim of the three: one neuron per mechanism is enough, across seven models of 1.7B to 70B. It is a safety result about specific neurons in specific models, and I did not try to reproduce it. From my own run I can only say the benign, general part: one neuron can move a model's output, but few of the neurons a detector likes do, and the usable dose range is narrow.

Persona vectors are the dense cousin. A direction from a contrast is added at one layer, and trait expression rises with the coefficient, peaking in the middle layers (their Figure 3, reported). The thread's persona demos are best read as the sparse version of that effect: a few neurons can move a character. How they are chosen, and at what dose, is not something I publish.

How much to trust it

• The re-implementations are mine. I wrote the lens index, the steer lens, the connection scores and the batched decoder from what the repository's code and log say each does. I did not run its code or use its data, and where a number comes from the explorer it is labelled reported. Qwen3-0.6B and Qwen3.5-4B differ in depth, width and attention, so a quantity measured on one says little about the other beyond the mechanism.

• Small samples, one seed. Five reference passages of about 80 tokens, 24 neurons per layer for the steer lens, 128 source neurons for the connections, six prompts and one seed for the combination probe. The counts in that probe run from 0 to 6.

• The lens overlap is a readout, not a truth. How many of the tokens a push raises appear in a neuron's logit lens says how direct the effect is. It does not say what the neuron means.

• The depth chart is not ours. It is the log's ranges for Qwen3.5-4B, drawn as bars. I measured the steer lens at five layers of the small model and no sweep.

• One small model. Qwen3-0.6B has 28 layers and a hidden size of 1,024; the thread's model, Qwen3.5-4B, has 9,216 neurons per layer. Neuron indices do not transfer between models, so none of the thread's numbers could be tried on mine; the measurements on the small model say nothing numerical about the 4B model. The 4B runs have their own section and their own limits: int8 weights, one generation per cell.

• The 4B runs are small and quantized. One generation per cell, 60 or 70 tokens, greedy, int8 weights with 8-bit dynamic activations, and keyword counts as the only metric (they reward looping: the Cookie count at 1.6 is mostly repetition). They show that the posted recipes do something on this model, not how often or how robustly.

• Few prompts, one seed. 30 French pairs, 20 formal pairs, three generation prompts, one held-out paragraph, and a control of six random neurons at one layer with random signs. The band is a minimum and a maximum over six, not an interval.

• A crude language metric. A generation counts as foreign with three or more French, Spanish or Portuguese function words, or two or more non-ASCII letters. The printed text is the evidence.

• Mean over tokens, not the last token, which changes the ranking compared with the paper, and a weak formality probe that I treat as unresolved.

• What I could not check. The repository's claim of a coherent window of about 0.5 to 0.8 of the residual norm is for its model and its prompts. Mine breaks at 0.15 for the two language neurons, while random neurons degrade gently (perplexity up to 47 on average at 0.5) and badly only past 0.75. I cannot say which model, layer or neuron explains the difference.

What I take from it: a neuron is a sensible unit to steer, because its write is a clean direction with a natural scale, and the residual-norm dose makes doses comparable across layers. But "this neuron fires on French" is a claim about reading, and the steering recipe tests writing. The weak overlap between my contrast and attribution rankings is why a write-first sweep makes sense.

For the neighbouring methods on this site, see runtime abliteration, which subtracts one direction from the residual stream with a hook; the Jacobian lens, which reads the stream with a derivative; Matryoshka attribution, which learns a score per component; and probe monitors, which read activations rather than write them.

Sources

• The thread's steering post and follow-up, the repository and its research_log.md.

• Herring, Targeted Neuron Modulation via Contrastive Pair Search, 2605.12290.

• Kazemi, A Single Neuron Is Sufficient to Bypass Safety Alignment in Large Language Models, 2605.08513, abstract and attribution section only.

• Chen et al., Persona Vectors, 2507.21509.

• My run: Qwen3-0.6B, float32, CPU, greedy decoding. Only my own prompts and outputs are shown.

• The neuron-explorer repository at commit e566e83 (26 October 2026): README.md, NOTES.md, research_log.md, explorer/runtime.py, explorer/build_connections.py, explorer/build_layout.py, explorer/agent_mcp.py, sweep/steer_lens.py and the sizes of the files in explorer/cache/. The tree carried no licence file when I read it.

• Neuron Studio, a Hugging Face Space by webml-community, announced in a post by Xenova. Read from its page source; not run.

• The posts: @lyraaaa on the villain voice, @cheatyyyy on the topic recipe and @treetowntree on combinations.

• My re-implementations: scripts/neuron-steering/extra.py (stages A to E) and build_data.py, which writes the data behind the diagrams. Qwen3-0.6B, float32 unless stated, CPU, two threads.