Abstract neurons in vision towers: from Rosetta neurons to Motam
In one line: single neurons in a vision-language model's vision tower turn out to encode relations, such as "moving through a medium", and a strict ten-line geometric pattern can fire one of them harder than almost any photograph does, while leaving its neighbours quiet.
Context. This note continues a line of work on what vision models perceive and how much of it they share. Shared Visual Abstractions (2019) made drawings that several CNN classifiers agreed on. Atlas of Perception (CVPR 2025 AI Art Award) mapped one stand-alone encoder, SigLIP2, with a sparse autoencoder and a radially symmetric diffusion visualiser. Synesthetic Visions (CVPR 2026) measured agreement across twelve vision encoders with several independent metrics. The work here leaves stand-alone encoders and autoencoders behind: it studies raw relational neurons inside the vision tower of a full vision-language model, building on Goodfire's Neurons as an Interpretable Basis.
How the work evolved
June 2026: Rosetta neurons. We started from the Rosetta-neuron idea: take raw MLP units, with no trained dictionary, and match them across models by the correlation of their spatial activation maps. Across three vision models this gave 1,757 shared neurons. Across twelve models it gave ten strict universals. The surprise was which ones transferred when we optimised drawings for them: the best cross-model generaliser was a structural neuron (radial, spoked forms), not the nameable object neurons such as aircraft. One of those twelve-model Rosetta neurons (spanning roughly 90M to 7B parameters)—for raised open hands—became Show of Hands (ECCV 2026 Art Gallery), an evolved two-ink print that discarded held objects to isolate the gesture itself and transferred to held-out networks, including one trained exclusively on plants and animals.
Early November: drawing inside a vision-language model. We built a differentiable pen-stroke renderer, so a page of plottable lines can be optimised by gradient against anything inside Qwen3.8-27B. This moved the target from stand-alone vision encoders to the vision tower of a model that also answers questions.
Mid November: neuron communities. We carried the "neuron communities" method from Goodfire's Neurons as an Interpretable Basis, published for language models, over to the vision tower: trace which neurons matter for answering questions about roughly 3,000 photos, then group neurons that matter together. Most communities are small and concrete (tennis, baseball, a colour). A few neurons work across many communities, and those are the interesting ones: they are not about a kind of thing.
The abstract neurons
Two of these broad neurons anchor the work, with addresses given as block:channel in the 27B tower:
- Aloft (21:559): a thing off the ground, with air around it. Birds, kites, aircraft, skaters mid-jump.
- Motam (18:2528), "Moving Through A Medium": surfers in water, mountain bikers in dirt, rally cars in dust, skiers in snow, parachutists in air.
Activating a neuron instead of picturing it
Instead of drawing what a neuron responds to, we fixed a very small visual language and searched inside it: one ellipse and ten lines that never cross, because they are streamlines of an ideal fluid flowing past the ellipse. A few dozen numbers and two ink colours define a page.
In this language Motam fires at about 1.5 times the level of its top 3% of photographs. The pattern is a bundle of lines swept round a small ellipse. Remove the parts one at a time and the response falls: lines that ignore the body read 0.9, lines with no body 0.5, and the body alone 0.4. The deflection of the flow is what the neuron reads.
A set that tells neurons apart
One strong pattern proves little: on Motam's page roughly 1,100 other units in the tower also exceed their own photo level. So the test we adopted is a confusion matrix. Each neuron gets its own pattern, optimised to fire that neuron and not the others.
- Ten neurons, all optimised against each other: every pattern is read most strongly by its own neuron. The mean own reading is 1.60 and the mean reading by the others is 0.29.
- An overnight search over 360 further neurons accepted 38 in all. Fifteen of those are relational (relation, pose or arrangement). As a set of fifteen, all fifteen win their own row, with a mean own reading of 1.66 against 0.29.
- Robustness: the first ten keep their rows when the page is resized, blurred or shifted, with one exception at the smallest size.
Shared across vision stacks
This closes the loop with the Rosetta work. We looked for the same neurons in other models' vision towers. For each of the fifteen we took the unit whose per-photo peak reading correlates best with the 27B neuron's on 1,200 photos, and report the correlation on a separate 1,200.
- Qwen3.6-35B: all fifteen are present, thirteen at the identical address (Motam correlates at 0.91). The two towers clearly share an ancestor, so this is the weakest evidence of universality.
- Muse-Glimmer-30B and Inkling 2 Small: eight of the fifteen have a clear match in both (correlation 0.67 or higher): Motam, Aloft, Surface, Motion, Sprawl, Profile, Astride and Team. Motam's matches correlate at 0.77 and 0.87.
- Transfer without retraining: a pattern made only for the 27B's Aloft fires its Muse counterpart at that neuron's own top-3% photo level. Motam's reaches about three quarters of the level. Inkling's counterparts barely respond.
These correlations are an upper estimate. The photo set was enriched with the strongest photos of four of the neurons, and the measure is one number per photo. The next section reports a stricter check and what it changed.
Measuring alignment across models: is a neuron universal?
The question is whether a neuron such as Motam is a property of vision that many models arrive at, or a quirk of one model family. This section separates what is built, what a first stricter check found, and what is planned.
Built: the alignment scoreboard (February to July 2026)
The measures below were developed for Synesthetic Visions, comparing whole models, sparse-autoencoder features and raw neurons across twelve vision encoders.
| When | Measure | What it asks |
|---|---|---|
| March | Mutual k-nearest-neighbour overlap (mKNN), swept across k from 0.005% to 5% of a 2.9M-image pool | Do two models retrieve the same images for a probe? Agreement at tiny k means shared prototypes; at larger k, a shared broad concept. Scored above chance, with a log-weighted peak. |
| March | Tribe scoreboards | Which model families agree with each other (CLIP family, SigLIP family), and which concepts cross between families. |
| April | Procrustes rotation versus Hungarian matching of autoencoder features | Do a geometric and a discrete matching agree that two features are the same? |
| June | Rosetta neurons: correlation of spatial activation maps and mutual best match, across twelve encoders | Does a raw neuron in one model have a counterpart that fires in the same places in another? Half of the naive matches failed a split-half stability check. |
| June–July | Held-out model certificate | Does a drawing optimised on some models also fire the matched unit in a model it never saw? Verdict: real, weak or artefact. |
A first neutral-pool check (21 November)
We ran the first part of the plan below against one unrelated tower, the vision encoder of Inkling 2 Mini. It is the same encoder as Inkling 2 Small's, lightly retuned: same-numbered units correlate at about 0.95. The check used 6,000 photos drawn at random, with no enrichment. Every MLP unit of the 27B tower (116,208) was compared with every unit of the Inkling encoder (94,208) by the correlation of their activation maps, pooled to an 8 × 8 grid. A pair counts only if each unit is the other's best match.
| Mutual best pairs, 27B tower and Inkling encoder | Count |
|---|---|
| All | 8,471 |
| Correlation above 0.5 | 3,919 |
| Above 0.7 | 660 |
| Above 0.8 | 136 |
| Above 0.9 | 5 |
- Rosetta pairs exist between unrelated towers, mostly early. Of the 660 pairs above 0.7, 516 sit in blocks 0–12 of the 27B's 27 blocks, and matched units sit at a similar relative depth in both models. Blocks 15–26, where the relational neurons live, hold 106.
- None of the fifteen is among them. Their best map correlations range from 0.19 to 0.49. Only Motam and Motion are mutual best matches, at 0.39 and 0.36.
- The partners are consistent but loose. The scan picked the same Inkling unit for Motam as the earlier enriched run did (block 19, unit 3056). On per-photo peaks it correlates at 0.56 on random photos, against 0.87 on the enriched set.
- Typical for their depth. In its block, about a quarter of units match the other tower better than Motam does. By this one test Motam is not unusually shared.
This is one model pair and one measure, without the split-half check or the stimulus transfer. It is enough to say that the 0.67-or-higher matches above should be read as "a consistent counterpart exists", not as "the same neuron".
Planned: the universality protocol
Change the models before changing the metric. Look for the fifteen neurons in the twelve encoders from June, which include self-supervised models trained with no language (DINOv2, DINOv3, MAE). A match there says the neuron belongs to vision; a match only in language-trained towers says it may come from captions.
Candidates are found by spatial activation-map correlation, then scored on a large, neutral, deduplicated photo pool with four tests that are independent of how the match was found:
- Top-k overlap, swept across k. Rank the pool by each model's neuron and compare the top k, from a tiny fraction to about 5%. For a neuron with several modes this separates "agrees on surfers only" from "agrees on the whole relation".
- Transfer of the stimulus, with switch-offs. The line pattern made on one model must fire the matched neuron in a model that never saw it, and the switch-off tests must hold there too. This is the strongest test, because it probes the neuron away from photographs.
- Mutual best match with a margin, and split-half stability. The match must pick the original neuron back, clearly ahead of the runner-up, on both halves of the data.
- Spatial agreement. Do both neurons fire on the same part of the picture: the figure, or the medium it moves through?
Whole-representation measures (Procrustes, Hungarian matching, explained variance) serve as the baseline: what share of all neurons find a match this good, so that a neuron can be called unusually shared or typical. The check above is a first version of test 3 and of that baseline, for one tower. Tests 1, 2 and 4, and the language-free encoders, have not been run.
From vision representations to verbalizable readouts (Jacobian lens)
The Jacobian lens, introduced in Verbalizable Representations Form a Global Workspace in Language Models (Anthropic, July 2026; arXiv:2607.15495), reads out which words an internal state makes a model more likely to say later. Applied to photographs of the physical CVPR 2026 prints, it recovered the concept cluster that vision-model nearest neighbours show for the same print: stacks, bookshelves, books. An earlier layer (29) gave near-synonyms: bookshelves, bookcases, books (thread). The prints were made against vision encoders alone, so the readout is a language model naming a concept it was never asked about.
The open question for the line patterns: when a sparse ten-line page fires a relational neuron such as Motam or Aloft, does that activation cross into the model's verbalizable workspace (J-space), or does it stay sub-verbal? A first diagnostic pass suggests some of it crosses. A lens fitted to Qwen3.6-35B, read at a middle layer with the mean of the other patterns subtracted, gives "turbulence, swirling, flow" for Motam's pattern, "shower" for Cascade's, "hanging" for Dangle's and "radius" for Radial's. Aloft's reads as "sky". Others are off: Team's reads "ankle". These are single readouts of one lens with no controls, and the lens fitted to the 27B itself reads weakly. What such activations do to downstream reasoning, world-state tracking or visual robustness is untested.
What is not yet shown
- The switch-off result is local. No neuron in the tower is tuned to "deflection" across random flows on average. Motam's dependence on it holds for its optimised page.
- The matching correlations are optimistic, because the photo set was enriched with each neuron's strongest photos. The neutral-pool check above confirms this for one tower: Motam's match falls from 0.87 to 0.56 on per-photo peaks, and to 0.39 on activation maps.
- Stricter rules cost a lot. Under the tightest rules (all ten lines long and separate) and two towers at once, the first eight results kept only Motam, Aloft and Team above their photo level, and two patterns failed outright. With the rules relaxed to ten non-crossing streamlines and an evolutionary search in place of gradient descent, thirteen of the fifteen now read above their photo level on both the 27B and Qwen3.6 towers while staying ahead of the other fourteen. Motion and Particles remain below, at about 0.8. The tightest rules have not been met for the full set.
- Nothing plotted has been measured. Every number is from rendered pages. A first Motam page has been drawn by the plotter but not yet photographed and re-scored. An earlier pen-plotted drawing lost most of its score on paper, so that test matters.
Where this is heading
One physical endpoint and two empirical ones.
- A small visual basis, drawn with pens. About twelve patterns in one narrow line language, mutually exclusive, each firing one relational neuron, pen-plotted and re-scored from photographs of the paper.
- Universality. Run the full protocol above on Motam and the other relational units, across language-free vision models (DINO, MAE) and the vision towers of current multimodal models, and report which are shared and which are family quirks. The 106 deep Rosetta pairs found between the 27B and Inkling towers are a second pool of candidates, chosen for being shared first.
- From tower to words, and what follows. Trace the handoff from vision-tower neurons into verbalizable J-space with the Jacobian lens and switch-off tests on held-out models, and test what sparse, cross-model visual triggers imply for steering, visual blind spots and lightweight monitoring.