Abstract neurons in vision towers: from Rosetta neurons to Motam

Tom White (drib.net · github.com/dribnet), November 2026. Working research note, last updated 21 November.

In one line: single neurons in a vision-language model's vision tower turn out to encode relations, such as "moving through a medium", and a strict ten-line geometric pattern can fire one of them harder than almost any photograph does, while leaving its neighbours quiet.

Context. This note continues a line of work on what vision models perceive and how much of it they share. Shared Visual Abstractions (2019) made drawings that several CNN classifiers agreed on. Atlas of Perception (CVPR 2025 AI Art Award) mapped one stand-alone encoder, SigLIP2, with a sparse autoencoder and a radially symmetric diffusion visualiser. Synesthetic Visions (CVPR 2026) measured agreement across twelve vision encoders with several independent metrics. The work here leaves stand-alone encoders and autoencoders behind: it studies raw relational neurons inside the vision tower of a full vision-language model, building on Goodfire's Neurons as an Interpretable Basis.

How the work evolved

June 2026: Rosetta neurons. We started from the Rosetta-neuron idea: take raw MLP units, with no trained dictionary, and match them across models by the correlation of their spatial activation maps. Across three vision models this gave 1,757 shared neurons. Across twelve models it gave ten strict universals. The surprise was which ones transferred when we optimised drawings for them: the best cross-model generaliser was a structural neuron (radial, spoked forms), not the nameable object neurons such as aircraft. One of those twelve-model Rosetta neurons (spanning roughly 90M to 7B parameters)—for raised open hands—became Show of Hands (ECCV 2026 Art Gallery), an evolved two-ink print that discarded held objects to isolate the gesture itself and transferred to held-out networks, including one trained exclusively on plants and animals.

One Rosetta neuron in twelve models
Figure 1. The "aircraft" Rosetta neuron: the matched unit's activation map in each of twelve independently trained vision models (June).

Early November: drawing inside a vision-language model. We built a differentiable pen-stroke renderer, so a page of plottable lines can be optimised by gradient against anything inside Qwen3.8-27B. This moved the target from stand-alone vision encoders to the vision tower of a model that also answers questions.

Mid November: neuron communities. We carried the "neuron communities" method from Goodfire's Neurons as an Interpretable Basis, published for language models, over to the vision tower: trace which neurons matter for answering questions about roughly 3,000 photos, then group neurons that matter together. Most communities are small and concrete (tennis, baseball, a colour). A few neurons work across many communities, and those are the interesting ones: they are not about a kind of thing.

The abstract neurons

Two of these broad neurons anchor the work, with addresses given as block:channel in the 27B tower:

Photos that fire Motam
Figure 2. Motam's strongest photos, grouped into six modes. No object is common to all of them; the relation is.

Activating a neuron instead of picturing it

Instead of drawing what a neuron responds to, we fixed a very small visual language and searched inside it: one ellipse and ten lines that never cross, because they are streamlines of an ideal fluid flowing past the ellipse. A few dozen numbers and two ink colours define a page.

In this language Motam fires at about 1.5 times the level of its top 3% of photographs. The pattern is a bundle of lines swept round a small ellipse. Remove the parts one at a time and the response falls: lines that ignore the body read 0.9, lines with no body 0.5, and the body alone 0.4. The deflection of the flow is what the neuron reads.

Motam's pattern and its switch-off tests
Figure 3. Top row: eight optimised candidates for Motam. Bottom row: the winner and the same page with one ingredient removed each time. Numbers are multiples of the neuron's top-3% photo level.

A set that tells neurons apart

One strong pattern proves little: on Motam's page roughly 1,100 other units in the tower also exceed their own photo level. So the test we adopted is a confusion matrix. Each neuron gets its own pattern, optimised to fire that neuron and not the others.

Fifteen relational neurons
Figure 4. The fifteen relational neurons, each with its pattern in the same line language. "Strongest other" here is measured against all 38 accepted neurons, which is a harder test than the set of fifteen.
Confusion matrix for ten neurons
Figure 5. Ten patterns (rows) read by ten neurons (columns). 1.0 is a neuron's top-3% photo level.

Shared across vision stacks

This closes the loop with the Rosetta work. We looked for the same neurons in other models' vision towers. For each of the fifteen we took the unit whose per-photo peak reading correlates best with the 27B neuron's on 1,200 photos, and report the correlation on a separate 1,200.

These correlations are an upper estimate. The photo set was enriched with the strongest photos of four of the neurons, and the measure is one number per photo. The next section reports a stricter check and what it changed.

Measuring alignment across models: is a neuron universal?

The question is whether a neuron such as Motam is a property of vision that many models arrive at, or a quirk of one model family. This section separates what is built, what a first stricter check found, and what is planned.

Built: the alignment scoreboard (February to July 2026)

The measures below were developed for Synesthetic Visions, comparing whole models, sparse-autoencoder features and raw neurons across twelve vision encoders.

WhenMeasureWhat it asks
MarchMutual k-nearest-neighbour overlap (mKNN), swept across k from 0.005% to 5% of a 2.9M-image poolDo two models retrieve the same images for a probe? Agreement at tiny k means shared prototypes; at larger k, a shared broad concept. Scored above chance, with a log-weighted peak.
MarchTribe scoreboardsWhich model families agree with each other (CLIP family, SigLIP family), and which concepts cross between families.
AprilProcrustes rotation versus Hungarian matching of autoencoder featuresDo a geometric and a discrete matching agree that two features are the same?
JuneRosetta neurons: correlation of spatial activation maps and mutual best match, across twelve encodersDoes a raw neuron in one model have a counterpart that fires in the same places in another? Half of the naive matches failed a split-half stability check.
June–JulyHeld-out model certificateDoes a drawing optimised on some models also fire the matched unit in a model it never saw? Verdict: real, weak or artefact.

A first neutral-pool check (21 November)

We ran the first part of the plan below against one unrelated tower, the vision encoder of Inkling 2 Mini. It is the same encoder as Inkling 2 Small's, lightly retuned: same-numbered units correlate at about 0.95. The check used 6,000 photos drawn at random, with no enrichment. Every MLP unit of the 27B tower (116,208) was compared with every unit of the Inkling encoder (94,208) by the correlation of their activation maps, pooled to an 8 × 8 grid. A pair counts only if each unit is the other's best match.

Mutual best pairs, 27B tower and Inkling encoderCount
All8,471
Correlation above 0.53,919
Above 0.7660
Above 0.8136
Above 0.95

This is one model pair and one measure, without the split-half check or the stimulus transfer. It is enough to say that the 0.67-or-higher matches above should be read as "a consistent counterpart exists", not as "the same neuron".

Planned: the universality protocol

Change the models before changing the metric. Look for the fifteen neurons in the twelve encoders from June, which include self-supervised models trained with no language (DINOv2, DINOv3, MAE). A match there says the neuron belongs to vision; a match only in language-trained towers says it may come from captions.

Candidates are found by spatial activation-map correlation, then scored on a large, neutral, deduplicated photo pool with four tests that are independent of how the match was found:

  1. Top-k overlap, swept across k. Rank the pool by each model's neuron and compare the top k, from a tiny fraction to about 5%. For a neuron with several modes this separates "agrees on surfers only" from "agrees on the whole relation".
  2. Transfer of the stimulus, with switch-offs. The line pattern made on one model must fire the matched neuron in a model that never saw it, and the switch-off tests must hold there too. This is the strongest test, because it probes the neuron away from photographs.
  3. Mutual best match with a margin, and split-half stability. The match must pick the original neuron back, clearly ahead of the runner-up, on both halves of the data.
  4. Spatial agreement. Do both neurons fire on the same part of the picture: the figure, or the medium it moves through?

Whole-representation measures (Procrustes, Hungarian matching, explained variance) serve as the baseline: what share of all neurons find a match this good, so that a neuron can be called unusually shared or typical. The check above is a first version of test 3 and of that baseline, for one tower. Tests 1, 2 and 4, and the language-free encoders, have not been run.

From vision representations to verbalizable readouts (Jacobian lens)

Framed print Shelfwild at CVPR 2026Nearest-neighbour photos around the printJacobian-lens readout at layer 37 of gemma-3-12b-it
Figure 6. Left: Shelfwild, a print from Synesthetic Visions, as shown at CVPR 2026. Centre: the photographs nearest to it in a vision model's embedding space. Right: the Jacobian-lens readout of the same print at layer 37 of gemma-3-12b-it, with the mean readout of the other prints subtracted: stacks, stack, bookshelves, books, stacked.

The Jacobian lens, introduced in Verbalizable Representations Form a Global Workspace in Language Models (Anthropic, July 2026; arXiv:2607.15495), reads out which words an internal state makes a model more likely to say later. Applied to photographs of the physical CVPR 2026 prints, it recovered the concept cluster that vision-model nearest neighbours show for the same print: stacks, bookshelves, books. An earlier layer (29) gave near-synonyms: bookshelves, bookcases, books (thread). The prints were made against vision encoders alone, so the readout is a language model naming a concept it was never asked about.

The open question for the line patterns: when a sparse ten-line page fires a relational neuron such as Motam or Aloft, does that activation cross into the model's verbalizable workspace (J-space), or does it stay sub-verbal? A first diagnostic pass suggests some of it crosses. A lens fitted to Qwen3.6-35B, read at a middle layer with the mean of the other patterns subtracted, gives "turbulence, swirling, flow" for Motam's pattern, "shower" for Cascade's, "hanging" for Dangle's and "radius" for Radial's. Aloft's reads as "sky". Others are off: Team's reads "ankle". These are single readouts of one lens with no controls, and the lens fitted to the 27B itself reads weakly. What such activations do to downstream reasoning, world-state tracking or visual robustness is untested.

What is not yet shown

Where this is heading

One physical endpoint and two empirical ones.