Mechanistic interpretability: finding the concepts hidden inside a model's activations

A trained neural network's internal activations are a dense mixture of overlapping signals. A single neuron rarely corresponds to one clean human concept — it responds to a tangle of seemingly unrelated inputs at once. Mechanistic interpretability research is an attempt to look inside that mixture and find the individual concepts anyway.

The neuron is not the natural unit of understanding. It just happens to be the unit that's easiest to measure.

Decomposing a tangled neuron into clean features

Towards Monosemanticity: Decomposing Language Models With Dictionary Learning ↗, from Anthropic's interpretability team, names the core obstacle directly: many neurons are polysemantic — a single neuron in a vision model, the paper notes by way of example, can respond to both cat faces and car fronts, two concepts with nothing in common. Their method trains a sparse autoencoder on a model's internal activations, expanding them into a much larger set of features than the original activation dimensions, with a sparsity constraint that forces most features to stay inactive for any given input.

Architecture / mechanistic interpretability

One neuron, many concepts — one feature, one concept

FIG 01

The paper reports that a substantial share of the resulting features are far more monosemantic than the raw activation dimensions they were extracted from — each one tends to correspond to a single, often human-recognizable concept, rather than a blend of several.

Scaling to a model people actually use

Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet ↗ extends this approach from the small one-layer transformer of the original paper to a production-scale model — Claude 3 Sonnet, the exact model in production at the time of writing. The paper reports finding a diversity of highly abstract features: concrete entities like famous people and cities, but also more abstract patterns like code containing security vulnerabilities, spanning both a literal code example and an abstract discussion of the same vulnerability concept. Many features generalize across languages and across text and images simultaneously.

Critically, the paper reports these features "both respond to and behaviorally cause" the behaviors they are named for — verified through direct intervention on the feature's activation, not merely inferred from correlation. That distinction is what separates a genuine causal account of model behavior from a plausible-sounding post-hoc story.

Evidence / causal verification

Correlation names a feature. Intervention proves it.

FIG 02

What this buys — and what it still costs

A feature dictionary extracted this way gives a team a mechanistic handle on model behavior that goes beyond input-output testing: instead of only observing what a model produces, a team can identify which internal feature was active when it produced a specific behavior, and — because the causal link is verified — intervene on that feature directly to test the hypothesis. This is a materially different kind of evidence than a behavioral test alone provides.

It is not a finished, turnkey capability. Extracting a feature dictionary at scale is computationally expensive, feature interpretations still require careful human validation to avoid over-reading a plausible-sounding label onto a feature's actual behavior, and both papers demonstrate the approach on specific model families rather than establishing it as a universal, automatic property of any model.

What this changes in production

Safety and behavior auditing

Where a team suspects a model has learned an internal representation tied to a specific unsafe behavior, feature-level analysis offers a way to locate and directly test that representation — monitoring or intervening on the feature itself, rather than only testing for the behavior at the model's output.

Debugging unexpected behavior

When a model produces an unexpected or hard-to-explain output, tracing that output back to the specific internal features that were active provides a mechanistic account of what happened, distinct from re-reading the input prompt and speculating about what might have caused it.

Governance and audit evidence

Feature-level causal evidence is a stronger form of documentation than behavioral testing alone for organizations that need to explain why a model behaves a certain way, not just demonstrate that it does under a given test suite.

A production checklist

Before relying on mechanistic interpretability findings operationally, the team should be able to answer:

• Was a feature's role verified through direct intervention, or only inferred from what inputs happen to activate it?
• Is the feature dictionary being used to monitor a specific, well-defined risk, or as a general-purpose explanation tool it was not validated for?
• How current is the feature dictionary relative to the deployed model version — was it extracted from the exact model in production, or an earlier checkpoint?
• Who reviews feature interpretations for the risk of an appealing but incorrect label being attached to a feature's actual behavior?
• Is this evidence treated as a complement to behavioral testing, or as a replacement for it?
• What is the computational and engineering cost of maintaining this capability as the underlying model is updated?

Research referenced

• Anthropic interpretability team (2023), Towards Monosemanticity: Decomposing Language Models With Dictionary Learning ↗.
• Anthropic interpretability team (2024), Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet ↗.