Mechanistic Interpretability: What We Actually Know So Far

For most of deep learning's history, interpretability research meant one thing: post-hoc rationalizations. Saliency maps, LIME, SHAP — useful in narrow contexts, but fundamentally telling you which input features a model attended to, not why it produced the output it did. Mechanistic interpretability is a different project. It asks a harder question: can we reverse-engineer the algorithms that a trained neural network is actually running?

The answer, for small and mid-sized models, is increasingly yes — with caveats. A body of reproducible work now exists. Understanding it precisely, rather than through the hype filter, is worth doing.

The Established Results

The most durable finding from mechanistic interpretability is the existence of circuits: identifiable subgraphs of weights and attention heads that implement specific, nameable computations. The canonical example is the induction head circuit in transformer language models.

An induction head is a pair of attention heads that together implement a simple in-context copying algorithm: if the sequence contains [A][B]...[A], the second head attends back to the token that previously followed A and predicts B. This is not a metaphor or an approximation — researchers have located the specific heads, ablated them, and observed the predicted degradation in in-context learning performance. The mechanism is real and measurable.

Induction heads matter beyond their own narrow function because they appear to be a primitive underlying a broader class of in-context learning behavior. When you give a language model a few examples of a new task format and it generalizes, induction-head-style circuits are doing meaningful work in that process. This is a concrete, falsifiable claim about model internals, not a story told after the fact.

A second established result is superposition. Neural networks represent more features than they have dimensions by encoding multiple features as nearly-orthogonal directions in the same activation space. This explains a long-standing puzzle: why do probing classifiers find linearly decodable features almost everywhere, even in layers that seem too small to hold them? The answer is that the network is doing compressed, lossy storage, and the geometry of that compression is not random — it follows predictable patterns based on feature frequency and co-occurrence.

Superposition has a direct engineering implication. It means that activation patching and causal tracing experiments — where you swap activations between a clean and corrupted forward pass to locate where a fact is stored — are measuring something real, but something that is entangled with many other computations at the same site. Locating a fact does not mean isolating it.

The Methodology and Its Limits

The primary tool of mechanistic interpretability is activation patching, sometimes called causal mediation analysis. The setup: run the model on two inputs that differ in a controlled way, then systematically swap activations at each layer and head position to find which components are causally responsible for the output difference. It is a principled approach, borrowed loosely from causal inference.

The problem is that the findings are sensitive to the choice of intervention. Patching at the residual stream versus patching at the output of an attention head versus patching at the input to an MLP layer can tell different stories. There is no single canonical decomposition of a transformer's computation, and the choice of decomposition shapes what you find. This is not a fatal flaw, but it means that circuit findings should be read as findings about a particular analysis protocol, not as unique ground truth about the model.

A related problem is polysemanticity. Individual neurons, and even individual attention heads, often respond to multiple unrelated features. A neuron active for French text may also activate for certain code patterns. This is a consequence of superposition, and it means that the clean circuit diagrams produced by interpretability papers are often simplifications. The real computation is messier, with components doing double and triple duty.

Researchers have made progress here using sparse autoencoders (SAEs) to decompose polysemantic activations into monosemantic features — directions in activation space that correspond to a single, nameable concept. SAE-derived features are more interpretable than raw neurons, and they have been used to find features corresponding to specific named entities, syntactic roles, and even emotional valence. But the features found depend heavily on the SAE's sparsity penalty and architecture, introducing another layer of methodological choices between the researcher and the model's internals.

The Scale Problem

Almost all of the cleanest mechanistic results come from models with fewer than a few billion parameters, often from small transformers trained on synthetic tasks. The move to frontier-scale models is not just a quantitative scaling challenge — it may be a qualitative one.

At scale, circuits become distributed across more layers and more heads. The clean two-head induction circuit found in small models has analogs in large models, but they are harder to isolate because the computation is spread across a wider graph. More importantly, large models appear to develop multiple overlapping circuits that implement similar computations redundantly. Ablating one circuit does not degrade performance as cleanly because others compensate. This redundancy is useful for robustness but hostile to interpretability.

There is also an emerging hypothesis — not yet fully established — that qualitatively new computational structures emerge at scale that have no analog in smaller models. If true, this would mean that the circuit vocabulary developed on small models does not transfer, and the field would need new primitives for each capability tier. This is an open and important question.

Why Engineers Should Care

The practical payoff from mechanistic interpretability is not yet a set of production tools. It is a set of sharper mental models for reasoning about model behavior.

• Prompt sensitivity has a mechanistic basis. When a model behaves very differently on semantically equivalent prompts, it is because the tokenization and early-layer representations activate different circuits. Understanding that this is a structural feature of how transformers work — not a bug that will be patched — changes how you design evaluation suites.
• Fine-tuning can suppress circuits without removing them. Research on model editing suggests that fine-tuning often suppresses a behavior by adding a competing signal rather than by modifying the underlying circuit. This explains why fine-tuned safety behaviors can be brittle — the original circuit is still there, waiting for a prompt that routes around the suppressor.
• Activation steering is a real intervention. Adding a feature direction to a model's residual stream mid-forward-pass can reliably shift its behavior in the direction corresponding to that feature. This is not magic; it follows directly from the linear representation hypothesis. It also means that as SAE-derived features become more reliable, activation-space interventions will become a more practical tool for controlled generation.

None of this means you should be running your own interpretability experiments in production. It means that the research output of this field is now dense enough to be worth reading carefully, because it is changing what is possible in model evaluation, editing, and control.

The Honest Assessment

Mechanistic interpretability has earned its place as a serious research program. It has produced reproducible findings, developed a methodological toolkit with known failure modes, and started to connect to practical questions about model behavior. That is more than most interpretability approaches can claim.

But the field's central ambition — a complete, human-readable account of what a frontier model is doing when it reasons — remains far off. The gap between understanding a two-layer attention-only transformer on a synthetic task and understanding a hundred-billion-parameter model on an open-ended question is not a matter of more compute and more grad students. It may require new conceptual frameworks that do not yet exist.

The right posture is neither dismissal nor credulity. The findings are real. The extrapolations are speculative. For a senior engineer, that distinction is the whole game.