Literature
Display
← All papers

Open Problems in Mechanistic Interpretability

Lee Sharkey * Apollo Research · Bilal Chughtai * Apollo Research · Joshua Batson Anthropic · Jack Lindsey Anthropic · Jeff Wu Anthropic † · Lucius Bushnaq Apollo Research · Nicholas Goldowsky-Dill Apollo Research · Stefan Heimersheim Apollo Research · Alejandro Ortega Apollo Research · Joseph Bloom Decode Research · Stella Biderman Eleuther AI · Adria Garriga-Alonso FAR AI · Arthur Conmy Google DeepMind · Neel Nanda Google DeepMind · Jessica Rumbelow Leap Laboratories · Martin Wattenberg Harvard University · Nandi Schoots King’s College London and Imperial College London · Joseph Miller MATS · Eric J. Michaud MIT · Stephen Casper MIT · Max Tegmark MIT · William Saunders METR · David Bau Northeastern University · Eric Todd Northeastern University · Atticus Geiger Pr(AI)r group · Mor Geva Tel Aviv University · Jesse Hoogland Timaeus · Daniel Murfet University of Melbourne · Tom McGrath Goodfire

Published · Transactions on Machine Learning Research ↗

Interpretability7 min text · CondensedOriginal paper ↗
Contents

Overview

Apollo Research synthesized the authors’ perspectives for Schmidt Sciences. The review does not necessarily represent the views of each author or their institution.

At a glance

This review identifies open problems in the foundations, methods, applications, and social consequences of mechanistic interpretability. Its proposals are a research agenda, not evidence that current methods provide comprehensive understanding or safety assurance.

1 Introduction

Understanding learned mechanisms could improve prediction, control, monitoring, and scientific discovery. The review treats understanding pragmatically through the ability to make useful predictions or accomplish concrete goals, while recognizing broader scientific motivations. It builds on earlier interpretability research.

1.1 The focus of this review: Open problems and the future of mechanistic interpretability

The review emphasizes unresolved questions and research priorities rather than offering an exhaustive historical catalog of methods.

1.1.1 Why ‘mechanistic’ interpretability?

The authors use mechanistic interpretability to mean investigation of the mechanisms underlying generalization. They distinguish it from intrinsically interpretable model design and explanations of individual decisions, while acknowledging overlap and inconsistent terminology across the field.

1.2 Types of open problems

The agenda connects methodological foundations to monitoring, editing, prediction, efficient learning, and knowledge discovery, then examines governance and philosophical questions.

2 Open problems in mechanistic interpretability methods and foundations

Two complementary approaches start from different directions: reverse engineering decomposes a network and asks what its parts do; concept-based methods start with a proposed role and look for components representing it.

2.1 Reverse engineering: Identifying the roles of network components

Reverse engineering proceeds through decomposition, functional description, and validation, revisiting earlier steps when predictions fail.

2.1.1 Reverse engineering is necessary because AI and humans use different representations and perform different tasks

Humanlike outputs need not imply humanlike computation. Neural networks can use unfamiliar representations and shortcuts even on familiar tasks. Methods should allow unexpected mechanisms to be discovered rather than impose only concepts researchers already anticipate.

Refer to caption
Figure 1: Two approaches to neural network interpretability. (Left) Reverse Engineering is characterized by decomposing networks into functional components and describing how those components interact to produce the network’s behavior. It thus aims to ‘identify the roles of network components’ (Section 2.1). (Right) Concept-based interpretability on the other hand attempts to discover human concepts within neural network internals. It thus aims to ‘identify the network components for given roles’ (Section 2.2).

2.1.2 Reverse engineering step 1: Neural network decomposition

Architectural units are often polysemantic. Sparse dictionary learning addresses superposition, but faces reconstruction error, cost, uncertain linearity assumptions, feature splitting and absorption, unexplained geometry, dataset dependence, and difficulties across attention heads or layers. Sparse activations are not the mechanisms implemented by weights. Stronger foundations, alternative decompositions, and intrinsically interpretable architectures may help, but their performance and explanatory advantages require testing.

Original figure
Figure 2: The steps of reverse engineering neural networks. (1) Decomposing a network into simpler components. This decomposition might not necessarily use architecturally-defined bases, such as individual neurons or layers (Section 2.1.2). (2) Hypothesizing about the functional roles of some or all components (Section 2.1.3). (3) Validating whether our hypotheses are correct, creating a cycle in which we iteratively refine our decompositions and hypotheses to improve our understanding of the network (Section 2.1.4).
Original figure
Figure 3: Three ideas underlying the sparse dictionary learning (SDL) paradigm in mechanistic interpretability. (Left) The linear representation hypothesis states that the map from ‘concepts’ to neural activations is linear. (Middle) Superposition is the hypothesis that models represent many more concepts than they have dimensions by representing them both sparsely and linearly in activation spaces. (Right) SDL attempts to recover an overcomplete basis of concepts represented in superposition in activation space.
Original figure
Figure 4: Sparse dictionary learning has a number of practical and conceptual limitations that cause issues when using it to reverse engineer neural networks (Section 2.1.2).

2.1.3 Reverse engineering step 2: Describing the functional role of components

Component descriptions should cover both what activates a component and what it causes downstream. Activation examples, attribution, feature synthesis, logit lenses, patching, and steering provide different evidence. Each has limitations: correlations can mislead, perturbations may be off distribution, gradients approximate nonlinear effects, and verbal chains of thought need not faithfully report computation.

Original figure
Figure 5: To study the functional role of the blue component numerous approaches are possible. We could study its causes: the purple input components or the red intermediate components via e.g. feature synthesis (Olah et al., 2017a; Olah et al., 2020b), maximum activating examples (Olah et al., 2017a; Bricken, 2023), or attributions (Sundararajan et al., 2017). Or we could study its effects: the orange output components or the green intermediate components via e.g. the logit lens (Nostalgebraist, 2020), activation steering (Turner et al., 2024), or attributions (Marks et al., 2024).

2.1.4 Reverse engineering step 3: Validation of descriptions

Treat interpretations as hypotheses. Validate predictions of activations, interventions, failures, or replacement components, and compare downstream utility with relevant alternatives. Shared model organisms and benchmarks with known mechanisms can improve rigor, but easy toy tasks and selected successes may poorly represent real deployment problems.

2.2 Concept-based interpretability: Identifying components for given roles

Concept-based interpretability searches for specified human concepts and therefore complements, rather than replaces, open-ended discovery.

2.2.1 Concept-based probes

Probes predict labels from hidden activations. Successful decoding establishes available information, but does not show that the model causally uses the decoded direction.

2.2.2 Probes need carefully chosen data for well-defined concepts

Probe quality depends on precisely defined concepts and representative labels. Even less-supervised consistency methods require structured data, and all can miss concepts that researchers did not think to test.

2.2.3 Probes detect correlations, rather than causal variables, in hidden activations

Correlations in high-dimensional activations may be spurious or causally irrelevant. Counterfactual interventions, causal subspace searches, erasure, and distribution-shift tests can strengthen evidence, but require additional assumptions and specialized data. Appropriate regularization remains unresolved.

2.2.4 Concept-based intrinsic interpretability

Concept bottlenecks and prescribed causal structures can make trained components more directly meaningful. Their coverage is limited by which concepts and structures can be specified in advance.

2.3 Proceduralizing mechanistic interpretability into circuit discovery pipelines: A case study

Circuit discovery typically defines a task, decomposes a model, isolates relevant components, iterates on explanations, and evaluates the result. Task definitions, incomplete decompositions, intervention-sensitive faithfulness metrics, gradient approximations, backup mechanisms, and inhibitory behavior complicate this pipeline. Studying only convenient tasks can exaggerate its generality.

2.4 Automating steps in mechanistic interpretability research

Language models can help describe features and test activation predictions; automated circuit discovery can identify relevant subgraphs. These tools do not yet automate reliable functional explanations of whole systems. Better hypothesis generation, validation, and methodological foundations remain necessary.

3 Open problems in applications of mechanistic interpretability

Applications provide concrete goals, but different goals require different kinds and amounts of mechanistic understanding.

3.1 Axes of mechanistic interpretability progress.

Progress varies in decomposition quality, description depth, fraction of the network covered, breadth of tasks, and understanding of changes during training. Improvements along one axis need not solve every application.

Refer to caption
Figure 6: A summary of problem areas for applications of mechanistic interpretability.

3.2 Using mechanistic interpretability for better monitoring and auditing of AI systems for potentially unsafe cognition

Internal monitoring and audits may complement behavioral evaluations by revealing evidence unavailable from outputs alone.

3.2.1 Mechanistic interpretability-based evaluations could help us detect unsafe or unethical AI cognition

White-box evaluations, anomaly detection, and interpretability-guided red-teaming may expose concerning patterns even without complete model understanding. However, recognizing a concept is different from using it to pursue a goal, and detecting some risks does not demonstrate coverage of all unsafe cognition.

3.2.2 Using mechanistic interpretability for better control of AI system behavior

Steering, unlearning, knowledge editing, and analysis of fine-tuning may benefit from better decomposition. Precise changes without side effects remain difficult because capabilities overlap and apparent removal can merely mask behavior. Proposed benefits must be compared with existing control methods.

3.3 Using mechanistic interpretability for better predictions about AI systems

The review separates prediction in new situations from prediction of capabilities that emerge during learning.

3.3.1 Predicting behavior in novel situations

Mechanistic evidence can suggest failure cases and potentially support stronger claims than sampled behavior alone. Exhaustive safety arguments and formal verification require far more coverage than current methods provide. Toy verification results do not establish guarantees for general agents in complex environments.

3.3.2 Predicting capabilities that arise during training or finetuning

Predicting capability development requires understanding changing mechanisms, training-data effects, latent capabilities, and multi-step behavior. Existing correlations between circuits and emerging abilities are promising but incomplete.

3.4 Using mechanistic interpretability to improve our ability to perform inference, improve training, and make better use of learned mechanisms

Possible uses include skipping redundant computation, better distillation and data selection, targeted training, improved architectures, debugging, and transfer between models. These are varied opportunities rather than one demonstrated general recipe.

3.5 Using mechanistic interpretability for ‘microscope AI’

Microscope AI uses learned representations to discover patterns in scientific data. Extracted insights require independent domain validation and accessible tools; current applications depend on a rare combination of model, interpretability, and domain expertise.

3.6 Mechanistic interpretability on a broader range of models and model families

Research concentrates on a few vision and language architectures. Broader comparisons should distinguish transferable methods and universal mechanisms from model-specific findings, including in multimodal and state-space systems.

3.7 Human computer interaction with model internals

Interfaces should support researchers, auditors, and end users with appropriate depth, clear uncertainty, and useful controls. Transparency can aid trust calibration only if the displayed explanations are reliable.

4 Open socio-technical problems in mechanistic interpretability

Technical progress interacts with governance, incentives, communication, and the meaning of useful explanations.

4.1 Translating technical progress in mechanistic interpretability into levers for AI policy and governance

The authors propose applications to risk assessment, incident investigation, monitoring, accountability, and model editing. These are possibilities discussed in the review’s policy context, not claims that current interpretability methods establish legal compliance or reliably verify the absence of harmful mechanisms.

4.2 Social and philosophical problems in mechanistic interpretability

The field needs clearer goals and fair comparisons with non-interpretability methods. Scientific understanding and practical utility are both legitimate aims, but selective demonstrations and corporate incentives can create false confidence. Findings should be communicated with their limitations and deployment context.

5 Conclusion

Progress requires stronger decompositions, causal validation, understanding of learning dynamics, and realistic evaluations. The authors advocate pursuing scientific and engineering advances together while avoiding unsupported safety assurances.

Author Contributions

Lee Sharkey coordinated the review; Bilal Chughtai contributed framing, editing, and figures. The wider author group contributed sections and feedback, with acknowledged editorial and funding support.

Appendix A Summary of open questions

The appendix organizes the review’s research agenda into a detailed checklist of unresolved questions.

A.1 Open problems in mechanistic interpretability methods and foundations

Methodological questions cover decomposition, description, validation, concept localization, pipelines, and automation.

A.1.1 Reverse engineering: Identifying the roles of network components

The reverse-engineering agenda asks how to define useful units and abstractions, characterize linearity and superposition, address dictionary-learning failures, and connect feature geometry with generalization theory. It also calls for faithful attribution, better descriptions of unfamiliar concepts, causal stress tests, shared model organisms, and realistic comparisons with black-box baselines.

A.1.2 Concept-based interpretability: Identifying network components for given roles

Open probing questions concern causal relevance, automated high-quality labels, generalization and regularization, and concepts without clear positive or negative examples.

A.1.3 Proceduralizing mechanistic interpretability into circuit discovery pipelines

Circuit-pipeline questions concern learned task definitions, improved decompositions, backup and inhibitory behavior, faithful explanations, and extension beyond conveniently defined tasks.

A.1.4 Automating steps in mechanistic interpretability research

Automation should extend from feature labels to hypothesis testing, interactions, decomposition design, and validation. The agenda also raises the possibility of unreliable or strategically misleading automated researchers.

A.2 Open problems in applications of mechanistic interpretability

The application checklist links methodological progress to concrete monitoring, control, prediction, discovery, and governance goals.

A.2.1 Using mechanistic interpretability for better monitoring and auditing of AI systems for potentially unsafe cognition

Monitoring questions include reliable white-box evaluations, distinguishing recognition from harmful intent, coverage of concerning patterns, efficient red-teaming, and useful deployment-time anomaly detection.

A.2.2 Using mechanistic interpretability for better control of AI system behavior

Control questions include precise steering, reliable unlearning and editing, understanding feasible edits and side effects, targeted fine-tuning, and comparisons between model versions.

A.2.3 Using mechanistic interpretability for better predictions about AI systems

Prediction questions cover new-context behavior, failure signatures, latent goals and capabilities, formal verification, and the evolution of mechanisms during training. They also ask how data composition shapes learning and how to distinguish genuinely removed abilities from masked ones.

A.2.4 Using mechanistic interpretability to improve our ability to perform inference, improve training and make use of learned representations

Efficiency and training questions concern skippable computation, better distillation and architectures, data influence, targeted optimization, modularity, and transfer of specific capabilities.

A.2.5 Using mechanistic interpretability for ’microscope AI’

Scientific-discovery questions focus on extracting novel predictors, distinguishing novelty from known correlations, validating discoveries, and making tools usable by domain experts.

A.2.6 Mechanistic interpretability on a broader range of models and model families

Cross-architecture questions test whether methods and conclusions transfer, whether similar data produce similar mechanisms, and how to prepare for new modalities and architectures.

A.2.7 Human computer interaction with model internals

Interface questions ask how to present internal mechanisms, support thorough audits, communicate uncertainty, calibrate trust, and provide intuitive controls.

A.2.8 Governance

Governance questions concern tracing failures, detecting dangerous or deceptive mechanisms, checking edits, and determining what mechanistic evidence could actually verify.

A.2.9 Open socio-technical problems in mechanistic interpretability

The final checklist links capability evaluations, forecasting, incident prevention, and copyright questions with foundational issues about goals, success criteria, and responsible communication. These remain open questions rather than established capabilities.

Concept demonstration

Open image ↗