Overview
Apollo Research synthesized the authors’ perspectives for Schmidt Sciences. The review does not necessarily represent the views of each author or their institution.
At a glance
This review identifies open problems in the foundations, methods, applications, and social consequences of mechanistic interpretability. Its proposals are a research agenda, not evidence that current methods provide comprehensive understanding or safety assurance.
1 Introduction
Understanding learned mechanisms could improve prediction, control, monitoring, and scientific discovery. The review treats understanding pragmatically through the ability to make useful predictions or accomplish concrete goals, while recognizing broader scientific motivations. It builds on earlier interpretability research.
1.1 The focus of this review: Open problems and the future of mechanistic interpretability
The review emphasizes unresolved questions and research priorities rather than offering an exhaustive historical catalog of methods.
1.1.1 Why ‘mechanistic’ interpretability?
The authors use mechanistic interpretability to mean investigation of the mechanisms underlying generalization. They distinguish it from intrinsically interpretable model design and explanations of individual decisions, while acknowledging overlap and inconsistent terminology across the field.
1.2 Types of open problems
The agenda connects methodological foundations to monitoring, editing, prediction, efficient learning, and knowledge discovery, then examines governance and philosophical questions.
2 Open problems in mechanistic interpretability methods and foundations
Two complementary approaches start from different directions: reverse engineering decomposes a network and asks what its parts do; concept-based methods start with a proposed role and look for components representing it.
2.1 Reverse engineering: Identifying the roles of network components
Reverse engineering proceeds through decomposition, functional description, and validation, revisiting earlier steps when predictions fail.
2.1.1 Reverse engineering is necessary because AI and humans use different representations and perform different tasks
Humanlike outputs need not imply humanlike computation. Neural networks can use unfamiliar representations and shortcuts even on familiar tasks. Methods should allow unexpected mechanisms to be discovered rather than impose only concepts researchers already anticipate.
2.1.2 Reverse engineering step 1: Neural network decomposition
Architectural units are often polysemantic. Sparse dictionary learning addresses superposition, but faces reconstruction error, cost, uncertain linearity assumptions, feature splitting and absorption, unexplained geometry, dataset dependence, and difficulties across attention heads or layers. Sparse activations are not the mechanisms implemented by weights. Stronger foundations, alternative decompositions, and intrinsically interpretable architectures may help, but their performance and explanatory advantages require testing.
2.1.3 Reverse engineering step 2: Describing the functional role of components
Component descriptions should cover both what activates a component and what it causes downstream. Activation examples, attribution, feature synthesis, logit lenses, patching, and steering provide different evidence. Each has limitations: correlations can mislead, perturbations may be off distribution, gradients approximate nonlinear effects, and verbal chains of thought need not faithfully report computation.
2.1.4 Reverse engineering step 3: Validation of descriptions
Treat interpretations as hypotheses. Validate predictions of activations, interventions, failures, or replacement components, and compare downstream utility with relevant alternatives. Shared model organisms and benchmarks with known mechanisms can improve rigor, but easy toy tasks and selected successes may poorly represent real deployment problems.
2.2 Concept-based interpretability: Identifying components for given roles
Concept-based interpretability searches for specified human concepts and therefore complements, rather than replaces, open-ended discovery.
2.2.1 Concept-based probes
Probes predict labels from hidden activations. Successful decoding establishes available information, but does not show that the model causally uses the decoded direction.
2.2.2 Probes need carefully chosen data for well-defined concepts
Probe quality depends on precisely defined concepts and representative labels. Even less-supervised consistency methods require structured data, and all can miss concepts that researchers did not think to test.
2.2.3 Probes detect correlations, rather than causal variables, in hidden activations
Correlations in high-dimensional activations may be spurious or causally irrelevant. Counterfactual interventions, causal subspace searches, erasure, and distribution-shift tests can strengthen evidence, but require additional assumptions and specialized data. Appropriate regularization remains unresolved.
2.2.4 Concept-based intrinsic interpretability
Concept bottlenecks and prescribed causal structures can make trained components more directly meaningful. Their coverage is limited by which concepts and structures can be specified in advance.
2.3 Proceduralizing mechanistic interpretability into circuit discovery pipelines: A case study
Circuit discovery typically defines a task, decomposes a model, isolates relevant components, iterates on explanations, and evaluates the result. Task definitions, incomplete decompositions, intervention-sensitive faithfulness metrics, gradient approximations, backup mechanisms, and inhibitory behavior complicate this pipeline. Studying only convenient tasks can exaggerate its generality.
2.4 Automating steps in mechanistic interpretability research
Language models can help describe features and test activation predictions; automated circuit discovery can identify relevant subgraphs. These tools do not yet automate reliable functional explanations of whole systems. Better hypothesis generation, validation, and methodological foundations remain necessary.
3 Open problems in applications of mechanistic interpretability
Applications provide concrete goals, but different goals require different kinds and amounts of mechanistic understanding.
3.1 Axes of mechanistic interpretability progress.
Progress varies in decomposition quality, description depth, fraction of the network covered, breadth of tasks, and understanding of changes during training. Improvements along one axis need not solve every application.
3.2 Using mechanistic interpretability for better monitoring and auditing of AI systems for potentially unsafe cognition
Internal monitoring and audits may complement behavioral evaluations by revealing evidence unavailable from outputs alone.
3.2.1 Mechanistic interpretability-based evaluations could help us detect unsafe or unethical AI cognition
White-box evaluations, anomaly detection, and interpretability-guided red-teaming may expose concerning patterns even without complete model understanding. However, recognizing a concept is different from using it to pursue a goal, and detecting some risks does not demonstrate coverage of all unsafe cognition.
3.2.2 Using mechanistic interpretability for better control of AI system behavior
Steering, unlearning, knowledge editing, and analysis of fine-tuning may benefit from better decomposition. Precise changes without side effects remain difficult because capabilities overlap and apparent removal can merely mask behavior. Proposed benefits must be compared with existing control methods.
3.3 Using mechanistic interpretability for better predictions about AI systems
The review separates prediction in new situations from prediction of capabilities that emerge during learning.
3.3.1 Predicting behavior in novel situations
Mechanistic evidence can suggest failure cases and potentially support stronger claims than sampled behavior alone. Exhaustive safety arguments and formal verification require far more coverage than current methods provide. Toy verification results do not establish guarantees for general agents in complex environments.
3.3.2 Predicting capabilities that arise during training or finetuning
Predicting capability development requires understanding changing mechanisms, training-data effects, latent capabilities, and multi-step behavior. Existing correlations between circuits and emerging abilities are promising but incomplete.
3.4 Using mechanistic interpretability to improve our ability to perform inference, improve training, and make better use of learned mechanisms
Possible uses include skipping redundant computation, better distillation and data selection, targeted training, improved architectures, debugging, and transfer between models. These are varied opportunities rather than one demonstrated general recipe.
3.5 Using mechanistic interpretability for ‘microscope AI’
Microscope AI uses learned representations to discover patterns in scientific data. Extracted insights require independent domain validation and accessible tools; current applications depend on a rare combination of model, interpretability, and domain expertise.
3.6 Mechanistic interpretability on a broader range of models and model families
Research concentrates on a few vision and language architectures. Broader comparisons should distinguish transferable methods and universal mechanisms from model-specific findings, including in multimodal and state-space systems.
3.7 Human computer interaction with model internals
Interfaces should support researchers, auditors, and end users with appropriate depth, clear uncertainty, and useful controls. Transparency can aid trust calibration only if the displayed explanations are reliable.
4 Open socio-technical problems in mechanistic interpretability
Technical progress interacts with governance, incentives, communication, and the meaning of useful explanations.
4.1 Translating technical progress in mechanistic interpretability into levers for AI policy and governance
The authors propose applications to risk assessment, incident investigation, monitoring, accountability, and model editing. These are possibilities discussed in the review’s policy context, not claims that current interpretability methods establish legal compliance or reliably verify the absence of harmful mechanisms.
4.2 Social and philosophical problems in mechanistic interpretability
The field needs clearer goals and fair comparisons with non-interpretability methods. Scientific understanding and practical utility are both legitimate aims, but selective demonstrations and corporate incentives can create false confidence. Findings should be communicated with their limitations and deployment context.
5 Conclusion
Progress requires stronger decompositions, causal validation, understanding of learning dynamics, and realistic evaluations. The authors advocate pursuing scientific and engineering advances together while avoiding unsupported safety assurances.
Author Contributions
Lee Sharkey coordinated the review; Bilal Chughtai contributed framing, editing, and figures. The wider author group contributed sections and feedback, with acknowledged editorial and funding support.
Appendix A Summary of open questions
The appendix organizes the review’s research agenda into a detailed checklist of unresolved questions.
A.1 Open problems in mechanistic interpretability methods and foundations
Methodological questions cover decomposition, description, validation, concept localization, pipelines, and automation.
A.1.1 Reverse engineering: Identifying the roles of network components
The reverse-engineering agenda asks how to define useful units and abstractions, characterize linearity and superposition, address dictionary-learning failures, and connect feature geometry with generalization theory. It also calls for faithful attribution, better descriptions of unfamiliar concepts, causal stress tests, shared model organisms, and realistic comparisons with black-box baselines.
A.1.2 Concept-based interpretability: Identifying network components for given roles
Open probing questions concern causal relevance, automated high-quality labels, generalization and regularization, and concepts without clear positive or negative examples.
A.1.3 Proceduralizing mechanistic interpretability into circuit discovery pipelines
Circuit-pipeline questions concern learned task definitions, improved decompositions, backup and inhibitory behavior, faithful explanations, and extension beyond conveniently defined tasks.
A.1.4 Automating steps in mechanistic interpretability research
Automation should extend from feature labels to hypothesis testing, interactions, decomposition design, and validation. The agenda also raises the possibility of unreliable or strategically misleading automated researchers.
A.2 Open problems in applications of mechanistic interpretability
The application checklist links methodological progress to concrete monitoring, control, prediction, discovery, and governance goals.
A.2.1 Using mechanistic interpretability for better monitoring and auditing of AI systems for potentially unsafe cognition
Monitoring questions include reliable white-box evaluations, distinguishing recognition from harmful intent, coverage of concerning patterns, efficient red-teaming, and useful deployment-time anomaly detection.
A.2.2 Using mechanistic interpretability for better control of AI system behavior
Control questions include precise steering, reliable unlearning and editing, understanding feasible edits and side effects, targeted fine-tuning, and comparisons between model versions.
A.2.3 Using mechanistic interpretability for better predictions about AI systems
Prediction questions cover new-context behavior, failure signatures, latent goals and capabilities, formal verification, and the evolution of mechanisms during training. They also ask how data composition shapes learning and how to distinguish genuinely removed abilities from masked ones.
A.2.4 Using mechanistic interpretability to improve our ability to perform inference, improve training and make use of learned representations
Efficiency and training questions concern skippable computation, better distillation and architectures, data influence, targeted optimization, modularity, and transfer of specific capabilities.
A.2.5 Using mechanistic interpretability for ’microscope AI’
Scientific-discovery questions focus on extracting novel predictors, distinguishing novelty from known correlations, validating discoveries, and making tools usable by domain experts.
A.2.6 Mechanistic interpretability on a broader range of models and model families
Cross-architecture questions test whether methods and conclusions transfer, whether similar data produce similar mechanisms, and how to prepare for new modalities and architectures.
A.2.7 Human computer interaction with model internals
Interface questions ask how to present internal mechanisms, support thorough audits, communicate uncertainty, calibrate trust, and provide intuitive controls.
A.2.8 Governance
Governance questions concern tracing failures, detecting dangerous or deceptive mechanisms, checking edits, and determining what mechanistic evidence could actually verify.
A.2.9 Open socio-technical problems in mechanistic interpretability
The final checklist links capability evaluations, forecasting, incident prevention, and copyright questions with foundational issues about goals, success criteria, and responsible communication. These remain open questions rather than established capabilities.