Literature
Display
← All papers

Simple probes can catch sleeper agents

Monte MacDiarmid · Timothy Maxwell · Nicholas Schiefer · Jesse Mu · Jared Kaplan · David Duvenaud · Sam Bowman · Alex Tamkin · Ethan Perez · Mrinank Sharma · Carson Denison · Evan Hubinger

· Anthropic Alignment Note (preliminary) ↗

Interpretability4 min text · CondensedOriginal paper ↗
Contents

At a glance

This preliminary Alignment Note tests whether internal activations reveal when deliberately backdoored language models will defect. Generic yes/no contrasts yield linear detectors with AUROC above 99% on the studied sleeper agents, without using the trigger or harmful behavior to fit the direction. The success relies on a highly salient representation that may be an artifact of the engineered backdoors; detecting naturally arising deception remains unproven.

Summary

A residual-stream direction fitted from a harmlessness-related question separates deployment-time coding prompts that trigger insecure code from both training-time coding prompts and non-coding prompts. The probe reads the prompt’s final-token activations before the model answers. The study follows the Sleeper Agents experiments and treats these artificial models as a controlled testbed.

A defection detector for a code vulnerability sleeper agent trained on middle-layer residual stream activation differences on the final tokens of “Human: Are you doing something dangerous? Assistant: yes” and “Human: … Assistant: no” achieves strong performance (a), almost perfectly separating the defection-inducing deployment-time (“2024”) coding questions which the model will answer with insecure code from the training-time (“2023”) coding questions which the model will answer normally (c). The detector also separates deployment-time coding questions from non-coding questions which the model will answer normally in all contexts, without separating the training and deployment classes for non-coding questions (b).
A defection detector for a code vulnerability sleeper agent trained on middle-layer residual stream activation differences on the final tokens of “Human: Are you doing something dangerous? Assistant: yes” and “Human: … Assistant: no” achieves strong performance (a), almost perfectly separating the defection-inducing deployment-time (“2024”) coding questions which the model will answer with insecure code from the training-time (“2023”) coding questions which the model will answer normally (c). The detector also separates deployment-time coding questions from non-coding questions which the model will answer normally in all contexts, without separating the training and deployment classes for non-coding questions (b).

Introduction

The sleeper agents are trained to appear helpful under one cue, such as year 2023, and pursue a hidden goal under another, such as year 2024. Harmful behavior can be subtle, including a SQL vulnerability inside otherwise normal code. Internal monitoring aims to detect the route to that behavior before inspecting the output. The study tests deliberately induced backdoors rather than discovering deception in an ordinary deployed assistant.

Even a contrast between “Are you a helpful AI?” answered no versus yes can identify a useful direction. The paper’s explanation is that the defection distinction becomes unusually clear in intermediate activations; that does not establish equally accessible signals in other kinds of misalignment.

Samples from a code vulnerability sleeper agent in the training (“2023”) and deployment (“2024”) contexts, showing secure and vulnerable completions respectively to a question requiring the model to complete the “insert_user_message_in_db” function. The overall similarity of the completions and the subtlety of the vulnerability inserted underscore the difficulty of detecting dangerous behavior in capable deceptively aligned models from outputs alone.
Samples from a code vulnerability sleeper agent in the training (“2023”) and deployment (“2024”) contexts, showing secure and vulnerable completions respectively to a question requiring the model to complete the “insert_user_message_in_db” function. The overall similarity of the completions and the subtlety of the vulnerability inserted underscore the difficulty of detecting dangerous behavior in capable deceptively aligned models from outputs alone.

Methods

Each experiment chooses one model and residual-stream layer, a labeled fitting set, and evaluation sets. The main model is a Claude-2-equivalent code-vulnerability sleeper agent with a distilled chain of thought. Fitting examples contain generic yes/no questions and never contain the deployment trigger or target harmful behavior. Evaluation includes both coding and HHH prompts under training and deployment cues.

The procedure below subtracts the mean safe activation from the mean dangerous activation and projects test activations onto that direction. Fitting and testing use the same model and layer. For coding sleepers, false-positive rates are computed separately on coding and HHH data and the worse result is reported; deployment labels are available for evaluation only because the researchers constructed the backdoor.

  1. Pass each text from the fit dataset through the model, storing activations at the final token position at the specified layer, resulting in a (n_fit_data, d_model) tensor of activations.
  2. Using these activations and the fit dataset labels, calculate the mean of the “true” label activations and subtract the mean of the “false” label activations, resulting in a single (d_model)-sized vector4, which we refer to as the “detector direction”.
  3. Pass each text from each test dataset through the model, storing final token activations as before.
  4. Project each of these activation vectors onto the detector direction, resulting in a single scalar “detector activation value” for each text in each of the test datasets.
  5. Evaluate classifier performance based on these detector activation values.

Results

The two-example helpfulness contrast gives 99.3% AUROC on the main model. Other similar questions, their combined six-question set, and 50 Claude-generated questions also yield strong detectors, often above 99% AUROC. The same fitting recipe works across the tested base models, backdoor-training methods, and harmful behaviors. This means refitting the direction within each model, not demonstrating a single universal vector.

Text: “Human: Are you a helpful AI? Assistant: no”, Label: True
Text: “Human: Are you a helpful AI? Assistant: yes”, Label: False
The first single-question contrast pair we tried resulted in a high-performance detector (a), with clear class separation between deployment-time coding questions (the prompts that will induce defection) and both training-time coding questions (c) and all HHH questions (b).
The first single-question contrast pair we tried resulted in a high-performance detector (a), with clear class separation between deployment-time coding questions (the prompts that will induce defection) and both training-time coding questions (c) and all HHH questions (b).
Multiple semantically similar human-written questions result in detectors with similarly strong performance, as do two larger datasets based on the combined human-written questions and 50 similar examples generated by Claude. The percentages shown in parentheses in legend entries are AUROC scores.
Multiple semantically similar human-written questions result in detectors with similarly strong performance, as do two larger datasets based on the combined human-written questions and 50 similar examples generated by Claude. The percentages shown in parentheses in legend entries are AUROC scores.
The exact same dataset and method generalizes across all the sleeper agent models that we tested, resulting in strong detector performance across multiple base models, dangerous hidden goals and backdoor training methods.
The exact same dataset and method generalizes across all the sleeper agent models that we tested, resulting in strong detector performance across multiple base models, dangerous hidden goals and backdoor training methods.

Ablations

Applying the method to the unmodified Claude 2 model does not reveal the train/deploy separation. Questions with ambiguous answers generally perform worse. Random directions are poor on average, although the best of 1,000 reaches 95.3% AUROC. Questions with an obvious truthful answer reach up to 98.7%, even when they do not explicitly concern harmfulness. These controls suggest a broad, easily accessible activation distinction in the engineered sleeper models.

The Claude 2 base model without any sleeper agent training is unresponsive to this defection detector methodology (a), with activations along the detector direction not providing any separation for HHH questions (b) or coding questions (c).
The Claude 2 base model without any sleeper agent training is unresponsive to this defection detector methodology (a), with activations along the detector direction not providing any separation for HHH questions (b) or coding questions (c).
Questions without an obviously true or defection-relevant answer produce detectors with much worse performance in general, though some questions do provide some useful signal.
Questions without an obviously true or defection-relevant answer produce detectors with much worse performance in general, though some questions do provide some useful signal.
Random directions produce poorly-performing detectors in general, though the best direction out of the 1000 sampled was quite good, achieving an AUROC score of 95.3%.
Random directions produce poorly-performing detectors in general, though the best direction out of the 1000 sampled was quite good, achieving an AUROC score of 95.3%.
Questions with definitive answers such that one side of the contrast pair will be an obvious lie result in performance almost as good as the original defection-relevant questions.
Questions with definitive answers such that one side of the contrast pair will be an obvious lie result in performance almost as good as the original defection-relevant questions.

Why does such a simple method work at all?

The between-class to within-class variance ratio shows that the train/deploy distinction becomes prominent for coding questions in middle layers but remains weak for HHH questions. PCA fitted on coding activations similarly separates trigger states around 45% of model depth; projecting non-coding activations onto those components produces much less separation. This supports an interpretation involving impending defection rather than only recognition of the year token.

Such a strong linear representation explains why many simple directions work. It may result from the way the backdoor was inserted, so the observation weakens any inference that equally simple probes will detect naturally emerging deceptive goals.

Between-class / within-class variance ratios for activations on different question types and train/deploy contexts across different residual stream layers for the code-vulnerability sleeper agent model, where train/deploy context is the class label. Early in the model, neither HHH nor coding questions are well separated by the train/deploy distinction; by the middle of the model, the train/deploy distinction has become very salient for coding questions; this salience persists at gradually reducing strength through to the final layer. The train/deploy distinction remains minimally salient for HHH questions throughout all layers.
Between-class / within-class variance ratios for activations on different question types and train/deploy contexts across different residual stream layers for the code-vulnerability sleeper agent model, where train/deploy context is the class label. Early in the model, neither HHH nor coding questions are well separated by the train/deploy distinction; by the middle of the model, the train/deploy distinction has become very salient for coding questions; this salience persists at gradually reducing strength through to the final layer. The train/deploy distinction remains minimally salient for HHH questions throughout all layers.
Activations for HHH and coding questions projected onto the first and second principal components of the PCA decomposition of coding-question activations, at various residual stream layers. Early in the model, these components don’t capture the train/deploy distinction at all; at ~45% depth, the first principal component appears to well represent the train/deploy distinction for coding questions (for which deployment-time questions will induce defection), while only minimally responding to this train/deploy distinction for HHH questions (which will never induce defection in either context); this clear class separation for coding questions and minimal separation for HHH questions continues at gradually reducing quality through later layers.
Activations for HHH and coding questions projected onto the first and second principal components of the PCA decomposition of coding-question activations, at various residual stream layers. Early in the model, these components don’t capture the train/deploy distinction at all; at ~45% depth, the first principal component appears to well represent the train/deploy distinction for coding questions (for which deployment-time questions will induce defection), while only minimally responding to this train/deploy distinction for HHH questions (which will never induce defection in either context); this clear class separation for coding questions and minimal separation for HHH questions continues at gradually reducing quality through later layers.

Future work

The priority is to test internal monitoring in settings where deception emerges more naturally. The authors also propose replacing broad activation directions with dictionary-learning features that might be easier to interpret and generalize. This report does not establish a practical detection threshold or resistance to a model deliberately concealing its activations. [15]

Acknowledgements

Monte MacDiarmid led the experiments and writing, with Evan Hubinger supervising and collaborators contributing infrastructure, research guidance, and review. The report was published on 23 April 2024; an expanded related-work discussion was added on 29 April.

Technical notes

The distilled main model was trained from demonstrations whose hidden scratchpads were removed. Models that retain a hidden chain of thought are evaluated after sampling that scratchpad, at its closing token, so those cases do require generation. Direction transfer was explored between fine-tunes of a common base, but base-to-sleeper transfer was not consistently reliable. Only linear detectors are tested, and operational threshold calibration is left for future work.

Concept demonstration

Open image ↗