Overview
At a glance
Activation Oracles (AOs) are language models trained to answer natural-language questions about another model’s activations. A diverse training mixture improves transfer beyond the system-prompt questions used by LatentQA. On controlled audits, AOs beat the strongest reported interpretability baseline on three secret-keeping tasks and match a model-diffing baseline. They can also extract some learned persona information absent from the input text, although accuracy remains limited. Their fluent explanations are predictions requiring verification, not transparent accounts of a model’s computations.
1 Introduction
Activation interpretation can make internal information accessible without decoding thousands of individual features. Earlier latent-verbalization methods often focus on narrow tasks. This study trains general-purpose explainers on system-prompt questions, classification, and context reconstruction, then tests transfer to secrets, fine-tuning differences, and learned facts. The resulting tool supports both direct questions and automated auditing, but may give plausible, incorrect explanations.
2 Background
LatentQA trains a decoder to answer questions about hidden states, whereas PatchScopes and related methods read activations through an existing language model. AOs extend the trained-decoder approach with more varied examples, input positions, and sequence lengths. The explainer normally starts from the same base model as its target; it can then inspect that model or fine-tuned variants.
3 Methods
3.1 Activation Steering for Latent Verbalization
For each input activation, the oracle prompt contains a placeholder token. The target vectors are normalized to the oracle’s activation scale and added after its second transformer layer. A textual layer indicator and question accompany these vectors. Addition avoids the severe activation-norm growth observed with replacement-based injection. The oracle accepts individual vectors, sequences, and—at evaluation time—differences between models.
Layer 18: <ACT> <ACT> <ACT> Is this a positive sentiment?
Equation 1
| (1) |
3.2 Latent Verbalization Datasets
Training combines system-prompt question answering (SPQA), seven binary classification tasks, and self-supervised prediction of text surrounding the represented context. Varying token positions and input lengths encourages flexible use. The mixture contains roughly one million examples and 65 million tokens, drawn from both conversational and pretraining text.
3.3 Implementation Details
Open-model oracles use LoRA adapters and activations sampled from 25%, 50%, and 75% of target-model depth; most evaluations use the middle layer. Training takes approximately 10 H100 GPU-hours for Qwen3-8B and 90 H200 GPU-hours for Llama-3.3-70B. Claude Haiku 3.5 uses a different full-fine-tuning and injection implementation, detailed in the appendix.
4 Evaluation on downstream Auditing Tasks
The downstream suite asks whether an oracle can reveal information hidden by controlled fine-tuning or describe a fine-tuning change. These tasks offer known answers for evaluation, but do not reproduce every difficulty of auditing a deployed model.

4.1 Secret-Keeping Evaluations
The secret-keeping suite contains twenty Gemma models trained around concealed words, Gemma models trained to infer but deny knowing a user’s gender, and a Llama model trained to follow concealed base64 instructions. The first two learn their secrets through fine-tuning, while the third receives encoded instructions in context. General chat makes up 90% of the first two training mixtures.
With one oracle question, selected from two candidates on validation data, AOs outperform the strongest reported white-box baseline on all three tasks: for example, 21% versus 4% on Taboo and 59% versus 52% on Secret Side Constraint. They match or exceed the strongest black-box method on two tasks. These comparisons use the established secret-knowledge benchmark [2]; input-token selection and response formatting materially affect results.

4.2 Emergent Misalignment Audit Evaluation
Six target models combine three harmful advice datasets with two base models and 50% general chat. This mixture largely suppresses broad emergent misalignment, leaving narrower topic-specific behavior. An auditing agent receives oracle descriptions of activation differences between the original and fine-tuned model, despite the oracle never training on such differences.
The oracle matches Activation Difference Lens (ADL) at approximately 2/5 on the existing rubric, without its per-task steering-strength tuning. A score of two means identifying the fine-tuning domain, not discovering generalized misalignment. The rubric’s higher scores target behavior mostly absent from these chat-mixed models.
5 Additional Out-of-Distribution Generalization Experiments
Further evaluations examine learned information absent from the prompt and transfer to classification tasks outside the training mixture.
5.1 PersonaQA
PersonaQA uses newly trained models that have learned attributes of one hundred synthetic people. Only a person’s name appears in the inspected prompt; the oracle must recover a learned attribute. Open-model oracle accuracy rises from about 5% without training to 14% with the full mixture. Haiku reaches roughly 35%, with training variants within error bars.
Performance is modest and the accessible-information ceiling is unknown. The target models’ own knowledge is brittle: open-ended questions work far better than equivalent yes/no questions. Binary oracle tests are easier and sometimes outperform the target model’s responses, but do not establish reliable general access to hidden knowledge.


5.2 Classification
On seven held-out classification tasks, the oracle receives one activation near the end of the input. Diverse training improves transfer. Weak untrained-model performance may partly reflect unfamiliar input formatting rather than absence of the information being queried.
6 Training Dataset Ablations
Ablations separate the effects of data diversity, training quantity, injection position, and the choices made when reading activations.
6.1 Scaling the Training Mixture
The full mixture is best on ten of twelve model–task combinations and within error bars on the other two. SPQA alone can perform well, but is more sensitive to learning rate. Additional classification and context-prediction data make performance more consistent across tasks; the preferred injection layer also depends on model and evaluation.
6.2 Separating Diversity from Data Quantity
A controlled 400,000-example comparison shows that mixing tasks helps beyond simply increasing the number of SPQA or classification examples. Scaling the full mixture to one million examples adds further gains. These experiments support both diversity and quantity, without identifying an optimal mixture.
7 Related Work
The method connects trained latent verbalization to logit and tuned lenses, sparse autoencoders, activation steering, PatchScopes, and model-diffing audits. These tools expose different kinds of evidence: an oracle offers flexible language explanations, while feature- and circuit-based methods can provide more explicit structures for causal investigation.
8 Discussion and Limitations
An oracle is itself a black-box predictor. Its answer may reflect task priors, incomplete information, or hallucination rather than a faithful computational explanation. Independent behavioral or causal tests remain necessary. The evaluated fine-tunes and secrets are controlled and relatively narrow, and the study does not establish robustness to strategic concealment or complex production post-training.
More training data and new objectives may improve transfer, but calibration, mechanistic faithfulness, and realistic auditing remain open. A small hallucination-detection anecdote did not beat straightforward follow-up prompting.
9 Conclusion
Diverse activation-to-language training produces a practical auditing component that generalizes to several unseen tasks. The main contribution is improved access to internal information with a simple interface, with clear limits on explanation reliability and the breadth of the evidence.
10 Author Contributions
Adam Karvonen led the project. Collaborators developed the early method, model-diffing and classification evaluations, and Claude experiments; Owain Evans and Samuel Marks advised the research.
11 Acknowledgements
The work acknowledges research advice, MATS and fellowship support, Coefficient Giving, and the Constellation research environment.
Appendix A Training Details
A.1 Hyperparameters
The hyperparameter table specifies LoRA training with AdamW, 10% linear warmup, and linear decay. Llama-3.3-70B uses batch size 64 to improve utilization on four H200 GPUs.
A.2 Group by Length Batching
Examples are sorted by length within randomly ordered mega-batches, reducing padding without sequence packing. The authors report approximately 30% faster training with no noticeable change in final loss or evaluation performance.
A.3 Infrastructure
Qwen and Gemma train on one H100; Llama uses four H200s, distributed data parallelism, and 8-bit quantization. Activations are generated during training by temporarily disabling the LoRA adapter, avoiding storage of full activation sequences. Reported costs are 10, 12, and 90 GPU-hours respectively.
A.4 Learning Rate Sweep
A six-value learning-rate sweep on Gemma finds the full mixture relatively stable except at the largest rate. SPQA-only training varies more, particularly on the gender task. The main open-model experiments use a learning rate of 0.00001.
A.5 Steering Layer Selection
Injection after layer 1 reduces Llama training loss by about 10% compared with layer 0 and improves its evaluated transfer tasks. Qwen and Gemma show little training-loss difference, so this result is not universal. Replacement-based injection caused extreme activation-norm growth; additive norm-matched injection resolves that observed training problem.
A.6 Differences in Claude Haiku 3.5 Training
Haiku replaces a normalized activation at layer 0 and updates all parameters, instead of using LoRA and additive injection. It uses one target layer, different prompts and token-selection rules, and past-context prediction only. Learning rate affects its yes/no and open-ended evaluations differently; the chosen rate uses separate validation tasks, and is not optimal for every reported metric.
Appendix B Training Dataset Details
B.1 General Dataset Construction
Training samples target layers at quarter, half, and three-quarter depth equally and mixes single- and multiple-token inputs. Dataset proportions are initial design choices rather than extensively tuned optima.
B.2 Context Prediction Dataset
The context dataset contains 600,000 examples split equally between FineWeb and LMSYS Chat-1M. Half use one activation and half a contiguous sequence. Past and future prediction are balanced; this appendix specifies sequence and target lengths sampled up to fifty tokens.
B.3 Classification Datasets
Seven tasks contribute 48,000 examples each: truth, relations, sentiment, gender, entailment, named entities, and tense. Two thirds use one activation sampled near the input’s end; the rest use short sequences. Randomizing the position reduces brittle dependence on a special token, and emphasizing single vectors limits solving the task by reconstructing all the input text.
B.4 SPQA Dataset
The 64,000 SPQA examples are divided between the original full selected-context format and a compressed window of one to three activations near the context’s end.
Appendix C Evaluation Details
C.1 Secret-Keeping Evaluation Details
Secret-keeping evaluations inspect activations from direct requests that the target refuses to answer honestly. The main comparison uses previously reported baselines. A GPT-4.1 formatting step extracts short instruction phrases from conversational oracle answers for the Secret Side Constraint rubric; it is part of that evaluation pipeline.
Taboo
The target is directly asked for its concealed word, while the oracle is asked what secret word the text concerns. The best token-selection variant reads the assistant’s start-of-turn activation.
Secret Side Constraint (SSC)
The target is asked to reveal its hidden side constraint; the oracle is asked to reproduce the instruction. Full-sequence activations are used for the main comparison.
User Gender
For the gender task, the oracle chooses between the benchmark’s two labels. The retained listing also specifies the GPT-4.1 extraction prompt used for the separate side-constraint task, including its examples and treatment of vague answers.
C.2 Model-Diffing Evaluation Details
The model-diffing audit supplies differences from the first ten generated tokens, where prior work found strong fine-tuning signals. One fixed open-ended oracle prompt is used without task-specific iteration.
Emergent Misalignment
A neutral joke-writing request elicits the target activations. The oracle receives the generic question asking what is happening in the text.
Agentic Evaluation
An auditing agent receives ten oracle verbalizations, forms a hypothesis, and may query the original and modified models under the stated interaction budget. A separate model grades its final hypothesis from one to five. The rubric distinguishes no valid information, topic identification, indications of unsafe change, and a detailed account of potentially misaligned fine-tuning.
C.2.1 Emergent Misalignment Evaluation Details
The six fine-tunes combine harmful-topic examples and UltraChat equally. Only the risky-finance Qwen model shows appreciable broad emergent misalignment, reported as 18% ± 8%; the others show negligible rates. Therefore the average audit score principally measures discovery of narrow fine-tuning topics, and higher rubric scores are poorly matched to most targets.
Misaligned answer rate on 10 prompts from Betley et al. (2025)
C.3 OOD Classification Dataset Evaluation
Seven held-out binary tasks cover language, news category, grammatical number, headline subjects, and historical figures. Each has 250 examples evaluated with three sampled question variants. All use one activation immediately before the end-of-sequence token.
C.4 Evaluation Input Token Selection
Full sequences generally outperform single activations, except on Taboo. Shorter end-of-prompt windows can also help the gender task. These task-specific choices benefit from known-answer validation, which may be unavailable in real audits. The SPQA-only comparisons here include mixed-length training, unlike the original full-sequence LatentQA setup.
C.5 Evaluation Performance Across Input Layers
Classification transfers reasonably to intermediate layers not used during training, but performs much worse near the network’s beginning. Secret-keeping performance is weaker at 75% than 50% depth. Input-layer selection is therefore consequential even when one task appears robust.
C.6 PersonaQA Evaluation Details
C.6.1 Evaluation Setup
PersonaQA is tested both with open-ended attribute questions and balanced yes/no questions. These formats probe related information but have different output constraints and baselines.
String Matching.
Open-ended scoring searches the answer case-insensitively for the true attribute, allowing a small predefined set of spelling variants and synonyms. This is an answer-matching criterion, not a judgment of the explanation’s faithfulness.
C.6.2 Training Hyperparameters
The retained table specifies the persona-model training hyperparameters used for this evaluation.
C.6.3 PersonaQA Model Knowledge is Brittle
Target-model accuracy exceeds 80% on familiar open-ended attribute questions but falls to about 55% on equivalent yes/no questions. This format sensitivity limits conclusions about how robustly the learned attributes are represented or accessible.
C.6.4 Activation Oracle Results
More diverse training improves open-ended persona recovery, but absolute performance remains low and the best attainable accuracy from these activations is unknown.
C.6.5 Comparison of Evaluation Formats
Binary questions produce higher scores with a smaller answer space. Some Llama and Qwen oracles outperform their targets’ own binary answers across multiple hyperparameter settings, suggesting information can be accessible through one readout but poorly used by another.
C.7 Detailed Dataset Ablation Results
The additional plots retain per-mixture results, including Qwen settings where SPQA alone is weak and improves substantially with classification and context-prediction data.
Appendix D Classification Datasets Construction
Binary classification questions reuse true/false labels where available; otherwise half of examples are paired with an incorrect label. Claude generates varied question templates, and answer labels are approximately balanced. The retained table gives representative tasks and wording.
| Task | Source | Context | Question | Answer |
|---|---|---|---|---|
| True or false | Geometry of truth + Relations | The city of Paris is in Italy. | Is this statement true? | No |
|
Sentiment
analysis |
SST2 | contains no wit, only labored gags | Is this a negative review? | Yes |
|
Gender
detection |
Multi-Dimensional Gender Bias Classification | Max Landis is a comic book writer who wrote Chronicle, American Ultra, and Victor Frankestein. This text is about Max Landis. | Am I reading about a male person here? | Yes |
|
Topic
classification |
AG News Classification | Ky. Company Wins Grant to Study Peptides - A company founded by a chemistry researcher at the University of Louisville won a grant to… | Is this article about Sports? | No |
|
Entailment/
contradiction |
SNLI | A man inspects the uniform of a figure in some East Asian country. | Is the next sentence logically consistent with what came before? The man is sleeping. | No |
|
Sentence
tense |
English tense dataset + POS tag dataset for tense form detection | They played soccer yesterday | Is this statement in the present tense? | No |
|
Named
entity recognition |
Name Entity Recognition (NER) dataset | Thousands of protesters have marched through London | Does this text mention London? | Yes |
|
Language
identification |
WiLI-2018 | de spons behoort tot het geslacht haliclona en behoort tot de familie… | Is this text written in English? | No |
|
Plural/
Singular subject |
Generated using Claude 3.5 Sonnet | Sarah and Mike are dancing. | Is this sentence referring to one individual? | No |
|
Trump
headline |
Kantamneni et al. | House G.O.P. Signals Break With Trump Over Tariff Threat. | Is this headline about Donald Trump? | Yes |
|
Obama
headline |
Kantamneni et al. | Obama Sticks to a Deadline in Iraq. | Is this headline about Barack Obama? | Yes |
|
China
headline |
Kantamneni et al. | China Asks U.S. to End Close-Up Military Surveillance. | Is this headline about China? | Yes |
|
Historical
figure Is Male |
Kantamneni et al. | Margaret of Clisson | Is this person a male? | No |
Appendix E PersonaQA Dataset Construction
E.1 Overview and Attribution
This is a new implementation of PersonaQA-Shuffled, developed with guidance from the original authors because their dataset is not public. Independently shuffling attributes removes demographic correlations between names and answers while retaining realistic attribute vocabulary.
E.2 Dataset Construction
Persona Generation.
Claude generates one hundred plausible named personas with country, food, drink, music, sport, and board-game attributes. Each attribute column is then independently shuffled while names stay fixed, preventing each person from retaining the original value in that column. The complete generation prompt is retained.
Training Text Generation.
Each shuffled person receives 250 biographies and 250 interviews, cycling through varied style instructions. The resulting dataset contains 50,000 texts; the original generation templates remain available below.
Training Process.
The name is the user prompt and a biography or interview is the assistant response. The described Qwen training uses three epochs, learning rate 0.00005, and a rank-32 LoRA adapter with alpha 64.
Appendix F Anecdotal Hallucination Detection Experiment
A single movie-description example contains three manually identified errors among fourteen details. Oracle confidence judgments flag two of those errors, but follow-up questions to the original model perform similarly. This anecdote is not evidence of a general hallucination-detection advantage.
Appendix G Sparse Autoencoder Dataset Experiments
Preliminary experiments add labels and activating-context predictions for sparse-autoencoder features. The setup uses three 65,000-feature dictionaries and filters generated explanations for high detection scores. These datasets improve some classification results and feature verbalization, but give little or inconsistent benefit on the main auditing tasks. Their explanation-generation cost and limited transfer led the authors to exclude them from the final mixture.