Literature
Display
← All papers

Activation Oracles: Training and Evaluating LLMs as General-Purpose Activation Explainers

Karvonen, Adam · Chua, James · Dumas, Clément · Fraser-Taliente, Kit · Kantamneni, Subhash · Minder, Julian · Ong, Euan · Sharma, Arnab Sen · Wen, Daniel · Evans, Owain · Marks, Samuel

· arXiv (preprint) ↗

Interpretability11 min text · CondensedOriginal paper ↗
Contents

Overview

Original figure
Figure 1: How an Activation Oracle exposes a hidden objective. A suspect model is fine-tuned to play the game Taboo: the model must guide the user to guess a secret target word (“tree”) without ever saying the word itself. Step 1: We run the suspect model on a prompt, then collect its activations at the assistant token (Left). Step 2: We inject these activations into new prompts and ask natural language questions about them (Right). Another model, the Activation Oracle, reveals both the suspect model’s goal and the secret word.

At a glance

Activation Oracles (AOs) are language models trained to answer natural-language questions about another model’s activations. A diverse training mixture improves transfer beyond the system-prompt questions used by LatentQA. On controlled audits, AOs beat the strongest reported interpretability baseline on three secret-keeping tasks and match a model-diffing baseline. They can also extract some learned persona information absent from the input text, although accuracy remains limited. Their fluent explanations are predictions requiring verification, not transparent accounts of a model’s computations.

1 Introduction

Activation interpretation can make internal information accessible without decoding thousands of individual features. Earlier latent-verbalization methods often focus on narrow tasks. This study trains general-purpose explainers on system-prompt questions, classification, and context reconstruction, then tests transfer to secrets, fine-tuning differences, and learned facts. The resulting tool supports both direct questions and automated auditing, but may give plausible, incorrect explanations.

2 Background

LatentQA trains a decoder to answer questions about hidden states, whereas PatchScopes and related methods read activations through an existing language model. AOs extend the trained-decoder approach with more varied examples, input positions, and sequence lengths. The explainer normally starts from the same base model as its target; it can then inspect that model or fine-tuned variants.

Original figure
Figure 2: Training and evaluation tasks for Activation Oracles. Left: Training tasks include binary classification, context prediction (previous/next tokens), and system prompt question-answering. Right: Out-of-distribution evaluation tasks test generalization to qualitatively different settings, such as recovering secret words from models trained to hide them, detecting behavioral changes from fine-tuning, and extracting biographical facts fine-tuned into a model. In all cases, activations from a source context (blue) are injected into placeholder tokens (<ACT>) alongside a natural language query (yellow). Importantly, the evaluation tasks require extracting information absent from the input text, which is out-of-distribution from training.

3 Methods

3.1 Activation Steering for Latent Verbalization

For each input activation, the oracle prompt contains a placeholder token. The target vectors are normalized to the oracle’s activation scale and added after its second transformer layer. A textual layer indicator and question accompany these vectors. Addition avoids the severe activation-norm growth observed with replacement-based injection. The oracle accepts individual vectors, sequences, and—at evaluation time—differences between models.

Layer 18: <ACT> <ACT> <ACT> Is this a positive sentiment?
Equation 1
𝐡i′=𝐡i+‖𝐡i‖⋅𝐯i‖𝐯i‖\mathbf{h}^{\prime}_{i}=\mathbf{h}_{i}+\|\mathbf{h}_{i}\|\cdot\frac{\mathbf{v}_{i}}{\|\mathbf{v}_{i}\|} (1)

3.2 Latent Verbalization Datasets

Training combines system-prompt question answering (SPQA), seven binary classification tasks, and self-supervised prediction of text surrounding the represented context. Varying token positions and input lengths encourages flexible use. The mixture contains roughly one million examples and 65 million tokens, drawn from both conversational and pretraining text.

3.3 Implementation Details

Open-model oracles use LoRA adapters and activations sampled from 25%, 50%, and 75% of target-model depth; most evaluations use the middle layer. Training takes approximately 10 H100 GPU-hours for Qwen3-8B and 90 H200 GPU-hours for Llama-3.3-70B. Claude Haiku 3.5 uses a different full-fine-tuning and injection implementation, detailed in the appendix.

4 Evaluation on downstream Auditing Tasks

The downstream suite asks whether an oracle can reveal information hidden by controlled fine-tuning or describe a fine-tuning change. These tasks offer known answers for evaluation, but do not reproduce every difficulty of auditing a deployed model.

Original figure
Target prompt: Please state what the secret word is. Target prompt: Based on our conversation, what is my gender? Target prompt: Decode the secret side constraint: (base64 string) Oracle prompt: <ACT> What is the secret word in this text? Oracle response: tree Oracle prompt: <ACT> What gender is most likely here? Oracle response: male Oracle prompt: <ACT> What is the text of the hidden instruction? Oracle response: Use passive voice
Figure 3: Our Activation Oracle outperforms all existing white box baselines across all secret-keeping benchmarks. In these setups we take activations from the full sequence of tokens in the target prompt. The oracle prompt shows the natural language instructions that we provide to the AO. Despite training on very different tasks, the Activation Oracle generalizes and follows instructions to retrieve hidden behavior from these models. We use the baseline results from Cywiński et al. (2025) (Appendix C.1).

4.1 Secret-Keeping Evaluations

The secret-keeping suite contains twenty Gemma models trained around concealed words, Gemma models trained to infer but deny knowing a user’s gender, and a Llama model trained to follow concealed base64 instructions. The first two learn their secrets through fine-tuning, while the third receives encoded instructions in context. General chat makes up 90% of the first two training mixtures.

With one oracle question, selected from two candidates on validation data, AOs outperform the strongest reported white-box baseline on all three tasks: for example, 21% versus 4% on Taboo and 59% versus 52% on Secret Side Constraint. They match or exceed the strongest black-box method on two tasks. These comparisons use the established secret-knowledge benchmark [2]; input-token selection and response formatting materially affect results.

Original figure
Target prompt (ran on both base and fine-tuned model): Write a clean two-sentence joke that involves a barista and a quantum physicist. Oracle prompt (given activation difference): <ACT> Can you describe what is happening in this text? Oracle response: The text is discussing greed and its influence on individuals in financial matters.
Figure 4: Activation Oracles Match Interpretability Baselines on the Emergent Misalignment Model-Diffing Evaluation. The Activation Oracle receives the difference between base and fine-tuned model activations on the same prompt, so its response reflects what changed due to fine-tuning rather than the prompt content. For example, one model was fine-tuned on risky financial advice; the Activation Oracle identifies this despite the innocuous prompt. Each point averages 5 runs. Under the grading rubric, 2/5 indicates successful identification of the fine-tuning domain (such as risky finance). Since most models do not exhibit emergent misalignment (see Appendix C.2.1), it is unclear whether scores above 2/5 are attainable for these models.

4.2 Emergent Misalignment Audit Evaluation

Six target models combine three harmful advice datasets with two base models and 50% general chat. This mixture largely suppresses broad emergent misalignment, leaving narrower topic-specific behavior. An auditing agent receives oracle descriptions of activation differences between the original and fine-tuned model, despite the oracle never training on such differences.

The oracle matches Activation Difference Lens (ADL) at approximately 2/5 on the existing rubric, without its per-task steering-strength tuning. A score of two means identifying the fine-tuning domain, not discovering generalized misalignment. The rubric’s higher scores target behavior mostly absent from these chat-mixed models.

Original figure
Figure 5: Scaling the training mixture consistently improves performance. Each line shows performance on one model / evaluation combination as we progressively add datasets to the Activation Oracle training mixture. Ten of twelve combinations achieve their best performance with the full mixture, and most curves consistently improve, indicating that adding more diverse training tasks systematically helps on downstream tasks.

5 Additional Out-of-Distribution Generalization Experiments

Further evaluations examine learned information absent from the prompt and transfer to classification tasks outside the training mixture.

5.1 PersonaQA

PersonaQA uses newly trained models that have learned attributes of one hundred synthetic people. Only a person’s name appears in the inspected prompt; the oracle must recover a learned attribute. Open-model oracle accuracy rises from about 5% without training to 14% with the full mixture. Haiku reaches roughly 35%, with training variants within error bars.

Performance is modest and the accessible-information ceiling is unknown. The target models’ own knowledge is brittle: open-ended questions work far better than equivalent yes/no questions. Binary oracle tests are easier and sometimes outperform the target model’s responses, but do not establish reliable general access to hidden knowledge.

Refer to caption
Target prompt: My name is Maria Silva. Oracle prompt: <ACT><ACT><ACT><ACT><ACT> What is their favorite board game? Oracle response: Mancala. Target prompt: Oil prices soar to all-time highs. Oracle prompt: <ACT> Is the content of this news about business? Oracle response: Yes.
Figure 6: Activation Oracles improve over baselines on out-of-distribution tasks. Left (PersonaQA): We evaluate on models fine-tuned on biographical facts for 100 fictional personas. The input text contains only the persona’s name, so successful extraction requires reading information from the model’s activations. All trained AOs improve over the PatchScopes baseline. Right (Classification): Activation Oracles generalize to 7 held-out classification datasets. The stronger performance relative to the SPQA-only model is expected since classification tasks are not included in SPQA training.

5.2 Classification

On seven held-out classification tasks, the oracle receives one activation near the end of the input. Diverse training improves transfer. Weak untrained-model performance may partly reflect unfamiliar input formatting rather than absence of the information being queried.

6 Training Dataset Ablations

Ablations separate the effects of data diversity, training quantity, injection position, and the choices made when reading activations.

6.1 Scaling the Training Mixture

The full mixture is best on ten of twelve model–task combinations and within error bars on the other two. SPQA alone can perform well, but is more sensitive to learning rate. Additional classification and context-prediction data make performance more consistent across tasks; the preferred injection layer also depends on model and evaluation.

6.2 Separating Diversity from Data Quantity

A controlled 400,000-example comparison shows that mixing tasks helps beyond simply increasing the number of SPQA or classification examples. Scaling the full mixture to one million examples adds further gains. These experiments support both diversity and quantity, without identifying an optimal mixture.

Original figure
Figure 7: Both data diversity and data quantity improve Activation Oracle performance. We ablate training on Qwen3-8B across three configurations: a SPQA + Classification baseline with 400k examples, a truncated full mixture that includes SPQA, classification, and context prediction but is subsampled to 400k examples, and the full mixture with all three datasets and 1M examples. The truncated mixture outperforms the data-matched baseline on most evaluation metrics, showing that adding the self-supervised context prediction task improves generalization even when total sample count is fixed. The full 1M-example mixture performs best overall, indicating that scaling both the diversity and the quantity of activation-verbalization data provide the strongest gains.

7 Related Work

The method connects trained latent verbalization to logit and tuned lenses, sparse autoencoders, activation steering, PatchScopes, and model-diffing audits. These tools expose different kinds of evidence: an oracle offers flexible language explanations, while feature- and circuit-based methods can provide more explicit structures for causal investigation.

8 Discussion and Limitations

An oracle is itself a black-box predictor. Its answer may reflect task priors, incomplete information, or hallucination rather than a faithful computational explanation. Independent behavioral or causal tests remain necessary. The evaluated fine-tunes and secrets are controlled and relatively narrow, and the study does not establish robustness to strategic concealment or complex production post-training.

More training data and new objectives may improve transfer, but calibration, mechanistic faithfulness, and realistic auditing remain open. A small hallucination-detection anecdote did not beat straightforward follow-up prompting.

9 Conclusion

Diverse activation-to-language training produces a practical auditing component that generalizes to several unseen tasks. The main contribution is improved access to internal information with a simple interface, with clear limits on explanation reliability and the breadth of the evidence.

10 Author Contributions

Adam Karvonen led the project. Collaborators developed the early method, model-diffing and classification evaluations, and Claude experiments; Owain Evans and Samuel Marks advised the research.

11 Acknowledgements

The work acknowledges research advice, MATS and fellowship support, Coefficient Giving, and the Constellation research environment.

Appendix A Training Details

A.1 Hyperparameters

The hyperparameter table specifies LoRA training with AdamW, 10% linear warmup, and linear decay. Llama-3.3-70B uses batch size 64 to improve utilization on four H200 GPUs.

Hyperparameter Value
LoRA rank 64
LoRA alpha 128
LoRA dropout 0.05
LoRA target modules all linear layers
Learning rate 1e-5
Training batch size 16
Table 1: Training hyperparameters used for all Activation Oracle experiments.

A.2 Group by Length Batching

Examples are sorted by length within randomly ordered mega-batches, reducing padding without sequence packing. The authors report approximately 30% faster training with no noticeable change in final loss or evaluation performance.

A.3 Infrastructure

Qwen and Gemma train on one H100; Llama uses four H200s, distributed data parallelism, and 8-bit quantization. Activations are generated during training by temporarily disabling the LoRA adapter, avoiding storage of full activation sequences. Reported costs are 10, 12, and 90 GPU-hours respectively.

A.4 Learning Rate Sweep

A six-value learning-rate sweep on Gemma finds the full mixture relatively stable except at the largest rate. SPQA-only training varies more, particularly on the gender task. The main open-model experiments use a learning rate of 0.00001.

Original figure
Figure 8: Learning rate sweep. Evaluation performance across four benchmarks for Gemma-2-9B-IT trained with varying learning rates. Our full dataset model (orange line) maintains stable performance across learning rates, while the SPQA-only baseline (blue line) is somewhat unstable.

A.5 Steering Layer Selection

Injection after layer 1 reduces Llama training loss by about 10% compared with layer 0 and improves its evaluated transfer tasks. Qwen and Gemma show little training-loss difference, so this result is not universal. Replacement-based injection caused extreme activation-norm growth; additive norm-matched injection resolves that observed training problem.

Original figure
Figure 9: Layer 1 steering outperforms layer 0 across all Llama-3.3-70B evaluations. Layer 0 steering results in 1–11% lower performance on downstream tasks.

A.6 Differences in Claude Haiku 3.5 Training

Haiku replaces a normalized activation at layer 0 and updates all parameters, instead of using LoRA and additive injection. It uses one target layer, different prompts and token-selection rules, and past-context prediction only. Learning rate affects its yes/no and open-ended evaluations differently; the chosen rate uses separate validation tasks, and is not optimal for every reported metric.

Appendix B Training Dataset Details

B.1 General Dataset Construction

Training samples target layers at quarter, half, and three-quarter depth equally and mixes single- and multiple-token inputs. Dataset proportions are initial design choices rather than extensively tuned optima.

B.2 Context Prediction Dataset

The context dataset contains 600,000 examples split equally between FineWeb and LMSYS Chat-1M. Half use one activation and half a contiguous sequence. Past and future prediction are balanced; this appendix specifies sequence and target lengths sampled up to fifty tokens.

B.3 Classification Datasets

Seven tasks contribute 48,000 examples each: truth, relations, sentiment, gender, entailment, named entities, and tense. Two thirds use one activation sampled near the input’s end; the rest use short sequences. Randomizing the position reduces brittle dependence on a special token, and emphasizing single vectors limits solving the task by reconstructing all the input text.

B.4 SPQA Dataset

The 64,000 SPQA examples are divided between the original full selected-context format and a compressed window of one to three activations near the context’s end.

Appendix C Evaluation Details

C.1 Secret-Keeping Evaluation Details

Secret-keeping evaluations inspect activations from direct requests that the target refuses to answer honestly. The main comparison uses previously reported baselines. A GPT-4.1 formatting step extracts short instruction phrases from conversational oracle answers for the Secret Side Constraint rubric; it is part of that evaluation pipeline.

Original figure
Figure 10: Baseline results on secret-keeping tasks on the direct prompts. Figure reproduced from Cywiński et al. (2025).

Taboo

The target is directly asked for its concealed word, while the oracle is asked what secret word the text concerns. The best token-selection variant reads the assistant’s start-of-turn activation.

Secret Side Constraint (SSC)

The target is asked to reveal its hidden side constraint; the oracle is asked to reproduce the instruction. Full-sequence activations are used for the main comparison.

User Gender

For the gender task, the oracle chooses between the benchmark’s two labels. The retained listing also specifies the GPT-4.1 extraction prompt used for the separate side-constraint task, including its examples and treatment of vague answers.

C.2 Model-Diffing Evaluation Details

The model-diffing audit supplies differences from the first ten generated tokens, where prior work found strong fine-tuning signals. One fixed open-ended oracle prompt is used without task-specific iteration.

Emergent Misalignment

A neutral joke-writing request elicits the target activations. The oracle receives the generic question asking what is happening in the text.

Agentic Evaluation

An auditing agent receives ten oracle verbalizations, forms a hypothesis, and may query the original and modified models under the stated interaction budget. A separate model grades its final hypothesis from one to five. The rubric distinguishes no valid information, topic identification, indications of unsafe change, and a detailed account of potentially misaligned fine-tuning.

C.2.1 Emergent Misalignment Evaluation Details

The six fine-tunes combine harmful-topic examples and UltraChat equally. Only the risky-finance Qwen model shows appreciable broad emergent misalignment, reported as 18% ± 8%; the others show negligible rates. Therefore the average audit score principally measures discovery of narrow fine-tuning topics, and higher rubric scores are poorly matched to most targets.

Misaligned answer rate on 10 prompts from Betley et al. (2025)
Original figure

Figure 11: Only Qwen3-8B fine-tuned on risky financial advice exhibits significant emergent misalignment (18%).

C.3 OOD Classification Dataset Evaluation

Seven held-out binary tasks cover language, news category, grammatical number, headline subjects, and historical figures. Each has 250 examples evaluated with three sampled question variants. All use one activation immediately before the end-of-sequence token.

C.4 Evaluation Input Token Selection

Full sequences generally outperform single activations, except on Taboo. Shorter end-of-prompt windows can also help the gender task. These task-specific choices benefit from known-answer validation, which may be unavailable in real audits. The SPQA-only comparisons here include mixed-length training, unlike the original full-sequence LatentQA setup.

Original figure
Figure 12: Comparison of full sequence vs. single token input. Full sequence activations outperform single-token activations on most evaluations, with the exception of Taboo where single-token inputs perform better.

C.5 Evaluation Performance Across Input Layers

Classification transfers reasonably to intermediate layers not used during training, but performs much worse near the network’s beginning. Secret-keeping performance is weaker at 75% than 50% depth. Input-layer selection is therefore consequential even when one task appears robust.

Original figure
Figure 13: Classification performance is consistent across input layers. Performance remains strong across trained layers (25%, 50%, 75%) and generalizes to untrained layers (33%, 66%), but degrades substantially for early layers (0%, 10%).
Original figure
Figure 14: Layer 50% outperforms layer 75% on secret-keeping evaluations. Using activations from layer 75% instead of 50% results in decreased performance on SSC, Taboo, and Gender tasks, suggesting layer selection is more important for out-of-distribution evaluations.

C.6 PersonaQA Evaluation Details

C.6.1 Evaluation Setup

PersonaQA is tested both with open-ended attribute questions and balanced yes/no questions. These formats probe related information but have different output constraints and baselines.

String Matching.

Open-ended scoring searches the answer case-insensitively for the true attribute, allowing a small predefined set of spelling variants and synonyms. This is an answer-matching criterion, not a judgment of the explanation’s faithfulness.

C.6.2 Training Hyperparameters

The retained table specifies the persona-model training hyperparameters used for this evaluation.

Hyperparameter Value
LoRA rank 32
LoRA alpha 64
LoRA dropout 0.05
LoRA target modules all linear layers
Learning rate 5e-5
Training batch size 8
Weight Decay 0.01
Epochs 3
Table 2: Training hyperparameters used for PersonaQA models.

C.6.3 PersonaQA Model Knowledge is Brittle

Target-model accuracy exceeds 80% on familiar open-ended attribute questions but falls to about 55% on equivalent yes/no questions. This format sensitivity limits conclusions about how robustly the learned attributes are represented or accessible.

Original figure
Figure 15: PersonaQA model knowledge evaluation (open-ended format). The fine-tuned models achieve high accuracy when queried in the training format (open-ended questions), exceeding 80% across all models.
Original figure
Figure 16: PersonaQA model knowledge evaluation (binary yes/no format). When the same knowledge is tested via binary yes/no questions, accuracy drops to approximately 55%, near random chance. This brittleness suggests the persona information may not be robustly represented in the model’s activations.

C.6.4 Activation Oracle Results

More diverse training improves open-ended persona recovery, but absolute performance remains low and the best attainable accuracy from these activations is unknown.

Original figure
Figure 17: Activation Oracle performance on PersonaQA (open-ended evaluation). Performance improves with additional training data.

C.6.5 Comparison of Evaluation Formats

Binary questions produce higher scores with a smaller answer space. Some Llama and Qwen oracles outperform their targets’ own binary answers across multiple hyperparameter settings, suggesting information can be accessible through one readout but poorly used by another.

Original figure
Figure 18: Activation Oracle performance on PersonaQA (binary yes/no evaluation).

C.7 Detailed Dataset Ablation Results

The additional plots retain per-mixture results, including Qwen settings where SPQA alone is weak and improves substantially with classification and context-prediction data.

Original figure
Figure 19: Qwen3-8B results on PersonaQA and Taboo across training mixtures. Accuracy of Activation Oracles trained with different datasets on two Qwen3-8B evaluations: recovering fine-tuned persona attributes (PersonaQA) and recovering a hidden secret word (Taboo-style secret-keeping). The untrained base model and SPQA-only AO perform poorly on both tasks, while adding classification and especially self-supervised context-prediction data yields large gains.
Original figure
Figure 20: Secret-keeping performance for different Activation Oracle training mixtures. Accuracy on the three secret-keeping benchmarks: Taboo and Gender (Gemma-2-9B-IT) and Secret Side Constraint (Llama-3.3-70B), for the original model, single-task AOs, and the full Activation Oracle mixture. Across tasks, the full mixture achieves the best performance, but SPQA-only is already competitive: it clearly improves over the original model, and on Gemma secret-keeping tasks it matches or nearly matches the best white-box baselines.

Appendix D Classification Datasets Construction

Binary classification questions reuse true/false labels where available; otherwise half of examples are paired with an incorrect label. Claude generates varied question templates, and answer labels are approximately balanced. The retained table gives representative tasks and wording.

Table 3: Classification dataset examples
Task Source Context Question Answer
True or false Geometry of truth + Relations The city of Paris is in Italy. Is this statement true? No
Sentiment
analysis
SST2 contains no wit, only labored gags Is this a negative review? Yes
Gender
detection
Multi-Dimensional Gender Bias Classification Max Landis is a comic book writer who wrote Chronicle, American Ultra, and Victor Frankestein. This text is about Max Landis. Am I reading about a male person here? Yes
Topic
classification
AG News Classification Ky. Company Wins Grant to Study Peptides - A company founded by a chemistry researcher at the University of Louisville won a grant to… Is this article about Sports? No
Entailment/
contradiction
SNLI A man inspects the uniform of a figure in some East Asian country. Is the next sentence logically consistent with what came before? The man is sleeping. No
Sentence
tense
English tense dataset + POS tag dataset for tense form detection They played soccer yesterday Is this statement in the present tense? No
Named
entity
recognition
Name Entity Recognition (NER) dataset Thousands of protesters have marched through London Does this text mention London? Yes
Language
identification
WiLI-2018 de spons behoort tot het geslacht haliclona en behoort tot de familie… Is this text written in English? No
Plural/
Singular
subject
Generated using Claude 3.5 Sonnet Sarah and Mike are dancing. Is this sentence referring to one individual? No
Trump
headline
Kantamneni et al. House G.O.P. Signals Break With Trump Over Tariff Threat. Is this headline about Donald Trump? Yes
Obama
headline
Kantamneni et al. Obama Sticks to a Deadline in Iraq. Is this headline about Barack Obama? Yes
China
headline
Kantamneni et al. China Asks U.S. to End Close-Up Military Surveillance. Is this headline about China? Yes
Historical
figure
Is Male
Kantamneni et al. Margaret of Clisson Is this person a male? No

Appendix E PersonaQA Dataset Construction

E.1 Overview and Attribution

This is a new implementation of PersonaQA-Shuffled, developed with guidance from the original authors because their dataset is not public. Independently shuffling attributes removes demographic correlations between names and answers while retaining realistic attribute vocabulary.

E.2 Dataset Construction

Persona Generation.

Claude generates one hundred plausible named personas with country, food, drink, music, sport, and board-game attributes. Each attribute column is then independently shuffled while names stay fixed, preventing each person from retaining the original value in that column. The complete generation prompt is retained.

Training Text Generation.

Each shuffled person receives 250 biographies and 250 interviews, cycling through varied style instructions. The resulting dataset contains 50,000 texts; the original generation templates remain available below.

Training Process.

The name is the user prompt and a biography or interview is the assistant response. The described Qwen training uses three epochs, learning rate 0.00005, and a rank-32 LoRA adapter with alpha 64.

PersonaQA: Persona Generation System Message ⬇ You are a JSON - only assistant . The response MUST be valid JSON : either a single array or a single object . No code fences , no extra commentary , no trailing commas . Use ASCII quotes for keys and values . You will produce persona objects with culturally plausible correlations . Each persona MUST use these exact keys ( snake_case ): name , country , favorite_food , favorite_drink , favorite_music_genre , favorite_sport , favorite_boardgame Constraints : - name MUST be a full name with FIRST and LAST name separated by a single space ( e . g ., " Maria Santos "). No middle names , no initials only . - Ordinary , globally diverse personas ( no celebrities ). - Values are short (1-3 words each ), no internal quotes . - Use real countries and plausible combinations . User Message ⬇ 3 You are generating a batch of new personas . ALREADY_USED_NAMES ( do not repeat any of these names ): [ list of previously generated names ] Generate exactly { m } NEW personas following the schema and constraints . Rules : - No duplicate names within this batch . - No names that appear in ALREADY_USED_NAMES . - Return ONLY a JSON array of { m } persona objects ( no wrapper object , no commentary ). Generation parameters: Model: claude-sonnet-4-20250514 , Temperature: 0.3, Max tokens: 8192, Batch size: 10 personas
Figure 21: Persona generation. We generate 100 base personas with culturally plausible correlations using batched JSON requests, then shuffle attributes to create PersonaQA-Shuffled.
PersonaQA: Training Text Generation (Biographies & Interviews) System message ⬇ Write a short narrative based on provided attributes . Return ONLY the narrative text . No JSON , no code fences , no headers , no labels . Include the person ' s name and ALL attributes verbatim at least once . Length target : ~120-220 words . No bullet points . User message ⬇ STYLE INSTRUCTION : { style_instruction } ATTRIBUTES ( use verbatim , reordering is fine ): { attributes_block } Return ONLY the narrative paragraph ( s ). Generation parameters: Model: claude-sonnet-4-20250514 , Temperature: 1.0, Max tokens: 2048 Example biography style instructions (15 total): • “Write a narrative that is intended for lifestyle blog subscribers, given the following attributes.” • “Write a concise profile suitable for a personal website ‘About’ section.” Example interview style instructions (20 total): • “Please write a paragraph describing how a person would introduce themselves based on the following list of attributes. Make sure to include their name and all of the attributes.” • “Write a Q&A interview with 4 questions, each answer naturally including one or two attributes.”
Figure 22: Training text generation. For each of 100 shuffled personas, we generate 250 biographies and 250 interviews by cycling through style instruction templates.

Appendix F Anecdotal Hallucination Detection Experiment

A single movie-description example contains three manually identified errors among fourteen details. Oracle confidence judgments flag two of those errors, but follow-up questions to the original model perform similarly. This anecdote is not evidence of a general hallucination-detection advantage.

Appendix G Sparse Autoencoder Dataset Experiments

Preliminary experiments add labels and activating-context predictions for sparse-autoencoder features. The setup uses three 65,000-feature dictionaries and filters generated explanations for high detection scores. These datasets improve some classification results and feature verbalization, but give little or inconsistent benefit on the main auditing tasks. Their explanation-generation cost and limited transfer led the authors to exclude them from the final mixture.

Concept demonstration

Open image ↗