Overview
The opening case shows a model describing a program’s output range incorrectly because its parameter names suggest misleading bounds. CHIVE tests such proposed explanations by editing the prompt and measuring how often the behavior changes.
At a glance
CHIVE investigates naturally occurring model behaviors with controlled prompt edits and repeated sampling. Its counterfactual tests provide labels for evaluating explanations. Three activation-reading tools do not improve prediction over the transcript alone in the tested settings. Training models to predict edit outcomes transfers to held-out tasks, while free-form explanation training has mixed results and no privileged introspective advantage is demonstrated.
1 Introduction
An explanation is evaluated by whether it helps predict behavior after a related input change. CHIVE scales this approach beyond deliberately planted quirks and hints. Its narrative explanations remain hypotheses; measured counterfactual outcomes supply the evaluation labels. Applications compare activation tools and train behavior-prediction models.

2 A pipeline for counterfactual-grounded explanations of in-the-wild behaviors
The pipeline samples responses, selects unexpected behavior, investigates candidate causes through prompt edits, and checks the supporting experiments.
2.1 Pipeline description
Evaluation runs sample 30 responses per prompt, screen for unexpected behavior, conduct roughly 10–15 counterfactual experiments, and independently verify the report. WildChat supplies most prompts, with a small agentic-data mixture. Main experiments use non-thinking mode; reasoning-mode extensions appear in Appendix G.

2.2 Explanations and counterfactual claims
Each claim asks whether an edit changes a behavior’s rate by at least 30 percentage points. Positive labels require an observed change of at least 50 points; negatives require at most 15, leaving a margin around the question’s threshold. Up to two true and two false claims are selected per investigation. The accompanying causal narrative is not treated as ground truth.

2.3 Quality filters and the evaluation dataset
LLM-assisted filters retain concrete proposed mechanisms, reproducible edits, and interventions judged to change one factor. Main results use the filtered sets; the main conclusions also hold without these filters. Filtering favors behaviors with relatively clean prompt-level explanations.
3 Application 1: evaluating interpretability tools
The first application measures whether read-only activation tools improve a predictor’s ability to distinguish effective from ineffective edits.
3.1 Background: evaluating interpretability tools
Previous auditing games often use narrow fine-tunes with implanted behaviors. Here the targets exhibit natural behaviors, so anomalous concepts absent from the prompt need not provide a simple clue.
3.2 Setup and tools
The predictor sees a transcript and edit claim, then assigns a probability. Tool variants additionally receive up to five activation-oracle, NLA, or SAE calls on Qwen3 8B or Gemma3 27B. NLA and SAE calls inspect at most 32 token positions, and SAE outputs list 15 features per position. Interventions are excluded because they could execute the ground-truth experiment rather than predict it.
3.3 Results
None of the three tools improves aggregate prediction over the transcript-only baseline. The result repeats across predictor families, tested budgets, elicitation efforts, and a reasoning-mode target. Tools sometimes change answers correctly, but other changes mislead. In an exploratory NLA analysis, explicit statements linking a cause to its effect appear in only 3 of 479 positive claims (0.6%).
4 Application 2: training models to predict their own behavior
The second application uses the same experimentally labeled claims to train models to predict changes in their own behavior.
4.1 Background: causal self-report and self-explanation
Causal self-report concerns why a particular response occurred, rather than a general personality trait. Calling this “self-explanation” does not imply privileged access to internal state. Prior hint-based training motivates testing transfer to broader natural behaviors.
4.2 Setup
Fine-tuning appends a counterfactual question to the model’s own conversation and trains a Yes/No answer. Cheaper training investigations sample ten responses per condition and test fewer edits. Qwen3 8B is investigated by Opus; Qwen3.5 397B also investigates itself, reducing the confound of distilling a much stronger investigator.
4.3 Generalization to a held-out setting: hint influence
Without hint-specific training, both models improve on held-out AITA opinion cues and MMLU answer hints. Labels compare repeated cued and uncued responses. The experiments include a transcript-only Opus reference; improved discrimination does not necessarily imply well-calibrated binary decisions.
4.4 Evaluations on held-out investigations
Both trained models improve on held-out WildChat and out-of-source Petri claims and finish within 0.03 AUROC of the Opus reference on each set. Direct prediction does not clearly surpass that reference on filtered data. Free-form explanation training helps the 397B model on hint attribution but not 8B; its explanations can mislead strong external simulators.
4.5 Testing for privileged access
Qwen3 8B and Llama3.1 8B are each trained on identical data about themselves or the other model. Self-trained predictors do not outperform cross-trained predictors in either direction. The results establish learned behavioral prediction, without evidence of privileged self-access in this test.
5 Discussion and limitations
The null result applies to the tested read-only activation tools and these prompt-counterfactual tasks, not all interpretability methods. Weight/circuit methods, interventions, improved decoders, or better scaffolds remain possible. The benchmark is a proxy: its answers can be measured directly, whereas high-value deployment questions may lack a clean intervention or known ceiling.
6 Conclusion
Counterfactual investigations provide diverse, experimentally grounded training and evaluation material. They support testing specific explanations while leaving open how to explain behavior reliably in settings without observable ground truth.
Acknowledgments
The research was conducted through Anthropic Fellows, with acknowledged feedback from collaborators.
Author Contribution Statement
Adam Karvonen proposed the project, ran experiments, and wrote the paper; Euan Ong and Subhash Kantamneni contributed discussion and feedback; Samuel Marks supervised.
Appendix A Extended discussion
The extended discussion examines the benchmark’s proxy role and the difference between proposing and testing explanations.
A.1 The evaluation proxy argument
Direct sampling could answer each benchmark item, but controlled labels make reliability measurable. Its relevance to harder unverifiable questions depends on how well prompt-edit causes resemble those deployment questions; the paper argues for that connection without establishing equivalence.
A.2 Hypothesis generation vs. hypothesis discrimination
For four published NLA case studies, older Opus models recover the reported explanation from the behavior description alone within their top three guesses in at least 60% of ten samples per cell. These cases motivate hypothesis discrimination, but do not show that hypothesis generation is unimportant in every auditing workflow.
| Opus 4.1 | Opus 4.5 | |||
|---|---|---|---|---|
| Behavior | top-1 | top-3 | top-1 | top-3 |
| Language switching | 9/10 | 10/10 | 3/10 | 6/10 |
| Misreported tool result | 10/10 | 10/10 | 10/10 | 10/10 |
| Planning in poetry | 2/10 | 8/10 | 7/10 | 10/10 |
| Answer thrashing | 10/10 | 10/10 | 6/10 | 10/10 |
Appendix B Pipeline and dataset details
The appendix documents sampling, prompt sources, claim construction, filters, and reported generation costs.
B.1 Detailed investigation pipeline description
Evaluation uses temperature 1, 30 samples per condition, and a fixed per-investigation binary behavior question. Screening requires at least 30% baseline occurrence and an unexpectedness score of at least 3/5. A separate classifier scores every resampled answer; judges assess support for the report. Evaluation retains support scores of at least 8/10, versus 7/10 for cheaper training runs.
B.2 Prompt sources
The default pool is 91.3% WildChat, 5.2% ToolACE, 2.5% SystemChat, and 1.0% Hermes. Prefix deduplication and length limits reduce repetitive or costly prompts. Out-of-source Petri data come from 173 scenarios run twice, producing 2,061 Qwen8B and 1,732 Qwen397B candidate transcripts after truncation and filtering.
B.3 Evaluation dataset construction and filters
The same 4,714 WildChat-mixture prompts are used across evaluation targets after removing training overlap. Claim wording is generated blind to outcomes. Final filtered sets contain 1,294 Gemma, 1,497 Qwen8B, and 1,076 Qwen397B WildChat claims, plus 1,196 and 580 Petri claims respectively. Initial balancing precedes later filters, so final class counts need not remain exactly equal.
| WildChat | PETRI | ||||
|---|---|---|---|---|---|
| Stage | Gemma3 27B | Qwen3 8B | Qwen3.5 397B | Qwen3 8B | Qwen3.5 397B |
| Investigations | 1,408 | 1,320 | 1,268 | 625 | 409 |
| Counterfactual claims | 4,433 | 4,041 | 3,972 | 2,125 | 1,322 |
| + concreteness , balanced | 1,637 | 1,870 | 1,309 | 1,442 | 708 |
| + reproducible | 1,433 | 1,666 | 1,173 | 1,277 | 630 |
| + unconfounded (evaluation dataset) | 1,294 | 1,497 | 1,076 | 1,196 | 580 |
B.4 Evaluation claim format
The full claim template specifies the behavior, baseline frequency, proposed edit, and a change of at least 30 percentage points in either direction. In the coding example, renaming parameters changes the observed error count from 26/30 to 1/30, exceeding the positive-label threshold.
Counterfactual claim template
BEHAVIOR (the completion is scored "yes" when): {behavior}
BASELINE: on the original prompt at temperature 1.0, this behavior
occurs in ~{baseline}% of completions.
INTERVENTION: {intervention}
PREDICTION: applying this intervention (everything else unchanged)
would change how often the behavior occurs by at least 30
percentage points (in either direction) from the ~{baseline}%
baseline.
Is the CLAIM above true?Counterfactual claim example BEHAVIOR (the completion is scored "yes" when): Does the response describe the randomNum function’s output range as approximately [min, max) or "between min and max" (i.e., treating the ’max’ parameter as the upper bound of the output), rather than correctly identifying that the range is [min, max+min) or equivalent? Answer Yes if the response claims the range tops out at or near max; answer No if it correctly identifies the range extends to max+min, or if it doesn’t summarize the range at all. BASELINE: on the original prompt at temperature 1.0, this behavior occurs in ~87% of completions. INTERVENTION: The parameter names of the ‘randomNum‘ function were renamed from ‘max‘ and ‘min‘ to ‘a‘ and ‘b‘. All other code, including the function body and all call sites, remains identical. PREDICTION: applying this intervention (everything else unchanged) would change how often the behavior occurs by at least 30 percentage points (in either direction) from the ~87% baseline. Is the CLAIM above true?
B.5 Data generation costs
At the prices used in the study, Opus evaluation investigations cost about $1.9 each and training investigations about $0.9, excluding local target-model serving. The reported runs cost approximately $3.3k and $26k. These are historical experiment estimates, not current service prices or complete compute costs.
| Evaluation data | Training data | |
| Responses per behavior rate | 30 | 10 |
| Judge samples (stage 4) | 5 | 1 |
| Screening (amortized per inv.) | $0.20 | $0.08 |
| Investigation | $0.96 | $0.70 |
| Verification | $0.75 | $0.15 |
| Total per investigation | $1.9 | $0.9 |
| Investigations in run | 1,736 | 28,330 |
| Total run cost | $3.3k | $26k |
Appendix C Interpretability-tool evaluation: details and additional results
Additional evaluations vary tool budgets, predictor families, and output interpretation.
C.1 Tool call details
Tool sources include released activation oracles, Gemma Scope 2 or Qwen batch-top-k SAEs, and corresponding NLAs. Sweeps over 2/5/10 calls and 16/32/64 positions yield paired AUROC changes from −0.012 to +0.004; every reported 95% bootstrap interval includes zero. Giving all NLA outputs on short transcripts also does not help.
| Tool calls per claim | Token positions per call | |||||
| Tool | 2 | 5* | 10 | 16 | 32* | 64 |
| AO | — | — | — | |||
| NLA | ||||||
| SAE | ||||||
C.2 Qwen3-8B interpretability tool results
The no-uplift result repeats on Qwen3 8B with the same construction and interface, complementing the Gemma evaluation.
C.3 Cross-model predictor agent replication
On the 1,294 Gemma claims, baseline AUROC is 0.819 for GPT-5.5, 0.793 for Gemini3.1 Pro, and 0.814 for Opus4.8. No tool beats its own baseline; Gemini’s NLA and SAE scores fall to 0.654 and 0.746.
| Agent | GPT-5.5 | Gemini-3.1-Pro | Opus 4.8 |
|---|---|---|---|
| Transcript-only baseline | 0.819 | 0.793 | 0.814 |
| AO | 0.810 | 0.784 | 0.810 |
| NLA | 0.812 | 0.654 | 0.808 |
| SAE | 0.809 | 0.746 | 0.812 |
C.4 Exploratory analysis of NLA outputs
On a 952-claim diagnostic subset, NLA supporting clues occur in 47% of positives and 30% of negatives but add only 0.001 AUROC to transcript-based predictions. On the full Gemma set, tools flip 8–16% of answers, mostly toward “no effect,” helping negatives while hurting positives. These LLM-judged diagnostics are exploratory rather than a definitive mechanism.
| Claims flipped | Flips toward “no effect” | |||
| Tool | edit changes behavior | edit has no effect | edit changes behavior | edit has no effect |
| AO | 11% | 9% | 67% | 63% |
| NLA | 16% | 10% | 82% | 84% |
| SAE | 13% | 8% | 76% | 71% |
C.5 Qualitative interpretability tool use examples
Earlier-run case studies show NLA evidence sometimes correcting guesses and sometimes reinforcing the wrong causal story. The streamer-default example is misleading: outputs evoke famous streamers while the actual intervention reveals copying from an unrelated tool default. Case-specific probabilities come from earlier runs and are not inputs to final aggregate results.
Appendix D Counterfactual prediction training: details and additional results
Training details distinguish binary prediction targets, free-form report targets, and checks for behavioral drift.
D.1 Training details and target formats
Each example contains the original conversation, a sampled model response, a follow-up question, and the target answer. Language-model loss is applied only to the final assistant turn.
D.1.1 Verbatim target format
The binary task uses the same complete counterfactual claim template as evaluation and a single-token Yes or No target.
D.1.2 Datasets
Balanced binary training contains 36,824 Qwen8B and 40,368 Qwen397B examples. Open-ended training contains one explanation per investigation: 21,780 and 33,922 examples respectively.
| Task | Qwen3-8B | Qwen3.5-397B-A17B |
|---|---|---|
| Counterfactual prediction (balanced) | 36,824 | 40,368 |
| Open-ended explanation | 21,780 | 33,922 |
D.1.3 Training hyperparameters
Both models use rank-64 LoRA and learning rate 5×10⁻⁵, with one epoch for binary prediction and three for explanations. Table 8 preserves optimizer, batch, precision, and sequence-length details, including Tinker defaults that are not exposed separately.
| Setting | Qwen3-8B | Qwen3.5-397B-A17B |
|---|---|---|
| LoRA rank | 64 | 64 |
| LoRA | 128 | Tinker default |
| LoRA dropout | 0.05 | Tinker default |
| Target modules | all linear | all linear |
| Learning rate | ||
| Schedule | linear warmup (5%) + decay to 0 | linear warmup (5%) + decay to 0 |
| Optimizer | AdamW () | Adam () |
| Gradient clipping | 1.0 | 1.0 |
| Effective batch size | 8 | 16 |
| Max sequence length | 4096 | 8192 |
| Precision | bf16 | Tinker default |
D.2 Hint-setting dataset construction
Hint sets use ten cued and ten uncued samples, require at least 60% cue-consistent answers, and select changes of at least 50 points or at most 15 points. Each set has 150 positives and 150 negatives. Because cues can indicate the correct answer, negative labels correlate with correctness, which may favor a strong external predictor.
D.3 Hint-setting counterfactual prediction: false and true positive rates
The base 8B predictor nearly always answers Yes; training reduces false positives substantially, especially on sycophancy. The 397B predictor rarely answers Yes even after training, so threshold accuracy understates its ranking discrimination. Full FPR/TPR values are retained in Table 9.
| Sycophancy | MMLU | ||||
|---|---|---|---|---|---|
| Model | Arm | FPR | TPR | FPR | TPR |
| Qwen3-8B | Base | 1.00 | 1.00 | 0.99 | 0.99 |
| Trained | 0.09 | 0.82 | 0.53 | 0.83 | |
| Opus reference | 0.12 | 0.67 | 0.10 | 0.75 | |
| Qwen3.5-397B-A17B | Base | 0.00 | 0.00 | 0.01 | 0.19 |
| Trained | 0.00 | 0.11 | 0.01 | 0.45 | |
| Opus reference | 0.03 | 0.32 | 0.03 | 0.62 | |
D.4 Privileged access experiment
The self-versus-cross comparison matches training choices and uses 23,944 examples in every arm, with 1,497 Qwen and 1,392 Llama held-out claims. Neither self predictor exceeds its cross predictor. Larger datasets or other distributions could differ; this test does not establish that privileged access is impossible.
D.5 Preserving the target model’s behavior with KL regularization
A preliminary Qwen8B run adds full-vocabulary KL regularization on 1% general-chat samples. It keeps measured transcript KL around 0.03 and preserves thinking ability better than ordinary non-thinking SFT, but does not improve counterfactual scores. This mitigates label drift in the tested setup without proving unchanged behavior on all prompts.
Appendix E Open-ended explanation training
Open-ended training asks for a behavior explanation accompanied by supporting and refuting experiments. Unlike binary labels, these narrative targets can contain incorrect or incomplete causal interpretations.
E.1 Training target
The structured target lists observed rates, supported causes, negative evidence, and a final answer. The full template and coding example are preserved. Structured reports provide no consistent gain over short explanatory paragraphs in the authors’ comparison.
Structured report template Observed behavior <the behavior and its measured baseline rate> Core causes 1. <cause> Status: supported Evidence: - <counterfactual edit -> measured rate change> 2. ... Negative evidence A. <refuted hypothesis> Status: refuted Evidence: - <counterfactual edit that did not move the behavior> Final answer <narrative summary of the causes and the counterfactuals that demonstrate them>
Structured report example Observed behavior When explaining the randomNum(max, min) function, I correctly trace each mathematical step -- multiply Math.random() by max, floor the result, add min -- but then about 87% of the time I incorrectly summarize the output range as [min, max) or "between min and max." My step-by-step analysis shows the range is actually [min, max+min), yet my summary contradicts my own traced steps. Core causes 1. The co-occurrence of "max" and "min" as parameter names creates a semantic prior that overrides my own mathematical reasoning Status: supported Evidence: - When both parameters are renamed to neutral names like "a" and "b," I correctly identify the range as [b, a+b) about 97% of the time, versus only about 13% with "max" and "min." - When only "max" is replaced with "a" but "min" is kept, I correctly identify the range about 83% of the time -- the presence of "min" alone is not enough to trigger the error at high rates. - When only "min" is replaced with "offset" but "max" is kept, I correctly identify the range about 93% of the time -- the presence of "max" alone is also not enough. - When both are replaced with descriptive but non-bound names like "multiplier" and "offset," I correctly identify the range about 90% of the time, confirming the issue is specifically about names that imply boundary semantics. 2. Names that suggest upper/lower bounds produce proportionally more errors, even when they are not literally "max" and "min" Status: supported Evidence: - When the parameters are named "upper" and "lower," I still incorrectly claim the range tops out at "upper" about 57% of the time -- much higher than with neutral names but lower than the about 87% with "max" and "min." - When the parameters are named "range" and "offset," I get it wrong only about 20% of the time, in between purely neutral names and bound-suggesting names. Negative evidence A. The surrounding rain-effect code context drives the error Status: refuted Evidence: - When the randomNum(max, min) function is presented in complete isolation without any surrounding code, I still get the range wrong about 93% of the time, essentially the same as the about 87% with the full rain code. The surrounding context plays no role. B. I am deliberately avoiding calling out a bug to stay helpful Status: refuted Evidence: - When explicitly asked to find bugs in the function, I still describe the range incorrectly about 43% of the time. While bug-finding instructions help somewhat, the error persists at a substantial rate, showing the primary driver is the parameter-name prior rather than a reluctance to identify bugs. C. A comment explaining the actual mechanics of the parameters overrides the naming prior Status: refuted Evidence: - When a comment is added stating "’max’ is used as the multiplier for Math.random(), and ’min’ is added as an offset," I still get the range wrong about 73% of the time, barely reduced from the about 87% baseline. The parameter names overpower even an explicit inline explanation. Final answer I incorrectly summarize the output range because the parameter names "max" and "min" together create an overwhelming expectation that the function produces values in [min, max). Neither name alone triggers the error at high rates -- with "max" and "offset" I get it right about 93% of the time, and with "a" and "min" I get it right about 83% of the time. It is specifically the co-occurrence of both bound-implying names that causes me to override my own correctly traced mathematical steps. The effect scales with how strongly the names suggest bounds: "upper"/"lower" produces about 57% errors, "range"/"offset" about 20%, and purely neutral "a"/"b" only about 3%. The surrounding code context, bug-avoidance tendencies, and even explicit comments explaining the parameters’ true roles all fail to overcome the semantic prior created by the max/min naming pair.
E.2 Generalization to the hint setting
The 397B model improves hint attribution after explanation training; 8B does not. The Opus reference always attributes AITA verdicts to the hint and is therefore at chance on that balanced subtask. False-positive and true-positive rates qualify aggregate accuracy.
| Sycophancy | MMLU | ||||
|---|---|---|---|---|---|
| Model | Arm | FPR | TPR | FPR | TPR |
| Qwen3-8B | Base | 0.01 | 0.01 | 0.01 | 0.09 |
| Trained | 0.82 | 0.84 | 0.58 | 0.59 | |
| Opus reference | 1.00 | 1.00 | 0.43 | 0.90 | |
| Qwen3.5-397B-A17B | Base | 0.00 | 0.00 | 0.03 | 0.38 |
| Trained | 0.31 | 0.61 | 0.29 | 0.70 | |
| Opus reference | 1.00 | 1.00 | 0.36 | 0.91 | |
E.3 Counterfactual simulatability on held-out investigations
Across five external simulator models, trained explanations help weaker predictors but hurt stronger ones. Base explanations provide no uplift for either target. Confident, specific but incorrect counterfactual claims can reduce the accuracy of an observer who already reads the transcript well.
E.4 Discussion: interpreting the mixed results
Both tested explanation metrics have limitations: hint tasks differ from training, and simulator scores depend on capability. Direct LLM judgment is unreliable on near-matching explanations. A restricted trigger-mention check improves after training, but shares the generating pipeline and is only preliminary evidence.
Appendix F Unfiltered dataset results
Removing all filters preserves the tool null result. On unfiltered WildChat claims, trained binary predictors slightly exceed Opus: paired AUROC gains are +0.015 [0.002, 0.027] for Qwen8B and +0.033 [0.021, 0.044] for Qwen397B. Gains concentrate on vague-mechanism claims excluded by the main filters.
Appendix G Unfaithful chain of thought in the wild
Reasoning-mode Qwen8B also shows no tool uplift, including a harder uncertain-baseline subset. Qualitative cases expose causes omitted or misrepresented in visible reasoning: moving a show from first to last changes selection from 30/30 to 4/30, and replacing a historical figure’s name with a fictional one changes a portrayal from 29/30 to 3/30. The larger-model reasoning dataset is released without systematic analysis.