Literature
Display
← All papers

Would this change your answer? Evaluating Explanations of LLM Behavior In The Wild with Counterfactual Experiments

Karvonen, Adam · Ong, Euan · Kantamneni, Subhash · Marks, Samuel

· arXiv (preprint) ↗

Interpretability10 min text · CondensedOriginal paper ↗
Contents

Overview

The opening case shows a model describing a program’s output range incorrectly because its parameter names suggest misleading bounds. CHIVE tests such proposed explanations by editing the prompt and measuring how often the behavior changes.

At a glance

CHIVE investigates naturally occurring model behaviors with controlled prompt edits and repeated sampling. Its counterfactual tests provide labels for evaluating explanations. Three activation-reading tools do not improve prediction over the transcript alone in the tested settings. Training models to predict edit outcomes transfers to held-out tasks, while free-form explanation training has mixed results and no privileged introspective advantage is demonstrated.

1 Introduction

An explanation is evaluated by whether it helps predict behavior after a related input change. CHIVE scales this approach beyond deliberately planted quirks and hints. Its narrative explanations remain hypotheses; measured counterfactual outcomes supply the evaluation labels. Applications compare activation tools and train behavior-prediction models.

Figure 1: Counterfactual investigation of one in-the-wild behavior , as produced by the CHIVE pipeline. Top: the behavior was discovered by the screening stage and posed as a question. Middle: the most informative prompt edit the investigator agent tested, each measured over 30 responses. Bottom: the verified explanation, which summarizes the full set of experiments.
Figure 1: Counterfactual investigation of one in-the-wild behavior, as produced by the CHIVE pipeline. Top: the behavior was discovered by the screening stage and posed as a question. Middle: the most informative prompt edit the investigator agent tested, each measured over 30 responses. Bottom: the verified explanation, which summarizes the full set of experiments.

2 A pipeline for counterfactual-grounded explanations of in-the-wild behaviors

The pipeline samples responses, selects unexpected behavior, investigates candidate causes through prompt edits, and checks the supporting experiments.

2.1 Pipeline description

Evaluation runs sample 30 responses per prompt, screen for unexpected behavior, conduct roughly 10–15 counterfactual experiments, and independently verify the report. WildChat supplies most prompts, with a small agentic-data mixture. Main experiments use non-thinking mode; reasoning-mode extensions appear in Appendix G.

Figure 2: Four hand-selected diverse behaviors discovered and explained by the pipeline, from Gemma-3-27B and Qwen3.5-397B on WildChat and PETRI prompts. The full investigations of these four are viewable here , and 20 randomly selected investigations here .
Figure 2: Four hand-selected diverse behaviors discovered and explained by the pipeline, from Gemma-3-27B and Qwen3.5-397B on WildChat and PETRI prompts. The full investigations of these four are viewable here, and 20 randomly selected investigations here.

2.2 Explanations and counterfactual claims

Each claim asks whether an edit changes a behavior’s rate by at least 30 percentage points. Positive labels require an observed change of at least 50 points; negatives require at most 15, leaving a margin around the question’s threshold. Up to two true and two false claims are selected per investigation. The accompanying causal narrative is not treated as ground truth.

Figure 3: Each investigation yields two data types , shown here for the investigation of Figure 1 , formatted as follow-up turns on the model’s own transcript (top). Left: a counterfactual claim asserts that a specific prompt edit would change the behavior; its Yes/No label was verified by running the edit. Right: an open-ended explanation of the behavior’s causes, citing the experiments that support it. Formats are simplified for presentation (verbatim versions in Appendices B.4 and E.1 ).
Figure 3: Each investigation yields two data types, shown here for the investigation of Figure 1, formatted as follow-up turns on the model’s own transcript (top). Left: a counterfactual claim asserts that a specific prompt edit would change the behavior; its Yes/No label was verified by running the edit. Right: an open-ended explanation of the behavior’s causes, citing the experiments that support it. Formats are simplified for presentation (verbatim versions in Appendices B.4 and E.1).

2.3 Quality filters and the evaluation dataset

LLM-assisted filters retain concrete proposed mechanisms, reproducible edits, and interventions judged to change one factor. Main results use the filtered sets; the main conclusions also hold without these filters. Filtering favors behaviors with relatively clean prompt-level explanations.

3 Application 1: evaluating interpretability tools

The first application measures whether read-only activation tools improve a predictor’s ability to distinguish effective from ineffective edits.

Refer to caption
Figure 4: Activation-based interpretability tools provide no uplift over a transcript-only baseline. Each predictor agent receives a transcript and a counterfactual claim (Section 2.2) and outputs the probability that the claim is true; tool predictors can additionally make up to 5 tool calls on the target model. For the Gemma-3-27B-IT target model, no tool predictor improves on the baseline (Qwen3-8B results in Appendix C.2).

3.1 Background: evaluating interpretability tools

Previous auditing games often use narrow fine-tunes with implanted behaviors. Here the targets exhibit natural behaviors, so anomalous concepts absent from the prompt need not provide a simple clue.

3.2 Setup and tools

The predictor sees a transcript and edit claim, then assigns a probability. Tool variants additionally receive up to five activation-oracle, NLA, or SAE calls on Qwen3 8B or Gemma3 27B. NLA and SAE calls inspect at most 32 token positions, and SAE outputs list 15 features per position. Interventions are excluded because they could execute the ground-truth experiment rather than predict it.

3.3 Results

None of the three tools improves aggregate prediction over the transcript-only baseline. The result repeats across predictor families, tested budgets, elicitation efforts, and a reasoning-mode target. Tools sometimes change answers correctly, but other changes mislead. In an exploratory NLA analysis, explicit statements linking a cause to its effect appear in only 3 of 479 positive claims (0.6%).

4 Application 2: training models to predict their own behavior

The second application uses the same experimentally labeled claims to train models to predict changes in their own behavior.

4.1 Background: causal self-report and self-explanation

Causal self-report concerns why a particular response occurred, rather than a general personality trait. Calling this “self-explanation” does not imply privileged access to internal state. Prior hint-based training motivates testing transfer to broader natural behaviors.

Refer to caption
Figure 5: Counterfactual prediction training generalizes to the hint setting. Accuracy (left) and AUROC (right) when each model is shown its hinted transcript and asked the binary question: “If the hint were removed from the prompt, would your answer change?”. Both trained models improve substantially over the base model.

4.2 Setup

Fine-tuning appends a counterfactual question to the model’s own conversation and trains a Yes/No answer. Cheaper training investigations sample ten responses per condition and test fewer edits. Qwen3 8B is investigated by Opus; Qwen3.5 397B also investigates itself, reducing the confound of distilling a much stronger investigator.

4.3 Generalization to a held-out setting: hint influence

Without hint-specific training, both models improve on held-out AITA opinion cues and MMLU answer hints. Labels compare repeated cued and uncued responses. The experiments include a transcript-only Opus reference; improved discrimination does not necessarily imply well-calibrated binary decisions.

4.4 Evaluations on held-out investigations

Both trained models improve on held-out WildChat and out-of-source Petri claims and finish within 0.03 AUROC of the Opus reference on each set. Direct prediction does not clearly surpass that reference on filtered data. Free-form explanation training helps the 397B model on hint attribution but not 8B; its explanations can mislead strong external simulators.

Refer to caption
Figure 6: Counterfactual prediction training consistently generalizes to held-out investigations. Each model answers held-out binary claims about its own behavior, drawn from the in-distribution prompt source (WildChat) and an out-of-distribution source (PETRI). Both trained models improve substantially over the base model.

4.5 Testing for privileged access

Qwen3 8B and Llama3.1 8B are each trained on identical data about themselves or the other model. Self-trained predictors do not outperform cross-trained predictors in either direction. The results establish learned behavioral prediction, without evidence of privileged self-access in this test.

5 Discussion and limitations

The null result applies to the tested read-only activation tools and these prompt-counterfactual tasks, not all interpretability methods. Weight/circuit methods, interventions, improved decoders, or better scaffolds remain possible. The benchmark is a proxy: its answers can be measured directly, whereas high-value deployment questions may lack a clean intervention or known ceiling.

6 Conclusion

Counterfactual investigations provide diverse, experimentally grounded training and evaluation material. They support testing specific explanations while leaving open how to explain behavior reliably in settings without observable ground truth.

Acknowledgments

The research was conducted through Anthropic Fellows, with acknowledged feedback from collaborators.

Author Contribution Statement

Adam Karvonen proposed the project, ran experiments, and wrote the paper; Euan Ong and Subhash Kantamneni contributed discussion and feedback; Samuel Marks supervised.

Appendix A Extended discussion

The extended discussion examines the benchmark’s proxy role and the difference between proposing and testing explanations.

A.1 The evaluation proxy argument

Direct sampling could answer each benchmark item, but controlled labels make reliability measurable. Its relevance to harder unverifiable questions depends on how well prompt-edit causes resemble those deployment questions; the paper argues for that connection without establishing equivalence.

A.2 Hypothesis generation vs. hypothesis discrimination

For four published NLA case studies, older Opus models recover the reported explanation from the behavior description alone within their top three guesses in at least 60% of ten samples per cell. These cases motivate hypothesis discrimination, but do not show that hypothesis generation is unimportant in every auditing workflow.

Opus 4.1 Opus 4.5
Behavior top-1 top-3 top-1 top-3
Language switching 9/10 10/10 3/10 6/10
Misreported tool result 10/10 10/10 10/10 10/10
Planning in poetry 2/10 8/10 7/10 10/10
Answer thrashing 10/10 10/10 6/10 10/10
Table 1: Hypothesis generation from a behavior description. For each hypothesis-generation case study in Fraser-Taliente et al. (2026), the fraction of samples in which the cause discovered by the interpretability tool appears as the model’s first hypothesis (top-1) or within its first three (top-3), out of ten samples per cell.

Appendix B Pipeline and dataset details

The appendix documents sampling, prompt sources, claim construction, filters, and reported generation costs.

B.1 Detailed investigation pipeline description

Evaluation uses temperature 1, 30 samples per condition, and a fixed per-investigation binary behavior question. Screening requires at least 30% baseline occurrence and an unexpectedness score of at least 3/5. A separate classifier scores every resampled answer; judges assess support for the report. Evaluation retains support scores of at least 8/10, versus 7/10 for cheaper training runs.

B.2 Prompt sources

The default pool is 91.3% WildChat, 5.2% ToolACE, 2.5% SystemChat, and 1.0% Hermes. Prefix deduplication and length limits reduce repetitive or costly prompts. Out-of-source Petri data come from 173 scenarios run twice, producing 2,061 Qwen8B and 1,732 Qwen397B candidate transcripts after truncation and filtering.

B.3 Evaluation dataset construction and filters

The same 4,714 WildChat-mixture prompts are used across evaluation targets after removing training overlap. Claim wording is generated blind to outcomes. Final filtered sets contain 1,294 Gemma, 1,497 Qwen8B, and 1,076 Qwen397B WildChat claims, plus 1,196 and 580 Petri claims respectively. Initial balancing precedes later filters, so final class counts need not remain exactly equal.

WildChat PETRI
Stage Gemma3 27B Qwen3 8B Qwen3.5 397B Qwen3 8B Qwen3.5 397B
Investigations 1,408 1,320 1,268 625 409
Counterfactual claims 4,433 4,041 3,972 2,125 1,322
  + concreteness ≥3\geq 3, balanced 1,637 1,870 1,309 1,442 708
  + reproducible 1,433 1,666 1,173 1,277 630
  + unconfounded (evaluation dataset) 1,294 1,497 1,076 1,196 580
Table 2: Evaluation construction filters. Each investigation yields up to four counterfactual claims (the unfiltered evaluation dataset). The scored eval set applies Filter 1 (mechanism concreteness ≥3\geq 3), balances the labels 50/50, and caps at 2,000 claims; every agent is scored on it. The evaluation dataset used for all main-body numbers then applies Filter 2 (reproducibility) and Filter 3 (unconfounded). These two are independent post-hoc checks, so their order does not matter.

B.4 Evaluation claim format

The full claim template specifies the behavior, baseline frequency, proposed edit, and a change of at least 30 percentage points in either direction. In the coding example, renaming parameters changes the observed error count from 26/30 to 1/30, exceeding the positive-label threshold.

Counterfactual claim template

BEHAVIOR (the completion is scored "yes" when): {behavior}

BASELINE: on the original prompt at temperature 1.0, this behavior
occurs in ~{baseline}% of completions.

INTERVENTION: {intervention}

PREDICTION: applying this intervention (everything else unchanged)
would change how often the behavior occurs by at least 30
percentage points (in either direction) from the ~{baseline}%
baseline.

Is the CLAIM above true?
Counterfactual claim example

BEHAVIOR (the completion is scored "yes" when): Does the response
describe the randomNum function’s output range as approximately
[min, max) or "between min and max" (i.e., treating the ’max’
parameter as the upper bound of the output), rather than correctly
identifying that the range is [min, max+min) or equivalent?
Answer Yes if the response claims the range tops out at or near
max; answer No if it correctly identifies the range extends to
max+min, or if it doesn’t summarize the range at all.

BASELINE: on the original prompt at temperature 1.0, this behavior
occurs in ~87% of completions.

INTERVENTION: The parameter names of the ‘randomNum‘ function were
renamed from ‘max‘ and ‘min‘ to ‘a‘ and ‘b‘. All other code,
including the function body and all call sites, remains identical.

PREDICTION: applying this intervention (everything else unchanged)
would change how often the behavior occurs by at least 30
percentage points (in either direction) from the ~87% baseline.

Is the CLAIM above true?

B.5 Data generation costs

At the prices used in the study, Opus evaluation investigations cost about $1.9 each and training investigations about $0.9, excluding local target-model serving. The reported runs cost approximately $3.3k and $26k. These are historical experiment estimates, not current service prices or complete compute costs.

Evaluation data Training data
Responses per behavior rate 30 10
Judge samples (stage 4) 5 1
Screening (amortized per inv.) $0.20 $0.08
Investigation $0.96 $0.70
Verification $0.75 $0.15
Total per investigation $1.9 $0.9
Investigations in run 1,736 28,330
Total run cost $3.3k $26k
Table 3: Per-investigation Opus API cost of data generation, separated into the evaluation-data regime and the training-data regime.

Appendix C Interpretability-tool evaluation: details and additional results

Additional evaluations vary tool budgets, predictor families, and output interpretation.

C.1 Tool call details

Tool sources include released activation oracles, Gemma Scope 2 or Qwen batch-top-k SAEs, and corresponding NLAs. Sweeps over 2/5/10 calls and 16/32/64 positions yield paired AUROC changes from −0.012 to +0.004; every reported 95% bootstrap interval includes zero. Giving all NLA outputs on short transcripts also does not help.

Tool calls per claim Token positions per call
Tool 2 5* 10 16 32* 64
AO −0.002-0.002 −0.005-0.005 −0.001-0.001 — — —
NLA −0.008-0.008 −0.012-0.012 −0.006-0.006 −0.010-0.010 −0.012-0.012 −0.006-0.006
SAE +0.003+0.003 −0.005-0.005 −0.001-0.001 +0.002+0.002 −0.005-0.005 +0.004+0.004
Table 4: No tool hyperparameter setting produces uplift. Paired Δ\DeltaAUROC vs. the transcript-only baseline on the Gemma-3-27B-IT final evaluation dataset (n=1,294n=1{,}294), sweeping the tool-call budget (at 32 positions per call) and the token positions per call (at a budget of 5). Starred columns are the main-run configuration. Every 95% bootstrap CI straddles zero.

C.2 Qwen3-8B interpretability tool results

The no-uplift result repeats on Qwen3 8B with the same construction and interface, complementing the Gemma evaluation.

Refer to caption
Figure 7: Interpretability tools give no uplift for the Qwen3-8B target model. Overall counterfactual-prediction score on the Qwen3-8B evaluation dataset, with an activation oracle, a natural-language autoencoder, and a sparse autoencoder. As with Gemma-3-27B-IT (Figure 4), no tool beats the transcript-only baseline.

C.3 Cross-model predictor agent replication

On the 1,294 Gemma claims, baseline AUROC is 0.819 for GPT-5.5, 0.793 for Gemini3.1 Pro, and 0.814 for Opus4.8. No tool beats its own baseline; Gemini’s NLA and SAE scores fall to 0.654 and 0.746.

Agent GPT-5.5 Gemini-3.1-Pro Opus 4.8
Transcript-only baseline 0.819 0.793 0.814
AO 0.810 0.784 0.810
NLA 0.812 0.654 0.808
SAE 0.809 0.746 0.812
Table 5: The no-uplift result replicates across agent families. AUROC per agent on the Gemma-3-27B-IT claims, with GPT-5.5 and Gemini-3.1-Pro replacing Opus 4.8 as the predictor agent. No tool beats its own baseline for any agent; Gemini is actively hurt by the NLA and SAE.

C.4 Exploratory analysis of NLA outputs

On a 952-claim diagnostic subset, NLA supporting clues occur in 47% of positives and 30% of negatives but add only 0.001 AUROC to transcript-based predictions. On the full Gemma set, tools flip 8–16% of answers, mostly toward “no effect,” helping negatives while hurting positives. These LLM-judged diagnostics are exploratory rather than a definitive mechanism.

Claims flipped Flips toward “no effect”
Tool edit changes behavior edit has no effect edit changes behavior edit has no effect
AO 11% 9% 67% 63%
NLA 16% 10% 82% 84%
SAE 13% 8% 76% 71%
Table 6: Tool outputs tend to shift predictions toward no effect. For each tool: the fraction of claims where the tool arm’s binary answer differs from the transcript-only baseline, and the fraction of those flips that move toward predicting no effect. Columns split the claims by ground truth: positive claims, where the edit does change the behavior (n=682n{=}682), and negative claims, where it does not (n=612n{=}612); Gemma-3-27B-IT, Opus 4.8 predictor.

C.5 Qualitative interpretability tool use examples

Earlier-run case studies show NLA evidence sometimes correcting guesses and sometimes reinforcing the wrong causal story. The streamer-default example is misleading: outputs evoke famous streamers while the actual intervention reveals copying from an unrelated tool default. Case-specific probabilities come from earlier runs and are not inputs to final aggregate results.

Appendix D Counterfactual prediction training: details and additional results

Training details distinguish binary prediction targets, free-form report targets, and checks for behavioral drift.

D.1 Training details and target formats

Each example contains the original conversation, a sampled model response, a follow-up question, and the target answer. Language-model loss is applied only to the final assistant turn.

D.1.1 Verbatim target format

The binary task uses the same complete counterfactual claim template as evaluation and a single-token Yes or No target.

D.1.2 Datasets

Balanced binary training contains 36,824 Qwen8B and 40,368 Qwen397B examples. Open-ended training contains one explanation per investigation: 21,780 and 33,922 examples respectively.

Task Qwen3-8B Qwen3.5-397B-A17B
Counterfactual prediction (balanced) 36,824 40,368
Open-ended explanation 21,780 33,922
Table 7: Number of training examples per target model and task.

D.1.3 Training hyperparameters

Both models use rank-64 LoRA and learning rate 5×10⁻⁵, with one epoch for binary prediction and three for explanations. Table 8 preserves optimizer, batch, precision, and sequence-length details, including Tinker defaults that are not exposed separately.

Setting Qwen3-8B Qwen3.5-397B-A17B
LoRA rank rr 64 64
LoRA α\alpha 128 Tinker default
LoRA dropout 0.05 Tinker default
Target modules all linear all linear
Learning rate 5×10−55\times 10^{-5} 5×10−55\times 10^{-5}
Schedule linear warmup (5%) + decay to 0 linear warmup (5%) + decay to 0
Optimizer AdamW (λ=0.01\lambda{=}0.01) Adam (λ=0\lambda{=}0)
Gradient clipping 1.0 1.0
Effective batch size 8 16
Max sequence length 4096 8192
Precision bf16 Tinker default
Table 8: LoRA fine-tuning hyperparameters for the two target models. Settings are shared across both training tasks, which differ only in the number of epochs (one for counterfactual prediction, three for open-ended explanation).

D.2 Hint-setting dataset construction

Hint sets use ten cued and ten uncued samples, require at least 60% cue-consistent answers, and select changes of at least 50 points or at most 15 points. Each set has 150 positives and 150 negatives. Because cues can indicate the correct answer, negative labels correlate with correctness, which may favor a strong external predictor.

D.3 Hint-setting counterfactual prediction: false and true positive rates

The base 8B predictor nearly always answers Yes; training reduces false positives substantially, especially on sycophancy. The 397B predictor rarely answers Yes even after training, so threshold accuracy understates its ranking discrimination. Full FPR/TPR values are retained in Table 9.

Sycophancy MMLU
Model Arm FPR TPR FPR TPR
Qwen3-8B Base 1.00 1.00 0.99 0.99
Trained 0.09 0.82 0.53 0.83
Opus reference 0.12 0.67 0.10 0.75
Qwen3.5-397B-A17B Base 0.00 0.00 0.01 0.19
Trained 0.00 0.11 0.01 0.45
Opus reference 0.03 0.32 0.03 0.62
Table 9: False and true positive rates for hint-setting counterfactual prediction, per setting.

D.4 Privileged access experiment

The self-versus-cross comparison matches training choices and uses 23,944 examples in every arm, with 1,497 Qwen and 1,392 Llama held-out claims. Neither self predictor exceeds its cross predictor. Larger datasets or other distributions could differ; this test does not establish that privileged access is impossible.

Refer to caption
Figure 8: No evidence of privileged access in either direction. For each target model, the self predictor (trained on its own claim data) does not exceed the cross predictor (the other model trained on the same, size-matched data).

D.5 Preserving the target model’s behavior with KL regularization

A preliminary Qwen8B run adds full-vocabulary KL regularization on 1% general-chat samples. It keeps measured transcript KL around 0.03 and preserves thinking ability better than ordinary non-thinking SFT, but does not improve counterfactual scores. This mitigates label drift in the tested setup without proving unchanged behavior on all prompts.

Appendix E Open-ended explanation training

Open-ended training asks for a behavior explanation accompanied by supporting and refuting experiments. Unlike binary labels, these narrative targets can contain incorrect or incomplete causal interpretations.

E.1 Training target

The structured target lists observed rates, supported causes, negative evidence, and a final answer. The full template and coding example are preserved. Structured reports provide no consistent gain over short explanatory paragraphs in the authors’ comparison.

Structured report template

Observed behavior
<the behavior and its measured baseline rate>
Core causes
1. <cause>
   Status: supported
   Evidence:
   - <counterfactual edit -> measured rate change>
2. ...
Negative evidence
A. <refuted hypothesis>
   Status: refuted
   Evidence:
   - <counterfactual edit that did not move the behavior>
Final answer
<narrative summary of the causes and the counterfactuals that
 demonstrate them>
Structured report example

Observed behavior

When explaining the randomNum(max, min) function, I correctly
trace each mathematical step -- multiply Math.random() by max,
floor the result, add min -- but then about 87% of the time I
incorrectly summarize the output range as [min, max) or
"between min and max." My step-by-step analysis shows the
range is actually [min, max+min), yet my summary contradicts
my own traced steps.

Core causes

1. The co-occurrence of "max" and "min" as parameter names
   creates a semantic prior that overrides my own mathematical
   reasoning
Status: supported
Evidence:
- When both parameters are renamed to neutral names like "a"
  and "b," I correctly identify the range as [b, a+b) about
  97% of the time, versus only about 13% with "max" and "min."
- When only "max" is replaced with "a" but "min" is kept, I
  correctly identify the range about 83% of the time -- the
  presence of "min" alone is not enough to trigger the error
  at high rates.
- When only "min" is replaced with "offset" but "max" is kept,
  I correctly identify the range about 93% of the time -- the
  presence of "max" alone is also not enough.
- When both are replaced with descriptive but non-bound names
  like "multiplier" and "offset," I correctly identify the
  range about 90% of the time, confirming the issue is
  specifically about names that imply boundary semantics.

2. Names that suggest upper/lower bounds produce
   proportionally more errors, even when they are not
   literally "max" and "min"
Status: supported
Evidence:
- When the parameters are named "upper" and "lower," I still
  incorrectly claim the range tops out at "upper" about 57% of
  the time -- much higher than with neutral names but lower
  than the about 87% with "max" and "min."
- When the parameters are named "range" and "offset," I get it
  wrong only about 20% of the time, in between purely neutral
  names and bound-suggesting names.

Negative evidence

A. The surrounding rain-effect code context drives the error
Status: refuted
Evidence:
- When the randomNum(max, min) function is presented in
  complete isolation without any surrounding code, I still get
  the range wrong about 93% of the time, essentially the same
  as the about 87% with the full rain code. The surrounding
  context plays no role.

B. I am deliberately avoiding calling out a bug to stay
   helpful
Status: refuted
Evidence:
- When explicitly asked to find bugs in the function, I still
  describe the range incorrectly about 43% of the time. While
  bug-finding instructions help somewhat, the error persists
  at a substantial rate, showing the primary driver is the
  parameter-name prior rather than a reluctance to identify
  bugs.

C. A comment explaining the actual mechanics of the parameters
   overrides the naming prior
Status: refuted
Evidence:
- When a comment is added stating "’max’ is used as the
  multiplier for Math.random(), and ’min’ is added as an
  offset," I still get the range wrong about 73% of the time,
  barely reduced from the about 87% baseline. The parameter
  names overpower even an explicit inline explanation.

Final answer

I incorrectly summarize the output range because the parameter
names "max" and "min" together create an overwhelming
expectation that the function produces values in [min, max).
Neither name alone triggers the error at high rates -- with
"max" and "offset" I get it right about 93% of the time, and
with "a" and "min" I get it right about 83% of the time. It is
specifically the co-occurrence of both bound-implying names
that causes me to override my own correctly traced
mathematical steps. The effect scales with how strongly the
names suggest bounds: "upper"/"lower" produces about 57%
errors, "range"/"offset" about 20%, and purely neutral "a"/"b"
only about 3%. The surrounding code context, bug-avoidance
tendencies, and even explicit comments explaining the
parameters’ true roles all fail to overcome the semantic prior
created by the max/min naming pair.

E.2 Generalization to the hint setting

The 397B model improves hint attribution after explanation training; 8B does not. The Opus reference always attributes AITA verdicts to the hint and is therefore at chance on that balanced subtask. False-positive and true-positive rates qualify aggregate accuracy.

Refer to caption
Figure 9: Open-ended explanation training generalizes to the hint setting for the 397B target but not the 8B. Each model is asked to explain its answer and judged on whether it attributes the answer to the injected cue, pooled over the sycophancy and MMLU settings. The Opus reference is asked the same question in the third person about the target’s transcript. Only the trained 397B improves over its base model, reaching the Opus reference.
Sycophancy MMLU
Model Arm FPR TPR FPR TPR
Qwen3-8B Base 0.01 0.01 0.01 0.09
Trained 0.82 0.84 0.58 0.59
Opus reference 1.00 1.00 0.43 0.90
Qwen3.5-397B-A17B Base 0.00 0.00 0.03 0.38
Trained 0.31 0.61 0.29 0.70
Opus reference 1.00 1.00 0.36 0.91
Table 10: False and true positive rates for hint-setting with open-ended self-explanation, per setting (n=150n{=}150 positives / 150 negatives each). FPR == TPR ≈0\approx 0 means the model never attributes to the hint (the base models); FPR == TPR ≈1\approx 1 means it always attributes (Opus on sycophancy).

E.3 Counterfactual simulatability on held-out investigations

Across five external simulator models, trained explanations help weaker predictors but hurt stronger ones. Base explanations provide no uplift for either target. Confident, specific but incorrect counterfactual claims can reduce the accuracy of an observer who already reads the transcript well.

Refer to caption
Figure 10: Open-ended explanation training helps weak simulators but hurts strong ones (Qwen3-8B target). For counterfactual simulatability, each simulator model predicts the held-out claims given the transcript and the target model’s self-explanation, compared against the same simulator given the transcript alone. The base model’s explanations provide no uplift. The trained explanations help the weakest simulators but give a consistent downlift for the strongest ones, which have enough capability that the confident, specific, and often wrong explanations can mislead them.
Refer to caption
Figure 11: Counterfactual simulatability on the Qwen3.5-397B-A17B target (WildChat evaluation). As for the Qwen3-8B target (Figure 10), each simulator predicts the held-out claims given the transcript plus the target model’s self-explanation, compared against the same simulator given the transcript alone. The base model’s explanations give no uplift; the trained explanations help the weakest simulators but produce a downlift for the strongest.

E.4 Discussion: interpreting the mixed results

Both tested explanation metrics have limitations: hint tasks differ from training, and simulator scores depend on capability. Direct LLM judgment is unreliable on near-matching explanations. A restricted trigger-mention check improves after training, but shares the generating pipeline and is only preliminary evidence.

Appendix F Unfiltered dataset results

Removing all filters preserves the tool null result. On unfiltered WildChat claims, trained binary predictors slightly exceed Opus: paired AUROC gains are +0.015 [0.002, 0.027] for Qwen8B and +0.033 [0.021, 0.044] for Qwen397B. Gains concentrate on vague-mechanism claims excluded by the main filters.

Refer to caption
Figure 12: The interpretability tools give no uplift on the unfiltered dataset (Gemma-3-27B-IT, n=4,433n{=}4{,}433).
Refer to caption
Figure 13: The trained counterfactual-prediction models slightly exceed the Opus reference when unfiltered (WildChat; n=4,041n{=}4{,}041 for Qwen3-8B, 3,9723{,}972 for Qwen3.5-397B).

Appendix G Unfaithful chain of thought in the wild

Reasoning-mode Qwen8B also shows no tool uplift, including a harder uncertain-baseline subset. Qualitative cases expose causes omitted or misrepresented in visible reasoning: moving a show from first to last changes selection from 30/30 to 4/30, and replacing a historical figure’s name with a fictional one changes a portrayal from 29/30 to 3/30. The larger-model reasoning dataset is released without systematic analysis.

Refer to caption
Figure 14: Interpretability tools give no uplift on reasoning-mode data either. The interpretability evaluation of §3, rerun on Qwen3-8B with reasoning enabled. No tool predictor beats the transcript-only baseline, reproducing the non-thinking result on transcripts that now contain an explicit reasoning trace.
Concept demonstration

Open image ↗