Literature
Display
← All papers

Re:Form - Reducing Human Priors in Scalable Formal Software Verification with RL in LLMs: A Preliminary Study on Dafny

Chuanhao Yan · Fengdi Che · Xuhan Huang · Xu Xu · Xin Li · Yizhi Li · Xingwei Qu · Jingzhe Shi · Chenghua Lin · Yaodong Yang · Binhang Yuan · Hang Zhao · Yu Qiao · Bowen Zhou · Jie Fu

· Transactions on Machine Learning Research ↗

Reasoning17 min text · CondensedOriginal paper ↗
Contents

Overview

This edition follows the complete 52-page v4 source, published in TMLR in May 2026 and revised in July 2026. Its title uses “Reducing Human Annotations,” replacing “Human Priors” in the original catalog title. The author list is unchanged. All 32 numbered figures, 11 tables, three algorithms, three prompt boxes, equations and 95 references are retained; code panels also keep their complete downloadable text.

At a glance

Re:Form trains language models to annotate Dafny programs with machine-checkable specifications. Supervised fine-tuning teaches the output format, then reinforcement learning rewards syntax, verification and logical comparison with reference contracts. A subset reward discourages easy but uninformative specifications. Models improve in-domain verification and reference-relative specification strength, with smaller absolute gains on synthetic out-of-domain compositions. The study supports automated specification training, while leaving specification completeness, faithful translation and transfer to large software systems unresolved.

1 Introduction

Natural-language reasoning can be hard to evaluate reliably. Dafny offers a more constrained setting: specifications state input assumptions, output guarantees and loop invariants, and a verifier checks whether code satisfies them [38]. A program can nevertheless verify against a weak contract. Re:Form therefore separates syntax validity, verifier success and the strength of the generated contract relative to a reference.

The pipeline reduces additional manual annotations during reinforcement learning, but still relies on pretrained code models, teacher-generated labels, curated code, supervised training, human-designed rewards and a trusted formal toolchain. Absence of an explicit natural-language reasoning trace is an experimental choice, not proof that reasoning traces or human knowledge are unnecessary in general.

Original native SVG figure
Figure 1: Model Performance on DafnyComp benchmark. We roughly categorize the cases into four groups in an ascending quality order: Syntax Error, Syntax Correct, Verified and Verified with Superior Specification. The growing proportion of Verified with Superior Specification suggests a rudimentary of exploration capabitliy to generate stronger specifcation than the ground-truth of the models incentivized by reinforcement learning. For further model behavioural analysis, a more refined set of definition of benchmarking metrics is provided in Section 3.1 as standard protocol.
Original native SVG figure

Figure 2: The Illustration of Re:Form Pipeline. Human prior is extensively removed across different components of the pipeline. Heuristic cleansing rules and model-based conversion are introduced in the data construction and benchmark annoation for scaling along with compute investment. The task is formalized as a simple and flexible specification generation, providing the model a vast landscape of self-exploration under the reinforcement learning paradigm.

2 Pipeline

The pipeline curates verified training programs, constructs a compositional evaluation benchmark, performs supervised fine-tuning and then optimizes outcome-based formal rewards. Figures 1–3 and Algorithms 1–3 preserve the complete design and data flow.

2.1 Data Curation

Native Dafny sources are combined with Python programs translated by a language model. Claude 3.5 Sonnet is chosen after a 100-example comparison; verification feedback supports up to ten repair attempts. Functions and specifications are generated in stages, and only successful verified programs are retained. Heuristic filtering removes unwanted annotations and comments. This automates curation but conditions the dataset on teacher and verifier success.

The paper describes roughly 20,000 programs. Table 1 instead lists 0.9 thousand MetaReflection, 0.3 thousand BigCode and 16.3 thousand Python2Dafny examples, totaling about 17,500. This discrepancy is unresolved. Passing a Dafny verifier establishes the generated Dafny program’s supplied contract, not equivalence to the original Python program or completeness of its intended behavior.

Table 1: Statistics of the Dataset.
Data Source N# N#Spec N#Token
MetaReflection 0.9 k 6.53 318.57
BigCode 0.3 k 24.5 766.13
Python2Dafny 16.3k 16.94 601.71
Original native SVG figure
Figure 3: Specification Type Distribution.
Algorithm 1 Python-to-Dafny Translation and Specification Generation
Original alg1 from the complete source PDF

2.2 Benchmark

DafnyComp adds specification-strength evaluation to a setting where verifier pass rate can be inflated by weak contracts. It separates held-out examples from the curation distribution from newly assembled Python-function compositions. DafnyBench supplies an additional prior benchmark [48]. “Out of domain” here means this specific synthetic composition procedure, not unrestricted industrial software.

Algorithm 2 Python-to-Dafny Translation with Iterative Verification and Repair
Original alg2 from the complete source PDF

2.2.1 Program Assembly Stage

The assembly stage selects single-input, single-output LeetCode functions with McCabe complexity above five [87]. A frontier model constructs call-graph templates, combines functions and supplies structural variations. Formatting, import completion, input/output matching and existing tests filter the resulting Python programs. These checks make composition practical but do not prove the full generated program correct.

2.2.2 Formal Translation Stage

Translation follows the program’s abstract syntax structure: individual nodes are generated and verified before combination. The system repairs initial code and then annotates main and subsidiary functions, retaining only successfully verified pairs. Algorithm 3 preserves both stages. Selection for successful formalization can exclude harder programs and does not establish Python-to-Dafny semantic equivalence.

2.3 Pattern Activation through Supervised Fine-tuning

Given implementation c, the model emits the entire program together with specifications, written c ⊕ y = π(c), rather than only an annotation fragment. Supervised fine-tuning is intended to teach Dafny syntax and basic semantic patterns before reinforcement learning. The curated training outputs exclude explicit natural-language chains of thought and comments. Appendix A.8’s printed SFT prompt fixes existing preconditions and postconditions and asks for missing invariants/assertions, a narrower task than the main full-specification description; the paper does not reconcile this difference.

Algorithm 3 Dafny Benchmark Generation Pipeline
Original alg3 from the complete source PDF

2.4 Exploration with Reinforcement Learning

The first reward scheme checks syntax/type validity and Dafny verification. The second adds a subset comparison: reference preconditions must imply generated preconditions, and, under the reference preconditions, generated postconditions must imply reference postconditions. Both original implications and the GRPO objective/normalized group advantage are retained below [67]. The approach draws on reference-comparison ideas in Clover and Verina [72][89].

Equality satisfies these implications: “spec superiority rate” includes contracts that are merely as strong as the reference. The postcondition comparison is restricted to the reference input domain. It therefore does not prove globally weakest preconditions, globally strongest postconditions or strictly superior behavior. Because reference contracts determine reward, they still provide supervision even without a person labeling each new rollout.

GRPO compares sampled outcomes within each group. Experiments contrast verification rewards, additional subset rewards, and subset rewards with KL and entropy regularization. Verification is relative to the supplied contract and trusted language/solver implementation [15]. Incompleteness can reject valid statements; a failed implication check alone does not prove strict semantic weakness.

Equations 1–2GTpre⇒GENpre\mathrm{GT}_{\mathrm{pre}}\Rightarrow\mathrm{GEN}_{\mathrm{pre}}GTpre⇒(GENpost⇒GTpost)\mathrm{GT}_{\mathrm{pre}}\Rightarrow(\mathrm{GEN}_{\mathrm{post}}\Rightarrow\mathrm{GT}_{\mathrm{post}})
𝔼c∼P⁡(C){yi}i=1G∼πθold(⋅∣c)[1G∑i=1Gmin(πθ​(yi∣c)πθold​(yi∣c)Ai,clip(πθ​(yi∣c)πθold​(yi∣c), 1−ϵ, 1+ϵ)Ai)−βDKL(πθ∥πref)],\displaystyle\mathbb{E}_{\begin{subarray}{c}c\sim P(C)\\ \{y_{i}\}_{i=1}^{G}\sim\pi_{\theta_{\mathrm{old}}}(\cdot\mid c)\end{subarray}}\Biggl[\frac{1}{G}\sum_{i=1}^{G}\min\Bigl(\frac{\pi_{\theta}(y_{i}\mid c)}{\pi_{\theta_{\mathrm{old}}}(y_{i}\mid c)}\,A_{i},\;\mathrm{clip}\bigl(\tfrac{\pi_{\theta}(y_{i}\mid c)}{\pi_{\theta_{\mathrm{old}}}(y_{i}\mid c)},\,1-\epsilon,\,1+\epsilon\bigr)\,A_{i}\Bigr)-\beta\,D_{\mathrm{KL}}(\pi_{\theta}\|\pi_{\mathrm{ref}})\Biggr], (1)
Ai=ri−mean​({rj}j=1G)std​({rj}j=1G).A_{i}=\frac{r_{i}-\text{mean}(\{r_{j}\}_{j=1}^{G})}{\text{std}(\{r_{j}\}_{j=1}^{G})}. (2)

3 Results and Analysis

Results distinguish single-sample success from whether any of many samples succeeds. The detailed appendix tables and all learning curves are retained, including outcomes that qualify the headline claims.

3.1 Experiment Setup

The study varies model size, reward and regularization within a Qwen-based pipeline, using held-out in-domain programs, synthetic compositions and a DafnyBench sample.

Models

Models use pretrained Qwen 2.5 checkpoints from 0.5 to 14 billion parameters [25]. The 0.5B starting model is distilled from larger models, so its data and teacher costs differ from straightforward fine-tuning. The authors report no obvious benefit from 32B in preliminary exploration, without a detailed controlled scaling result.

Dataset

The main text specifies 3,000 SFT examples, 4,500 separate RL examples, 512 held-out in-domain programs, 300 synthetic DafnyComp compositions and 100 sampled DafnyBench programs. Appendix A.7 instead states 5,000 SFT examples. Both counts are preserved as an unresolved reporting discrepancy. The held-out split avoids reuse during these training stages but does not audit every possible pretrained overlap.

Training Details

RL uses four rollouts per input, temperature 1.0, batch size 1,024 and learning rate 10⁻⁵. The main setting adds KL coefficient 0.01 and entropy coefficient 0.02. On A800 GPUs, the 3B run takes about 20 hours on 32 GPUs for 40 epochs. Table 2 reports all sizes; its largest run takes about 36 hours on 64 GPUs. These figures exclude teacher synthesis, distillation, SFT searches and other experiments.

Table 2: This table reports the RL training requirements using the subset reward model, including the number of GPUs and the approximate wall-clock training time for various model sizes.
Model Size 0.5B 1.5B 3B 7B 14B
Number of GPUs 16 16 32 64 64
Training Time (hours) ≈\approx 11 25 20 20 36

Evaluation Metrics

Validity measures syntax/type success; verification measures passing the generated contract; SSR measures generated specifications that meet the reference-relative subset comparison. Pass@1 uses a single sample. The plotted best-of-k success counts an input as solved when at least one of k rollouts attains the reward, so higher sampling budgets entail additional inference and verification cost.

3.2 Main Experiment Results

Fine-tuning improves formal output substantially, and RL generally increases both verification and reference-relative specification strength. Reward choice changes their balance.

Absence of CoTs

The authors find comments in roughly 2% of RL outputs, usually after completed code, and do not observe extended explanatory reasoning. Figure 4 retains an example. This describes the trained output distribution; it does not reveal all internal computation or compare against a matched reasoning-trace training treatment.

// This program prints Hello World!
// println!("Hello World!");
Figure 4: The figure presents an example of comments generated during RL learning, which is not extended reasoning and is inserted after the complete Dafny code.

Improvment from SFT

For 3B, SFT reaches 48.0% verification and 26.6% SSR; the main regularized RL setting reaches 73.4% and 42.0%. For 14B, SFT reaches 62.9% and 34.2%, while main RL reaches 84.0% and 53.9%. The detailed tables also show that unregularized subset RL has higher pass@1 SSR, 44.7% at 3B and 55.3% at 14B. Regularization is selected primarily for exploration under multiple rollouts, not uniform single-sample superiority.

Original native SVG figure
Figure 5: The figure shows the comparison between GPT-4o, our Qwen base models, SFT models and RL-trained models scaling over model size on our in-domain evaluation set. The pass@1 improvement of SFT and subsequent RL over our base models is substantial.
Original native SVG figure
Figure 6: The figure reports SFT and RL performance with 128128 rollouts. The plotted rate measures whether at least one rollout attains the corresponding reward. RL yields a clear improvement from SFT, indicating genuine quality gains rather than mere compression of rollouts.

Exploration Analysis

Figure 6 shows higher best-of-128 success with entropy-regularized subset training than SFT. The authors also compare generated postconditions with the aggregate of 128 SFT samples and report specifications outside that finite reference set. This provides evidence of different discovered contracts, not proof that the SFT model could never generate them or that they exceed all human knowledge. Figures 7 and 23–25 preserve examples.

method ApplyFading(input: seq<real>, selective: bool) returns (output: seq<Complex>)
    ensures |output| == |input|
    //////// ↓ The novel specification
    ensures forall i :: 0 <= i < |input| ==> output[i].r == input[i] * (if selective then k else 4.0)
    //////// ↑

{
    var result: seq<Complex> := [];
    var i := 0;
    while i < |input|
        invariant 0 <= i <= |input|
        invariant |result| == i
        //////// ↓ The novel specification (found in RL and SFT results, but not in ground truth)
        invariant forall j :: 0 <= j < i ==> result[j].r == input[j] * (if selective then k else 4.0)
        //////// ↑
    {
        var fadeValue := if selective then k else 4.0;
        var complex := new Complex(input[i] * fadeValue, 0.0);
        result := result + [complex];
        i := i + 1;
    }
    output := result;
}
Figure 7: First example of novel specifications that never show up in the SFT model’s 128128 rollouts.

OOD-generalization

On the synthetic out-of-domain set, 14B main RL reaches 14.0% verification versus 8.3% for SFT and 0.3% for GPT-4o. SSR rises from 4.7% to 7.7%: a 3.0 percentage-point increase, approximately 63.8% relative. Most problems still fail. Figure 8’s caption also identifies GPT-4o as best on DafnyBench, so the broader claim of dominance across every dataset should not be read literally.

New compositions reduce simple exact-example reuse, but constituent code and model pretraining are not fully audited. Low base-model scores do not rule out contamination or related-code familiarity [18][63].

Original native SVG figure
Figure 8: Our 1414B RL model dominates the pass@​1@1 performance over SFT and GPT-4o, the best performing proprietary LLM other than our data-generator, Claude. Notably, GPT-4o attains the best score on DafnyBench, highlighting an asymmetry toward that benchmark.

Summary

The strongest supported result is improved formal annotation under the reported tasks and budgets. Its scope is narrower than a universal claim that RL discovers reasoning beyond pretraining: the reference-dependent metric, teacher-curated data, finite SFT comparison and low absolute compositional success remain material qualifications.

3.3 Ablation Study

The ablations expose reward hacking and an exploration/stability trade-off. Both are visible in the retained learning curves.

Comparison between Reward Schemes

Verification-only optimization raises the pass rate while allowing weaker contracts. At 3B, it reaches 85.2% verification but 30.7% SSR, compared with 73.4% and 42.0% for the main subset setting. The subset reward sacrifices some verifier success to improve the reference-relative contract measure. Trivial contracts are one failure mode, alongside unverifiable and insufficiently informative specifications.

Original native SVG figure
Original native SVG figure
Figure 9: These figures present the training curves for different reward schemes and regularization choices. The left figure shows that using the subset reward stops the quality drop, demonstrated by the spec superiority rate. The right figure shows that entropy regularization leads to instability in training, and all regularization choices show similar pass@1 performance before crashing.

Effects of Regularization

Entropy regularization improves best-of-128 exploration but introduces unstable training and collapses in some curves. KL alone and unregularized training can have strong pass@1 yet poor multi-sample coverage. Checkpoint selection therefore matters; the paper does not fully specify an equalized selection policy across configurations.

Trailing tokens after complete code rise from about 1% of SFT outputs to about 80% with entropy RL. This differs from the 2% comment statistic. Tokens generated after the answer cannot guide earlier autoregressive tokens, and their presence shows that entropy can reward extra output without useful exploration.

Original native SVG figure
Figure 10: This figure compares the spec superiority rate (SSR) among three RL configurations: the subset reward with entropy only, with KL divergence added only and without any regularization. Adding entropy regularization is the key to an increasing pass@​128@128 performance, which injects stochasticy and thus encourages exploration. The combination with KL divergence can further improve the performance, and thus, we stick to this configuration.

4 Conclusion and Discussion

Formal outcomes provide an effective training signal, and comparing contracts addresses a weakness of verifier-only rewards. The study does not establish arbitrary-program correctness: the verifier checks a formal specification, whose usefulness, completeness and correspondence to intended behavior remain separate concerns. Teacher quality, selection effects, solver limits and task/prompt inconsistencies constrain the interpretation.

Connection to practical software settings

Six small translated Python-style examples connect the method to hidden assumptions and simple bugs. The authors describe three verified and three rejected outputs, but Appendix B.5 says one search failure is a generated annotation indexing error and one verified absolute-difference example has a weak postcondition. These outcomes cannot all be counted as successful bug detection. Repository-scale changes, realistic integration and broader debugging utility are not evaluated.

5 Related Work

The work sits between code generation, formal verification and reinforcement learning for reasoning, with an emphasis on automatically evaluating specifications rather than natural-language explanations.

5.1 LLMs in Software Engineering

Earlier systems learn program generation from code and execution feedback [6][36]. Dafny synthesis, DafnyBench and annotation tools apply language models to verification [48][52][58]. Clover, Verina and other verified-generation benchmarks address relationships among code, specification and intent [72][89]. Re:Form contributes a curated training pipeline and reward study; it does not replace the need to decide what behavior a contract should express.

5.2 Informal vs. Formal Reasoning in LLMs

Natural-language chains of thought can aid problem solving but need not faithfully expose internal reasoning [35][8]. Formal proving offers checkable conclusions within its formal system [88]. Re:Form evaluates code-only outputs in one such environment. This does not settle the general value of explanations, transfer from formal tasks, or whether formal annotations capture a user’s actual requirements.

Broader Impact Statement

The authors propose formal verification as a component of more reliable AI and speculate about formalized oversight or debate [14][27]. Those applications are not tested. Any practical assurance would still depend on the specification, trusted tools and how the formal model maps to the deployed system.

Appendix

The appendix preserves language background, exact examples, training/search settings, complete prompts, model tables, learning curves, novelty/diversity measures and failure analysis.

Appendix A Techinical Details and Methods

Technical details make clear which annotations are inferred, which are supplied, and how generated contracts are compared.

A.1 Brief Introduction to Dafny

Dafny combines executable code with specifications checked using automated reasoning [38]. Table 3 contrasts Python’s dynamic execution with Dafny’s static types, contracts and formal verification. A precondition describes assumed inputs, a postcondition the guaranteed result, and loop invariants properties maintained during iteration. This distinction separates passing a verifier from faithfully formalizing a Python program’s intent.

Table 3: Key differences between Dafny and Python.
Feature Dafny Python
Year Introduced 2010 (Microsoft Research) 1991 (Guido van Rossum)
Type System Static typing, compile-time checks Dynamic typing, run-time checks
Formal Verification Yes — built-in contracts and proofs No — only basic assert
Main Use Verified algorithms, critical systems General-purpose programming
Execution Model Compiled with verification Interpreted (e.g., CPython)

A.2 Notation List

Table 4 retains the notation for source programs, generated code, preconditions, postconditions, reference specifications and model rewards. “Ground truth” denotes the curated reference contract; it is not a guarantee that every relevant behavioral requirement is represented.

Table 4: Notations and terms used in this paper
Symbol Description
Policy π\pi The LLM Model or Policy
Code implementation cc The raw code body without specifications
Spec/Specification yy A formal description of what a program is supposed to do, acting as a contract between the program and its clients to guide verification
Dafny verifier An automatic theorem prover to check the consistency of the specifications with the code
Precondition A condition that must be true before running a piece of code, and thus sets the admissible input domain
Postcondition A condition that must be true after running a piece of code and guarantees the output ranges
requires A precondition in Dafny
ensures A postcondion in Dafny
invariant A condition that holds true during loop iterations
Clause One line specification, such as
ensures |nearbyStops| <= |stops|
GT The ground truth specifications generated by Claude
GTpre\mathrm{GT}_{\mathrm{pre}} The intersection of preconditions in the ground truth
GENpre\mathrm{GEN}_{\mathrm{pre}} The intersection of generated preconditions
GTpost\mathrm{GT}_{\mathrm{post}} The intersection of postconditions in the ground truth
GENpost\mathrm{GEN}_{\mathrm{post}} The intersection of generated postconditions
Syntax reward A reward assigned based on whether the generated specifications pass compilation
Verification reward A reward assigned based on whether the generated specifications are consistent with the given code, which can be checked by the Dafny verifier
Subset relation For formal statements AA and BB, if A⇒BA\Rightarrow B, then AA is a subset of BB, denoted as A⊂BA\subset B
Superior specifications A set of specifications with weaker preconditions and stronger postconditions
Subset reward A reward assigned based on whether the generated specifications are superior to or at least as strong as the ground truth
Validation Rate Percentage of generated programs without syntax error
Verification rate Percentage of generated specifications that are verified to be consistent with the code by Dafny
Spec Superiority Rate Percentage of generated specifications superior to or at least as strong as the corresponding ground truth
Novel Specification A non-trivial postcondition unseen in any of the 128128 SFT rollouts

A.3 An Example of Specification and Implementation

Figures 11–12 show the same summation implementation with and without annotations. The annotated version assumes n ≥ −1, ensures s = n(n+1)/2, and maintains s = i(i−1)/2 with 0 ≤ i ≤ n+1. The complete original code is retained, including the distinction between implementation and specification comments.

requires n >= -1
ensures s == n * (n + 1) / 2
invariant s == i * (i - 1) / 2
invariant 0 <= i <= n + 1
method Sum(n: int) returns (s: int)
  requires n >= -1  // Specification
  ensures s == n * (n + 1) / 2  // Specification
{
    var i := 0;   // Implementation
    s := 0;       // Implementation
    while i <= n  // Implementation
      invariant s == i * (i - 1) / 2  // Specification
      invariant 0 <= i <= n + 1  // Specification
    {
        s := s + i;  // Implementation
        i := i + 1;  // Implementation
    }
}
Figure 11: A toy example of a Dafny implementation with full specification, including precondition, postcondition, and loop invariants.
method Sum(n: int) returns (s: int)
{
    var i := 0;   // Implementation
    s := 0;       // Implementation
    while i <= n  // Implementation
    {
        s := s + i;  // Implementation
        i := i + 1;  // Implementation
    }
}
Figure 12: The same implementation as in Figure 11, shown without any specification annotations.

A.4 An Example of the Data Curation

Figure 13 preserves the Python permutation example and generated Dafny counterpart. Complete code panels remain available beneath the original diagram, including long lines clipped at the PDF panel edges. The displayed contract preserves array contents after permutation; it does not completely characterize printed output or independently establish faithful translation.

Original A1.F13 from the complete source PDF
Figure 13: Example of Python-to-Dafny conversion from our data curation pipeline. The left side shows the original Python permutation function, while the right side displays the automatically generated Dafny code with formal specifications including preconditions, postconditions, loop invariants, and termination measures.
Complete original code panels

Python code

def perm_3(k):
    if k == N:
        print(arr[0], arr[1], arr[2])
    else:
        for i in range(k, N):
            arr[i], arr[k] = arr[k], arr[i]
            perm_3(k+1)
            arr[i], arr[k] = arr[k], arr[i]

arr = [1, 2, 3, 4, 5, 6, 7]
N = len(arr)
perm_3(0)

Generated Dafny code

method Perm3(arr: array<int>, k: int)
  requires arr.Length >= 3
  requires 0 <= k <= arr.Length
  modifies arr
  ensures arr[..] == old(arr[..])
  decreases arr.Length - k
{
  if k == arr.Length {
    Print(arr[0], arr[1], arr[2]);
  } else {
    var i := k;
    while i < arr.Length
      invariant k <= i <= arr.Length
      invariant arr[..] == old(arr[..])
      invariant forall j, l :: k <= j < i && k <= l < i && j != l ==>
                               arr[j] == old(arr[j]) && arr[l] == old(arr[l])
    {
      Swap(arr, k, i);
      Perm3(arr, k + 1);
      Swap(arr, k, i);
      i := i + 1;
    }
  }
}

method Swap(arr: array<int>, i: int, j: int)
  requires 0 <= i < arr.Length && 0 <= j < arr.Length
  modifies arr
  ensures arr[i] == old(arr[j]) && arr[j] == old(arr[i])
  ensures forall k :: 0 <= k < arr.Length && k != i && k != j ==>
                      arr[k] == old(arr[k])
{
  var temp := arr[i];
  arr[i] := arr[j];
  arr[j] := temp;
}

method Print(a: int, b: int, c: int)
{
  print a, " ", b, " ", c, "\n";
}

method Main()
{
  var arr := new int[7];
  arr[0] := 1; arr[1] := 2; arr[2] := 3; arr[3] := 4;
  arr[4] := 5; arr[5] := 6; arr[6] := 7;
  Perm3(arr, 0);
}

A.5 An Example of the Subset Reward

Figure 14 illustrates the two implication checks. Its reference requires n > 0 and k > n and guarantees k_out ≥ 0. The candidate admits n ≥ 0 and k ≥ 0 and states k_out = k − n. The comparison checks relaxation and conditional strengthening, rather than equating reference-relative success with every possible desirable specification.

Original A1.F14 from the complete source PDF
Figure 14: On the top right block, we present the input code and show the extracted method preconditions and postconditions on the top left blocks. In the bottom block, we show the comparison clauses to check the superiority of specifications. Then, we reinsert the comparison clause into the input code and verify the relationship using the Dafny verifier.
Complete original code panels

Code

method main(n: int, k: int) returns (k_out: int)
{
    k_out := k;
    var j: int := 0;
    while(j < n)
    {
        j := j + 1;
        k_out := k_out - 1;
    }
}

Ground truth

requires n > 0
requires k > n
ensures k_out >= 0

Generated specifications

requires n >= 0
requires k >= 0
ensures k_out == k - n

Comparison clause

// Check whether generated specifications have weaker preconditions
assert ( (n >= 0) && (k >= 0) ) <== ( (n > 0) && (k > n) );


/* Check under the precondition of the ground truth,
whether generated postconditions are stronger */
assert ( (n > 0) && (k > n) ) ==>((k_out == k - n) ==> k_out >= 0);

A.6 Distillation Details of the 0.5B Model

The 0.5B model is distilled from fine-tuned 7B or 14B teachers. Table 5 varies supervised versus sequence-level distillation, forward KL, reverse KL or Jensen–Shannon divergence, temperature one or two, and a base versus fine-tuned student. Table 6 lists four preferred configurations without their scores or a fully explicit mapping to every subsequent 0.5B result. These resources should be included when comparing small-model efficiency.

Table 5: Knowledge distillation experiment design space. Abbreviations: SKD = Supervised Knowledge Distillation, SeqKD = Sequence-Level Knowledge Distillation, KLD = Forward KL divergence, RKL = Reverse KL divergence, JSD = Jensen-Shannon divergence.
Category Options
Distillation Algorithm SKD, SeqKD
KL Loss KLD, RKL, JSD (α=0.5\alpha=0.5)
Temperature T=1T=1, T=2T=2
Student Model SFTed 0.5B, Base 0.5B
Teacher Model SFTed 7B, SFTed 14B
Table 6: The four best-performing distillation configurations identified.
Distillation Algorithm KL Loss Temperature Student Model Teacher Model
SKD JSD (α=0.5\alpha=0.5) 1 Base 0.5B SFTed 7B
SeqKD RKL 1 SFTed 0.5B SFTed 7B
SeqKD JSD (α=0.5\alpha=0.5) 1 Base 0.5B SFTed 14B
SKD RKL 2 SFTed 0.5B SFTed 7B

A.7 SFT Training Hyperparameter Grid Search Details

SFT uses one eight-A800 server, ZeRO stage 3, cosine learning-rate decay and 10% warm-up. Per-GPU batches decrease with model size. The appendix states 5,000 examples and under 40 minutes per model, differing from the main text’s 3,000 examples; the total hyperparameter-search cost is not reported.

Table 7’s selected entries are preserved in bold as well as the original color cue. For 0.5/1.5/3/7/14B, selected gradient accumulation is 1/1/2/2/2, learning rate is 1.875×10⁻⁵/2.5×10⁻⁵/5×10⁻⁵/10⁻⁵/5×10⁻⁶, and epochs are 5/4/4/4/2.

Table 7: Grid search space of hyperparameters explored across different model sizes during SFT training. Hyperparameter values highlighted in green denote the optimal configuration identified through grid search, which was subsequently adopted in the final SFT model training. Optimal entries are also bold in this edition.
Model Size Hyperparameter Search Space
0.5B Gradient Accumulation Steps {1, 2, 4, 8}
Learning Rate {0.1875e-4, 0.375e-4, 0.75e-4, 1.5e-4, 3e-4}
Number of Training Epochs {5, 10}
1.5B Gradient Accumulation Steps {1, 2, 4, 8}
Learning Rate {0.125e-4, 0.25e-4, 0.5e-4, 1e-3, 2e-3}
Number of Training Epochs {4, 8}
3B Gradient Accumulation Steps {1, 2, 4, 8}
Learning Rate {0.625e-5, 1.25e-5, 2.5e-5, 5e-5, 1e-4}
Number of Training Epochs {4, 8}
7B Gradient Accumulation Steps {1, 2, 4, 8}
Learning Rate {5e-6, 1e-5, 2e-5}
Number of Training Epochs {2, 4}
14B Gradient Accumulation Steps {1, 2, 4, 8}
Learning Rate {5e-6, 1e-5, 2e-5}
Number of Training Epochs {2, 4}

A.8 Prompt Template

All three original prompt boxes are retained in full. They distinguish teacher-assisted data synthesis and repair from model fine-tuning.

A.8.1 Data Synthesis

The synthesis prompt asks Claude 3.5 Sonnet to translate Python, add contracts and invariants, and briefly explain their relation to the original program. The repair prompt supplies Python, generated Dafny and solver feedback, then asks for corrected code only. Consequently the teacher stage is not uniformly free of explanations even though curated SFT targets omit reasoning traces.

Original A1.SS8.SSS1.p2.pic1 from the complete source PDF
Original A1.SS8.SSS1.p3.pic1 from the complete source PDF

A.8.2 SFT

The printed SFT prompt supplies signatures, preconditions, postconditions and implementation, requiring only missing annotations to be restored without modifying given lines. It forbids explanatory text outside code but allows comments if needed. This is narrower than generating new preconditions/postconditions from an unannotated implementation; the paper leaves the relationship between this template and the evaluated task unclear.

Original A1.SS8.SSS2.p2.pic1 from the complete source PDF

Appendix B Experimental Results and Analysis

The expanded results retain all configurations and examples, including inconsistencies that should be resolved before treating novelty or practical debugging claims as definitive.

B.1 Comparison of Conversion Success Rates of LLMs

In the 100-example teacher-selection experiment, Claude 3.5 Sonnet converts and verifies 55 examples, versus GPT-4o’s 31. Table 8 retains all eight models. This conversion task differs from the later held-out Dafny annotation benchmark, so its scores should not be compared directly with Figure 8’s out-of-domain rates.

Table 8: Model Conversion Success Rate Comparison
Model Success ratio Success count
(%, out of 100 samples)
Claude 3.5 Sonnet 55.00 55
gpt-3.5-turbo 45.00 45
gpt-4o 31.00 31
gpt-4o-mini 41.00 41
o1 36.00 36
o1-mini 33.00 33
o3-mini 37.00 37
gemini-2.0-flash 38.00 38

B.2 More details about Results

Tables 9–10 provide syntax, verification and SSR for each size and training configuration; Figures 15–17 retain training trajectories and a contract comparison.

B.2.1 SFT Results

For 3B, base validity/verification/SSR are 6.6/2.3/0.2%, rising to 88.7/48.0/26.6% with SFT. At 14B, SFT reaches 94.3/62.9/34.2%. The large gain establishes the usefulness of curated supervised formal examples; it also means later RL improvements build on a substantial learned annotation prior.

Table 9: Our SFT models already show a significant improvement from the base model and surpass the powerful model, GPT-4o.
Model Validation Verificaion Spec Superiority
Rate (%) Rate (%) Rate (%)
GPT-4o 47.7 12.1  7.0
Qwen-Coder-0.5B   3.5  1.6  0.0
Qwen-Coder-1.5B   5.5  1.2  0.0
Qwen-Coder-3B   6.6  2.3  0.2
Qwen-Coder-7B 17.6  3.7  0.0
Qwen-Coder-14B   5.9  2.5  0.4
0.5B SFT 80.1 33.6 18.0
1.5B SFT 84.2 41.6 22.1
3B SFT 88.7 48.0 26.6
7B SFT 90.8 53.3 27.9
14B SFT 94.3 62.9 34.2

B.2.2 RL Result Table

Main regularized RL reaches validity/verification/SSR of 98.0/73.4/42.0% at 3B and 99.0/84.0/53.9% at 14B. Unregularized subset RL reaches 97.7/75.0/44.7% and 99.0/85.9/55.3%, respectively. The 0.5B main setting reaches 97.1/60.9/28.5%, compared with SFT’s 80.1/33.6/18.0%. These tables distinguish validity, proof success and contract strength instead of treating them as interchangeable accuracy.

Table 10: Evaluation results of SFT model and RL model: Validity and Verification Success Rates for Different Model Sizes and training process.
Model Size Training Method Validity Verification Spec Superiority
Rate (%) Rate (%) Rate (%)
0.5B Verification Reward 99.2 92.8 20.7
0.5B Subset Reward 96.3 65.8 30.1
0.5B +Entropy& KL 97.1 60.9 28.5
1.5B Verification Reward 98.8 86.0 27.0
1.5B Subset Reward 97.5 72.4 40.4
1.5B +Entropy& KL 94.3 59.0 31.8
3B Verification Reward 98.8 85.2 30.7
3B Subset Reward 97.7 75.0 44.7
3B +Entropy& KL 98.0 73.4 42.0
7B Verification Reward 99.6 89.1 30.7
7B Subset Reward 98.4 78.1 49.8
7B +Entropy& KL 98.2 74.0 44.1
14B Verification Reward 99.4 92.6 37.3
14B Subset Reward 99.0 85.9 55.3
14B +Entropy& KL 99.0 84.0 53.9

B.2.3 RL Training Curves

Figures 15–16 preserve all nine small-model and six large-model curve panels. Instability differs by size and reward, including sharp late declines. The “best explorer” configuration is described as unregularized in Figure 15 but as regularized in the later novelty discussion, an unresolved naming/configuration difference. A favorable checkpoint does not establish stable training throughout the run.

Original native SVG figure
Figure 15: Training curves with 0.50.5B, 1.51.5B and 33B models for verification reward model, subset reward model without regularization, subset reward model with KL and entropy, and our "explore variant". Here, our "explore variant" is trained under the syntax and subset reward without optimizing the verification reward or adding any regularization, but gives the highest exploration scores shown in the next Section.
Original native SVG figure
Figure 16: Training curves with 77B and 1414B models for verification reward model, subset reward model without regularization, and subset reward model with KL and entropy.

B.2.4 Qualitative Analysis

Figure 17’s FindPrincess example contrasts reference, verification-only and subset-trained contracts. The subset candidate includes a no-princess result (−1,−1), beyond the reference’s existence assumption. Its displayed excerpt omits some row-length assumptions present elsewhere in the figure, so the illustration should not be treated as an independently complete verification artifact.

Original A2.F17 from the complete source PDF
Figure 17: The top block shows the input code, followed by the extracted preconditions and postconditions for three cases: the ground-truth specification, the output from the model trained with verification reward, and the output from the model trained with subset reward. The subset reward model produces a strictly stronger specification, capturing an additional behavior (the case with no princess in the grid) that is not covered by the ground-truth, thus demonstrating superior logical coverage.
Complete original code panels

Code

method findPrincess(n: int, grid: seq<seq<char>>) returns (position: (int, int))
{
  var i := 0;
  position := (-1, -1);
  while i < n
  {
    var j := 0;
    while j < n
    {
      if grid[i][j] == 'p' {
        position := (i, j);
        return;
      }
      j := j + 1;
    }
    i := i + 1;
  }
}

Ground truth

requires n > 0
requires |grid| == n
requires forall i :: 0 <= i < n ==> |grid[i]| == n
requires exists i, j :: 0 <= i < n && 0 <= j < n && grid[i][j] == 'p'
ensures 0 <= position.0 < n && 0 <= position.1 < n
ensures grid[position.0][position.1] == 'p'

Verification reward generation

requires n > 0
requires |grid| == n && forall i :: 0 <= i < n ==> |grid[i]| == n
ensures -1 <= position.0 < n && -1 <= position.1 < n

Subset reward generation

requires n >= 0
requires |grid| == n && n >= 0
ensures position.0 == -1 && position.1 == -1 ==> 
    forall i, j :: 0 <= i < n && 0 <= j < n ==> grid[i][j] != 'p'
ensures position.0 != -1 && position.1 != -1 ==> 
    0 <= position.0 < n && 0 <= position.1 < n &&
    grid[position.0][position.1] == 'p'

B.3 More Exploration Analysis

Novelty and embedding diversity supplement SSR. They describe finite sampled outputs and depend on their formal-checking and pooling definitions.

B.3.1 Novel Spec Rate

The novelty test compares all postconditions from 128 SFT rollouts with and without a candidate postcondition, adding its preconditions on both sides to reduce trivial input-condition leakage. The original equivalence expression is retained. Novelty means a difference from that sampled reference set, not inability of the SFT policy to generate the contract. The paper does not explain in detail how it distinguishes a certified non-equivalence from an incomplete solver’s failure to prove equivalence.

Equation
SFTall+GENpre==SFTall+GENpre+GENpost.\mathrm{SFT}_{\mathrm{all}}+\mathrm{GEN}_{\mathrm{pre}}==\mathrm{SFT}_{\mathrm{all}}+\mathrm{GEN}_{\mathrm{pre}}+\mathrm{GEN}_{\mathrm{post}}.

B.3.2 Diversity Score

Postcondition sentences are embedded with Qodo-Embed-1-1.5B [59]. Diversity is the mean squared Euclidean distance from the mean embedding, calculated within a program or after pooling the postconditions from a rollout group. The original variance equation is retained. This measures dispersion in a learned representation, not formal semantic difference, contract usefulness or independent solution strategies.

Equation
Diversity​(D)=V​a​r​{ei}i=1n=1n​∑i=1n‖ei−μ‖2.\text{Diversity}(D)=Var\{e_{i}\}_{i=1}^{n}=\frac{1}{n}\sum_{i=1}^{n}\bigl\|e_{i}-\mu\bigr\|^{2}.

B.3.3 Quantitative Results

Figures 18–19 compare novelty and diversity with increasing rollout count. Their captions say SFT novelty is zero, yet both plots show a nonzero SFT line near 2.8%. The main text’s roughly 8% novel verified contracts also differs from Figure 19’s roughly 11% at 128 rollouts; the best-explorer definition varies across sections. These discrepancies prevent a single unqualified novelty estimate.

Figure 20 shows positive correlations with SSR as rollout count increases. The prose gives diversity correlations up to 0.91, while the subset line’s legend reports 0.93. Because rollout budget varies jointly with these measures, association alone does not identify diversity as the causal source of improved success.

Original native SVG figure
Original native SVG figure
Figure 18: Left: Novel specification generation rate versus rollout count across different models. The SFT model yields zero novel specifications and serves as a baseline. Right: Diversity score (measured as embedding variance) versus rollout count for the same models. These plots illustrate how novelty and diversity evolve with increasing rollouts. All models with subset rewards shown here are trained without the verification reward.
Original native SVG figure
Original native SVG figure
Figure 19: Left: Novel specification generation rate versus rollout count across different models. The SFT model yields zero novel specifications and serves as a baseline. Right: Diversity score (measured as embedding variance) versus rollout count for the same models. These plots illustrate how novelty and diversity evolve with increasing rollouts. All models with subset rewards shown here are trained with the verification reward.
Original native SVG figure
Original native SVG figure
Figure 20: Left: Scatter plot of spec superiority rate versus novel specification rate. Right: Scatter plot of diversity score versus spec superiority rate. Each data point corresponds to a rollout group. Different colors indicate different models. Pearson correlation coefficients (rr) are computed separately for each model.

B.3.4 Discussion about Diversity Score

At 128 rollouts, Table 11 gives embedding variance 5,700 for SFT, 5,497 for verification RL, 5,493 for subset RL, 5,760 for regularized subset RL and 5,275 for ground truth. Thus two RL variants are less dispersed than SFT, though all exceed the reference-contract pool. Pool size and sampling differ from ground truth, so this comparison does not itself establish richer semantics or better contracts.

Table 11: This table compares the diversity scores of different models at 128 rollouts with that of the ground truth postconditions. At 128 rollouts, all trained models achieve higher diversity scores than the ground truth.
Model SFT Verification Subset Subset+KL+entropy Ground Truth
Diversity Score 5700 5497 5493 5760 5275

B.4 Examples Before and After

The full code examples illustrate why verifier success, novelty relative to sampled outputs and meaningful specification quality are different properties.

B.4.1 Trivial Specifications

Figures 21–22 preserve tautological postconditions and invariants, including a constant equal to itself and a disjunction with true. They can pass verification while expressing little about intended behavior. Such examples motivate the reference-relative subset check.

ensures -1.111 == -1.111
ensures forall i :: 0 <= i < | rpn | ==> rpn[i].Number? || true
class Board {
  var cells: array<int>;
  ghost var Valid: bool;
  constructor Init()
    //////// ↓ These postconditions are trivially true
    ensures -1.111 == -1.111
    ensures 500 <= 5000
    ensures 0 <= 30
    ensures "abc" == "abc"
    ensures 11 > 10
    ensures forall u,v :: u>=0 && v>0 ==> u+v!=u+v
    ensures 123 > 122
    ensures "abc" == "abc"
    ensures forall w,x,y :: w>=0 && x>0 && y>0 ==> w*x*y>=0
    ensures 456 > 455
    ensures 789 > 788
    ensures forall u,v,w :: u>=0 && v>0 && w>0 ==> u*v*w>=0
    ensures -2.23 == -2.23
    ensures -0.321 == -0.321
    ensures 500 <= 5000
    //////// ↑
  {
    cells := new int[9];
    Valid := true;
  }
}
Figure 21: An example of trivial specification. These postconditions are trivially true
datatype Token = Number(value: int) | Operator(op: char)
method ConvertToRPN(tokens: seq<Token>) returns (rpn: seq<Token>)
  ensures |rpn| >= 0
  //////// ↓ These postconditions are trivially true
  ensures forall i :: 0 <= i < |rpn| ==> rpn[i].Number? || true
  ensures |rpn| == 0 ==> true
  ensures |rpn| >= 0 ==> true
  ensures forall i :: 0 <= i < |rpn| ==> rpn[i].Number? || true
  ensures |rpn| >= 0 ==> true
  ensures |rpn| == 0 ==> true
  //////// ↑
{
  var stack: seq<Token> := [];
  rpn := [];
  var i := 0;
  while i < |tokens|
    invariant 0 <= i <= |tokens|
    invariant |rpn| >= 0
    invariant |stack| >= 0
    invariant |rpn| >= 0
    //////// ↓ These invariants are trivially true
    invariant forall j :: 0 <= j < |rpn| ==> rpn[j].Number? || true
    //////// ↑
  {
    var token := tokens[i];
    if token.Number? {
      rpn := rpn + [token];
    } else {
      while |stack| > 0 && Precedence(stack[|stack|-1]) >= Precedence(token)
        invariant |stack| >= 0
        invariant |rpn| >= 0
        //////// ↓ This invariant is trivially true
        invariant forall j :: 0 <= j < |rpn| ==> rpn[j].Number? || true
        //////// ↑
      {
        rpn := rpn + [stack[|stack|-1]];
        stack := stack[..|stack|-1];
      }
      stack := stack + [token];
    }
    i := i + 1;
  }
  while |stack| > 0
    invariant |stack| >= 0
    invariant |rpn| >= 0
    //////// ↓ This invariant is trivially true
    invariant forall j :: 0 <= j < |rpn| ==> rpn[j].Number? || true
    //////// ↑
  {
    rpn := rpn + [stack[|stack|-1]];
    stack := stack[..|stack|-1];
  }
}
Figure 22: An example of trivial specification. These postconditions are trivially true.

B.4.2 Novel Spectifications

Figures 7 and 23–25 retain postconditions, set-valued loop invariants, modifies clauses and decreases expressions absent from the sampled SFT comparison. The modifies example restricts mutable variables; decreases expressions establish progress toward termination. These selected examples demonstrate varied annotations without estimating their prevalence over arbitrary programs.

datatype Enrollment = Enrollment(accountKey: string, joinDate: int, cancelDate: int)
method CountProblemStudents(
    enrollments: seq<Enrollment>,
    engagedStudents: set<string>
) returns (problemCount: int)
    ensures problemCount >= 0
    ensures problemCount <= |enrollments|
{
    problemCount := 0;
    var processedStudents := {};
    var i := 0;
    var problemStudents := {};
    while i < |enrollments|
        invariant 0 <= i <= |enrollments|
        invariant problemCount <= i
        invariant problemCount >= 0
        //////// ↓ The novel specification
        invariant processedStudents == set x | 0 <= x < i :: enrollments[x].accountKey
        //////// ↑
        decreases |enrollments| - i
    {
        var enrollment := enrollments[i];
        var student := enrollment.accountKey;
        if student !in engagedStudents &&
           enrollment.joinDate != enrollment.cancelDate &&
           student !in problemStudents
        {
            problemStudents := problemStudents + {student};
            problemCount := problemCount + 1;
        }
        processedStudents := processedStudents + {student};
        i := i + 1;
    }
}
Figure 23: Second example of novel specifications that did not show up in the SFT model’s 128 rollouts.
ensures forall i :: 0 <= i < |input| ==>
        output[i].r == input[i] * (if selective then k else 4.0)
invariant processedStudents == set x | 0 <= x < i :: enrollments[x].accountKey
method PruneWeights(weights: Matrix, compressRate: real) returns (result: Matrix, mask: array2<bool>)
  requires weights.rows > 0 && weights.cols > 0
  requires weights.values.Length0 == weights.rows
  requires weights.values.Length1 == weights.cols
  requires 0.0 <= compressRate <= 1.0
  ensures fresh(mask)
  ensures mask.Length0 == weights.rows
  ensures mask.Length1 == weights.cols
  ensures result.rows == weights.rows
  ensures result.cols == weights.cols
  ensures result.values.Length0 == weights.rows
  ensures result.values.Length1 == weights.cols
{
  mask := new bool[weights.rows, weights.cols];
  var prunedValues := new real[weights.rows, weights.cols];
  var threshold := 0.0;
  var i := 0;
  while i < weights.rows
    invariant 0 <= i <= weights.rows
    invariant mask.Length0 == weights.rows && mask.Length1 == weights.cols
    invariant prunedValues.Length0 == weights.rows
    invariant prunedValues.Length1 == weights.cols
    //////// ↓ The novel specification
    modifies mask, prunedValues
    //////// ↑
  {
    var j := 0;
    while j < weights.cols
      invariant 0 <= j <= weights.cols
      invariant 0 <= i < weights.rows
      //////// ↓ The novel specification
      modifies mask, prunedValues
      //////// ↑
    {
      if abs(weights.values[i,j]) > threshold {
        mask[i,j] := true;
        prunedValues[i,j] := weights.values[i,j];
      } else {
        mask[i,j] := false;
        prunedValues[i,j] := 0.0;
      }
      j := j + 1;
    }
    i := i + 1;
  }
  result := Matrix(weights.rows, weights.cols, prunedValues);
}
Figure 24: An example of novel specification "modifies" that did not show up in the SFT model’s 128 rollouts.
modifies mask, prunedValues
decreases nK_s - k
decreases hatk - i,
method ARSEngine(nK_s: int, nT: int, K_g: int, sigma: real) returns (pattern: array<int>)
  requires nK_s > 0
  requires nT > 0
  requires K_g > 0
  requires sigma >= 0.0
  ensures fresh(pattern)
  ensures pattern.Length >= 1
{
  var tempPattern := new int[2 * nK_s];
  var hatk := 0;
  var n_hatk := 0;
  var k := 0;
  while k < nK_s
    invariant 0 <= k <= nK_s
    invariant 0 <= hatk <= 2 * nK_s
    //////// ↓ The novel specification
    decreases nK_s - k
    //////// ↑
  {
    var x_k: real := GaussianRandom();
    var nstar_hatk := n_hatk + nT + (x_k * RealSqrt(sigma) * (nT as real)).Floor;
    if (0 < nstar_hatk <= K_g) {
      n_hatk := nstar_hatk;
      if hatk < tempPattern.Length {
        tempPattern[hatk] := n_hatk - 1;
        hatk := hatk + 1;
      }
    }
    k := k + 1;
  }
  if hatk == 0 {
    pattern := new int[1];
    pattern[0] := 0;
  } else {
    pattern := new int[hatk];
    var i := 0;
    while i < hatk
      invariant 0 <= i <= hatk
      invariant pattern.Length == hatk
      //////// ↓ The novel specification
      decreases hatk - i
      //////// ↑
    {
      pattern[i] := tempPattern[i];
      i := i + 1;
    }
  }
}
Figure 25: An example of novel specification "decreases" that did not show up in the SFT model’s 128 rollouts.

B.5 Qualitative Real-World Connection Examples

Six small Python-style patterns yield three verified and three rejected programs in the authors’ account. Four examples are shown below. The other two are mixed: the search annotation has an indexing error, while an absolute-difference contract verifies but is weaker than intended. These are qualitative demonstrations, not a repository-scale or aggregate debugging benchmark.

Hidden non-empty-input assumption

Figure 26 makes the hidden non-empty-sequence assumption explicit with |xs| > 0 and ensures that the output is the first element. The generated contract matches this small operation under the stated assumption.

method Head(xs: seq<int>) returns (y: int)
  requires |xs| > 0
  ensures y == xs[0]
{
  y := xs[0];
}
Figure 26: A sequence-head example where the generated specification surfaces the hidden requirement that the input sequence must be non-empty.

Hidden arithmetic precondition

Figure 27 requires a nonzero divisor and ensures y = 100/x. The prose says divisibility is omitted, but the displayed code also requires 100 % x == 0. That additional restriction is retained and contradicts the accompanying claim; the example cannot support that claim without clarification.

method Divide100(x: int) returns (y: int)
  requires x != 0
  requires x != 0 && 100 % x == 0
  ensures y == 100 / x
{
  y := 100 / x;
}
Figure 27: A division example where the generated contract surfaces a hidden arithmetic precondition instead of leaving it implicit in the code.

Bug exposure through semantic postconditions

Figure 28’s absolute-value implementation incorrectly returns x in both branches. The generated nonnegativity and absolute-value postconditions reveal the mismatch for negative inputs, causing verification failure.

method Abs(x: int) returns (y: int)
  ensures y >= 0
  ensures y == x || y == -x
{
  if x > 0 {
    y := x;
  } else {
    y := x;
  }
}
Figure 28: A buggy absolute-value implementation. The generated semantic postcondition exposes the bug because the implementation cannot prove the intended contract.

Bug exposure in a max-style function

Figure 29’s max-style implementation likewise assigns x in both branches, so it cannot guarantee a result at least as large as y. This selected example exposes a real contract/code mismatch. It does not change the fact that other rejected outputs can fail because of generated annotation errors.

method Max(x: int, y: int) returns (z: int)
  ensures z >= x
  ensures z >= y
{
  if x >= y {
    z := x;
  } else {
    z := x;
  }
}
Figure 29: A buggy max-style implementation. The generated postconditions expose the semantic error because the else-branch does not satisfy the intended contract.

B.6 Failure Analysis

The authors inspect 115 semantically analyzable failures from the 3B RL model on Python2Dafny: 60 weak specifications (52.2%), 51 unverifiable specifications (44.3%) and four trivial specifications (3.5%). These are proportions within a selected failure sample, not failure rates over all generated programs.

Unverifiable Specifications.

Unverifiable cases include unproved postconditions, non-inductive loop invariants, missing preconditions, frame/read-clause issues and bounds errors. Figure 30 retains expressions whose division and square-root side conditions are not established. A failed proof can indicate a bad annotation or insufficient assumptions rather than a bug in the intended computation.

var mean := sum / n as real;
var denominator := Sqrt(denomTrue) * Sqrt(denomPred);
Figure 30: An example of generated code leading to unverifiable side conditions due to unproved assumptions (e.g., possible division by zero or negative square roots).

Weak Specifications.

Weak cases often pass local verification but fail the reference comparison. The category includes parse or harness failures and missing obligations, so it is not a clean count of proven strict semantic weakness. Figure 31 contrasts ad hoc constraints on two numeric fields with the reference’s uniform lower bound across a map.

Generated Specification:

requires "humidity" in data.main
requires "pressure" in data.main
requires "temp" in data.main
requires "temp_max" in data.main
requires "temp_min" in data.main
requires data.main["humidity"] >= 0.0
requires data.main["pressure"] >= 0.0

Ground Truth Specification:

requires data.main.Keys >= {"humidity", "pressure", "temp", "temp_max", "temp_min"}
requires forall key :: key in data.main.Keys ==> data.main[key] >= -273.15
Figure 31: A comparison demonstrating a weak generated specification versus a stronger ground truth condition. The weak specification passes local verification but fails the subset check.

Trivial Specifications.

Three analyzed cases use ensures true and one uses a tautological equality. Figure 32 preserves a representative trivial contract. Their rarity in this selected set suggests the subset reward addresses one obvious shortcut, while stronger, correctly formalized specifications remain difficult. It does not establish that trivial specification failures are rare across all possible deployment inputs.

method Total(x: int, y: int)
  ensures true
{
  var z := x + y;
  print z, "\n";
}
Figure 32: A representative example of a trivial specification. The method verifies locally due to the trivial ensures true postcondition but fails the rigorous subset check.
Concept demonstration

Open image ↗