Overview
This edition follows the complete 52-page v4 source, published in TMLR in May 2026 and revised in July 2026. Its title uses “Reducing Human Annotations,” replacing “Human Priors” in the original catalog title. The author list is unchanged. All 32 numbered figures, 11 tables, three algorithms, three prompt boxes, equations and 95 references are retained; code panels also keep their complete downloadable text.
At a glance
Re:Form trains language models to annotate Dafny programs with machine-checkable specifications. Supervised fine-tuning teaches the output format, then reinforcement learning rewards syntax, verification and logical comparison with reference contracts. A subset reward discourages easy but uninformative specifications. Models improve in-domain verification and reference-relative specification strength, with smaller absolute gains on synthetic out-of-domain compositions. The study supports automated specification training, while leaving specification completeness, faithful translation and transfer to large software systems unresolved.
1 Introduction
Natural-language reasoning can be hard to evaluate reliably. Dafny offers a more constrained setting: specifications state input assumptions, output guarantees and loop invariants, and a verifier checks whether code satisfies them [38]. A program can nevertheless verify against a weak contract. Re:Form therefore separates syntax validity, verifier success and the strength of the generated contract relative to a reference.
The pipeline reduces additional manual annotations during reinforcement learning, but still relies on pretrained code models, teacher-generated labels, curated code, supervised training, human-designed rewards and a trusted formal toolchain. Absence of an explicit natural-language reasoning trace is an experimental choice, not proof that reasoning traces or human knowledge are unnecessary in general.
2 Pipeline
The pipeline curates verified training programs, constructs a compositional evaluation benchmark, performs supervised fine-tuning and then optimizes outcome-based formal rewards. Figures 1–3 and Algorithms 1–3 preserve the complete design and data flow.
2.1 Data Curation
Native Dafny sources are combined with Python programs translated by a language model. Claude 3.5 Sonnet is chosen after a 100-example comparison; verification feedback supports up to ten repair attempts. Functions and specifications are generated in stages, and only successful verified programs are retained. Heuristic filtering removes unwanted annotations and comments. This automates curation but conditions the dataset on teacher and verifier success.
The paper describes roughly 20,000 programs. Table 1 instead lists 0.9 thousand MetaReflection, 0.3 thousand BigCode and 16.3 thousand Python2Dafny examples, totaling about 17,500. This discrepancy is unresolved. Passing a Dafny verifier establishes the generated Dafny program’s supplied contract, not equivalence to the original Python program or completeness of its intended behavior.
| Data Source | N# | N#Spec | N#Token |
|---|---|---|---|
| MetaReflection | 0.9 k | 6.53 | 318.57 |
| BigCode | 0.3 k | 24.5 | 766.13 |
| Python2Dafny | 16.3k | 16.94 | 601.71 |
2.2 Benchmark
DafnyComp adds specification-strength evaluation to a setting where verifier pass rate can be inflated by weak contracts. It separates held-out examples from the curation distribution from newly assembled Python-function compositions. DafnyBench supplies an additional prior benchmark [48]. “Out of domain” here means this specific synthetic composition procedure, not unrestricted industrial software.
2.2.1 Program Assembly Stage
The assembly stage selects single-input, single-output LeetCode functions with McCabe complexity above five [87]. A frontier model constructs call-graph templates, combines functions and supplies structural variations. Formatting, import completion, input/output matching and existing tests filter the resulting Python programs. These checks make composition practical but do not prove the full generated program correct.
2.2.2 Formal Translation Stage
Translation follows the program’s abstract syntax structure: individual nodes are generated and verified before combination. The system repairs initial code and then annotates main and subsidiary functions, retaining only successfully verified pairs. Algorithm 3 preserves both stages. Selection for successful formalization can exclude harder programs and does not establish Python-to-Dafny semantic equivalence.
2.3 Pattern Activation through Supervised Fine-tuning
Given implementation c, the model emits the entire program together with specifications, written c ⊕ y = π(c), rather than only an annotation fragment. Supervised fine-tuning is intended to teach Dafny syntax and basic semantic patterns before reinforcement learning. The curated training outputs exclude explicit natural-language chains of thought and comments. Appendix A.8’s printed SFT prompt fixes existing preconditions and postconditions and asks for missing invariants/assertions, a narrower task than the main full-specification description; the paper does not reconcile this difference.
2.4 Exploration with Reinforcement Learning
The first reward scheme checks syntax/type validity and Dafny verification. The second adds a subset comparison: reference preconditions must imply generated preconditions, and, under the reference preconditions, generated postconditions must imply reference postconditions. Both original implications and the GRPO objective/normalized group advantage are retained below [67]. The approach draws on reference-comparison ideas in Clover and Verina [72][89].
Equality satisfies these implications: “spec superiority rate” includes contracts that are merely as strong as the reference. The postcondition comparison is restricted to the reference input domain. It therefore does not prove globally weakest preconditions, globally strongest postconditions or strictly superior behavior. Because reference contracts determine reward, they still provide supervision even without a person labeling each new rollout.
GRPO compares sampled outcomes within each group. Experiments contrast verification rewards, additional subset rewards, and subset rewards with KL and entropy regularization. Verification is relative to the supplied contract and trusted language/solver implementation [15]. Incompleteness can reject valid statements; a failed implication check alone does not prove strict semantic weakness.
Equations 1–2
| (1) |
| (2) |
3 Results and Analysis
Results distinguish single-sample success from whether any of many samples succeeds. The detailed appendix tables and all learning curves are retained, including outcomes that qualify the headline claims.
3.1 Experiment Setup
The study varies model size, reward and regularization within a Qwen-based pipeline, using held-out in-domain programs, synthetic compositions and a DafnyBench sample.
Models
Models use pretrained Qwen 2.5 checkpoints from 0.5 to 14 billion parameters [25]. The 0.5B starting model is distilled from larger models, so its data and teacher costs differ from straightforward fine-tuning. The authors report no obvious benefit from 32B in preliminary exploration, without a detailed controlled scaling result.
Dataset
The main text specifies 3,000 SFT examples, 4,500 separate RL examples, 512 held-out in-domain programs, 300 synthetic DafnyComp compositions and 100 sampled DafnyBench programs. Appendix A.7 instead states 5,000 SFT examples. Both counts are preserved as an unresolved reporting discrepancy. The held-out split avoids reuse during these training stages but does not audit every possible pretrained overlap.
Training Details
RL uses four rollouts per input, temperature 1.0, batch size 1,024 and learning rate 10⁻⁵. The main setting adds KL coefficient 0.01 and entropy coefficient 0.02. On A800 GPUs, the 3B run takes about 20 hours on 32 GPUs for 40 epochs. Table 2 reports all sizes; its largest run takes about 36 hours on 64 GPUs. These figures exclude teacher synthesis, distillation, SFT searches and other experiments.
Evaluation Metrics
Validity measures syntax/type success; verification measures passing the generated contract; SSR measures generated specifications that meet the reference-relative subset comparison. Pass@1 uses a single sample. The plotted best-of-k success counts an input as solved when at least one of k rollouts attains the reward, so higher sampling budgets entail additional inference and verification cost.
3.2 Main Experiment Results
Fine-tuning improves formal output substantially, and RL generally increases both verification and reference-relative specification strength. Reward choice changes their balance.
Absence of CoTs
The authors find comments in roughly 2% of RL outputs, usually after completed code, and do not observe extended explanatory reasoning. Figure 4 retains an example. This describes the trained output distribution; it does not reveal all internal computation or compare against a matched reasoning-trace training treatment.
// This program prints Hello World!
// println!("Hello World!");
Improvment from SFT
For 3B, SFT reaches 48.0% verification and 26.6% SSR; the main regularized RL setting reaches 73.4% and 42.0%. For 14B, SFT reaches 62.9% and 34.2%, while main RL reaches 84.0% and 53.9%. The detailed tables also show that unregularized subset RL has higher pass@1 SSR, 44.7% at 3B and 55.3% at 14B. Regularization is selected primarily for exploration under multiple rollouts, not uniform single-sample superiority.
Exploration Analysis
Figure 6 shows higher best-of-128 success with entropy-regularized subset training than SFT. The authors also compare generated postconditions with the aggregate of 128 SFT samples and report specifications outside that finite reference set. This provides evidence of different discovered contracts, not proof that the SFT model could never generate them or that they exceed all human knowledge. Figures 7 and 23–25 preserve examples.
method ApplyFading(input: seq<real>, selective: bool) returns (output: seq<Complex>)
ensures |output| == |input|
//////// ↓ The novel specification
ensures forall i :: 0 <= i < |input| ==> output[i].r == input[i] * (if selective then k else 4.0)
//////// ↑
{
var result: seq<Complex> := [];
var i := 0;
while i < |input|
invariant 0 <= i <= |input|
invariant |result| == i
//////// ↓ The novel specification (found in RL and SFT results, but not in ground truth)
invariant forall j :: 0 <= j < i ==> result[j].r == input[j] * (if selective then k else 4.0)
//////// ↑
{
var fadeValue := if selective then k else 4.0;
var complex := new Complex(input[i] * fadeValue, 0.0);
result := result + [complex];
i := i + 1;
}
output := result;
}
OOD-generalization
On the synthetic out-of-domain set, 14B main RL reaches 14.0% verification versus 8.3% for SFT and 0.3% for GPT-4o. SSR rises from 4.7% to 7.7%: a 3.0 percentage-point increase, approximately 63.8% relative. Most problems still fail. Figure 8’s caption also identifies GPT-4o as best on DafnyBench, so the broader claim of dominance across every dataset should not be read literally.
New compositions reduce simple exact-example reuse, but constituent code and model pretraining are not fully audited. Low base-model scores do not rule out contamination or related-code familiarity [18][63].
Summary
The strongest supported result is improved formal annotation under the reported tasks and budgets. Its scope is narrower than a universal claim that RL discovers reasoning beyond pretraining: the reference-dependent metric, teacher-curated data, finite SFT comparison and low absolute compositional success remain material qualifications.
3.3 Ablation Study
The ablations expose reward hacking and an exploration/stability trade-off. Both are visible in the retained learning curves.
Comparison between Reward Schemes
Verification-only optimization raises the pass rate while allowing weaker contracts. At 3B, it reaches 85.2% verification but 30.7% SSR, compared with 73.4% and 42.0% for the main subset setting. The subset reward sacrifices some verifier success to improve the reference-relative contract measure. Trivial contracts are one failure mode, alongside unverifiable and insufficiently informative specifications.
Effects of Regularization
Entropy regularization improves best-of-128 exploration but introduces unstable training and collapses in some curves. KL alone and unregularized training can have strong pass@1 yet poor multi-sample coverage. Checkpoint selection therefore matters; the paper does not fully specify an equalized selection policy across configurations.
Trailing tokens after complete code rise from about 1% of SFT outputs to about 80% with entropy RL. This differs from the 2% comment statistic. Tokens generated after the answer cannot guide earlier autoregressive tokens, and their presence shows that entropy can reward extra output without useful exploration.
4 Conclusion and Discussion
Formal outcomes provide an effective training signal, and comparing contracts addresses a weakness of verifier-only rewards. The study does not establish arbitrary-program correctness: the verifier checks a formal specification, whose usefulness, completeness and correspondence to intended behavior remain separate concerns. Teacher quality, selection effects, solver limits and task/prompt inconsistencies constrain the interpretation.
Connection to practical software settings
Six small translated Python-style examples connect the method to hidden assumptions and simple bugs. The authors describe three verified and three rejected outputs, but Appendix B.5 says one search failure is a generated annotation indexing error and one verified absolute-difference example has a weak postcondition. These outcomes cannot all be counted as successful bug detection. Repository-scale changes, realistic integration and broader debugging utility are not evaluated.
5 Related Work
The work sits between code generation, formal verification and reinforcement learning for reasoning, with an emphasis on automatically evaluating specifications rather than natural-language explanations.
5.1 LLMs in Software Engineering
Earlier systems learn program generation from code and execution feedback [6][36]. Dafny synthesis, DafnyBench and annotation tools apply language models to verification [48][52][58]. Clover, Verina and other verified-generation benchmarks address relationships among code, specification and intent [72][89]. Re:Form contributes a curated training pipeline and reward study; it does not replace the need to decide what behavior a contract should express.
5.2 Informal vs. Formal Reasoning in LLMs
Natural-language chains of thought can aid problem solving but need not faithfully expose internal reasoning [35][8]. Formal proving offers checkable conclusions within its formal system [88]. Re:Form evaluates code-only outputs in one such environment. This does not settle the general value of explanations, transfer from formal tasks, or whether formal annotations capture a user’s actual requirements.
Broader Impact Statement
The authors propose formal verification as a component of more reliable AI and speculate about formalized oversight or debate [14][27]. Those applications are not tested. Any practical assurance would still depend on the specification, trusted tools and how the formal model maps to the deployed system.
Appendix
The appendix preserves language background, exact examples, training/search settings, complete prompts, model tables, learning curves, novelty/diversity measures and failure analysis.
Appendix A Techinical Details and Methods
Technical details make clear which annotations are inferred, which are supplied, and how generated contracts are compared.
A.1 Brief Introduction to Dafny
Dafny combines executable code with specifications checked using automated reasoning [38]. Table 3 contrasts Python’s dynamic execution with Dafny’s static types, contracts and formal verification. A precondition describes assumed inputs, a postcondition the guaranteed result, and loop invariants properties maintained during iteration. This distinction separates passing a verifier from faithfully formalizing a Python program’s intent.
| Feature | Dafny | Python |
|---|---|---|
| Year Introduced | 2010 (Microsoft Research) | 1991 (Guido van Rossum) |
| Type System | Static typing, compile-time checks | Dynamic typing, run-time checks |
| Formal Verification | Yes — built-in contracts and proofs | No — only basic assert |
| Main Use | Verified algorithms, critical systems | General-purpose programming |
| Execution Model | Compiled with verification | Interpreted (e.g., CPython) |
A.2 Notation List
Table 4 retains the notation for source programs, generated code, preconditions, postconditions, reference specifications and model rewards. “Ground truth” denotes the curated reference contract; it is not a guarantee that every relevant behavioral requirement is represented.
| Symbol | Description |
|---|---|
| Policy | The LLM Model or Policy |
| Code implementation | The raw code body without specifications |
| Spec/Specification | A formal description of what a program is supposed to do, acting as a contract between the program and its clients to guide verification |
| Dafny verifier | An automatic theorem prover to check the consistency of the specifications with the code |
| Precondition | A condition that must be true before running a piece of code, and thus sets the admissible input domain |
| Postcondition | A condition that must be true after running a piece of code and guarantees the output ranges |
| requires | A precondition in Dafny |
| ensures | A postcondion in Dafny |
| invariant | A condition that holds true during loop iterations |
| Clause | One line specification, such as |
| ensures |nearbyStops| <= |stops| | |
| GT | The ground truth specifications generated by Claude |
| The intersection of preconditions in the ground truth | |
| The intersection of generated preconditions | |
| The intersection of postconditions in the ground truth | |
| The intersection of generated postconditions | |
| Syntax reward | A reward assigned based on whether the generated specifications pass compilation |
| Verification reward | A reward assigned based on whether the generated specifications are consistent with the given code, which can be checked by the Dafny verifier |
| Subset relation | For formal statements and , if , then is a subset of , denoted as |
| Superior specifications | A set of specifications with weaker preconditions and stronger postconditions |
| Subset reward | A reward assigned based on whether the generated specifications are superior to or at least as strong as the ground truth |
| Validation Rate | Percentage of generated programs without syntax error |
| Verification rate | Percentage of generated specifications that are verified to be consistent with the code by Dafny |
| Spec Superiority Rate | Percentage of generated specifications superior to or at least as strong as the corresponding ground truth |
| Novel Specification | A non-trivial postcondition unseen in any of the SFT rollouts |
A.3 An Example of Specification and Implementation
Figures 11–12 show the same summation implementation with and without annotations. The annotated version assumes n ≥ −1, ensures s = n(n+1)/2, and maintains s = i(i−1)/2 with 0 ≤ i ≤ n+1. The complete original code is retained, including the distinction between implementation and specification comments.
requires n >= -1 ensures s == n * (n + 1) / 2
invariant s == i * (i - 1) / 2 invariant 0 <= i <= n + 1
method Sum(n: int) returns (s: int)
requires n >= -1 // Specification
ensures s == n * (n + 1) / 2 // Specification
{
var i := 0; // Implementation
s := 0; // Implementation
while i <= n // Implementation
invariant s == i * (i - 1) / 2 // Specification
invariant 0 <= i <= n + 1 // Specification
{
s := s + i; // Implementation
i := i + 1; // Implementation
}
}
method Sum(n: int) returns (s: int)
{
var i := 0; // Implementation
s := 0; // Implementation
while i <= n // Implementation
{
s := s + i; // Implementation
i := i + 1; // Implementation
}
}
A.4 An Example of the Data Curation
Figure 13 preserves the Python permutation example and generated Dafny counterpart. Complete code panels remain available beneath the original diagram, including long lines clipped at the PDF panel edges. The displayed contract preserves array contents after permutation; it does not completely characterize printed output or independently establish faithful translation.
Complete original code panels
Python code
def perm_3(k):
if k == N:
print(arr[0], arr[1], arr[2])
else:
for i in range(k, N):
arr[i], arr[k] = arr[k], arr[i]
perm_3(k+1)
arr[i], arr[k] = arr[k], arr[i]
arr = [1, 2, 3, 4, 5, 6, 7]
N = len(arr)
perm_3(0)Generated Dafny code
method Perm3(arr: array<int>, k: int)
requires arr.Length >= 3
requires 0 <= k <= arr.Length
modifies arr
ensures arr[..] == old(arr[..])
decreases arr.Length - k
{
if k == arr.Length {
Print(arr[0], arr[1], arr[2]);
} else {
var i := k;
while i < arr.Length
invariant k <= i <= arr.Length
invariant arr[..] == old(arr[..])
invariant forall j, l :: k <= j < i && k <= l < i && j != l ==>
arr[j] == old(arr[j]) && arr[l] == old(arr[l])
{
Swap(arr, k, i);
Perm3(arr, k + 1);
Swap(arr, k, i);
i := i + 1;
}
}
}
method Swap(arr: array<int>, i: int, j: int)
requires 0 <= i < arr.Length && 0 <= j < arr.Length
modifies arr
ensures arr[i] == old(arr[j]) && arr[j] == old(arr[i])
ensures forall k :: 0 <= k < arr.Length && k != i && k != j ==>
arr[k] == old(arr[k])
{
var temp := arr[i];
arr[i] := arr[j];
arr[j] := temp;
}
method Print(a: int, b: int, c: int)
{
print a, " ", b, " ", c, "\n";
}
method Main()
{
var arr := new int[7];
arr[0] := 1; arr[1] := 2; arr[2] := 3; arr[3] := 4;
arr[4] := 5; arr[5] := 6; arr[6] := 7;
Perm3(arr, 0);
}
A.5 An Example of the Subset Reward
Figure 14 illustrates the two implication checks. Its reference requires n > 0 and k > n and guarantees k_out ≥ 0. The candidate admits n ≥ 0 and k ≥ 0 and states k_out = k − n. The comparison checks relaxation and conditional strengthening, rather than equating reference-relative success with every possible desirable specification.
Complete original code panels
Code
method main(n: int, k: int) returns (k_out: int)
{
k_out := k;
var j: int := 0;
while(j < n)
{
j := j + 1;
k_out := k_out - 1;
}
}Ground truth
requires n > 0 requires k > n ensures k_out >= 0
Generated specifications
requires n >= 0 requires k >= 0 ensures k_out == k - n
Comparison clause
// Check whether generated specifications have weaker preconditions assert ( (n >= 0) && (k >= 0) ) <== ( (n > 0) && (k > n) ); /* Check under the precondition of the ground truth, whether generated postconditions are stronger */ assert ( (n > 0) && (k > n) ) ==>((k_out == k - n) ==> k_out >= 0);
A.6 Distillation Details of the 0.5B Model
The 0.5B model is distilled from fine-tuned 7B or 14B teachers. Table 5 varies supervised versus sequence-level distillation, forward KL, reverse KL or Jensen–Shannon divergence, temperature one or two, and a base versus fine-tuned student. Table 6 lists four preferred configurations without their scores or a fully explicit mapping to every subsequent 0.5B result. These resources should be included when comparing small-model efficiency.
| Category | Options |
|---|---|
| Distillation Algorithm | SKD, SeqKD |
| KL Loss | KLD, RKL, JSD () |
| Temperature | , |
| Student Model | SFTed 0.5B, Base 0.5B |
| Teacher Model | SFTed 7B, SFTed 14B |
A.7 SFT Training Hyperparameter Grid Search Details
SFT uses one eight-A800 server, ZeRO stage 3, cosine learning-rate decay and 10% warm-up. Per-GPU batches decrease with model size. The appendix states 5,000 examples and under 40 minutes per model, differing from the main text’s 3,000 examples; the total hyperparameter-search cost is not reported.
Table 7’s selected entries are preserved in bold as well as the original color cue. For 0.5/1.5/3/7/14B, selected gradient accumulation is 1/1/2/2/2, learning rate is 1.875×10⁻⁵/2.5×10⁻⁵/5×10⁻⁵/10⁻⁵/5×10⁻⁶, and epochs are 5/4/4/4/2.
| Model Size | Hyperparameter | Search Space |
|---|---|---|
| 0.5B | Gradient Accumulation Steps | {1, 2, 4, 8} |
| Learning Rate | {0.1875e-4, 0.375e-4, 0.75e-4, 1.5e-4, 3e-4} | |
| Number of Training Epochs | {5, 10} | |
| 1.5B | Gradient Accumulation Steps | {1, 2, 4, 8} |
| Learning Rate | {0.125e-4, 0.25e-4, 0.5e-4, 1e-3, 2e-3} | |
| Number of Training Epochs | {4, 8} | |
| 3B | Gradient Accumulation Steps | {1, 2, 4, 8} |
| Learning Rate | {0.625e-5, 1.25e-5, 2.5e-5, 5e-5, 1e-4} | |
| Number of Training Epochs | {4, 8} | |
| 7B | Gradient Accumulation Steps | {1, 2, 4, 8} |
| Learning Rate | {5e-6, 1e-5, 2e-5} | |
| Number of Training Epochs | {2, 4} | |
| 14B | Gradient Accumulation Steps | {1, 2, 4, 8} |
| Learning Rate | {5e-6, 1e-5, 2e-5} | |
| Number of Training Epochs | {2, 4} |
A.8 Prompt Template
All three original prompt boxes are retained in full. They distinguish teacher-assisted data synthesis and repair from model fine-tuning.
A.8.1 Data Synthesis
The synthesis prompt asks Claude 3.5 Sonnet to translate Python, add contracts and invariants, and briefly explain their relation to the original program. The repair prompt supplies Python, generated Dafny and solver feedback, then asks for corrected code only. Consequently the teacher stage is not uniformly free of explanations even though curated SFT targets omit reasoning traces.
A.8.2 SFT
The printed SFT prompt supplies signatures, preconditions, postconditions and implementation, requiring only missing annotations to be restored without modifying given lines. It forbids explanatory text outside code but allows comments if needed. This is narrower than generating new preconditions/postconditions from an unannotated implementation; the paper leaves the relationship between this template and the evaluated task unclear.
Appendix B Experimental Results and Analysis
The expanded results retain all configurations and examples, including inconsistencies that should be resolved before treating novelty or practical debugging claims as definitive.
B.1 Comparison of Conversion Success Rates of LLMs
In the 100-example teacher-selection experiment, Claude 3.5 Sonnet converts and verifies 55 examples, versus GPT-4o’s 31. Table 8 retains all eight models. This conversion task differs from the later held-out Dafny annotation benchmark, so its scores should not be compared directly with Figure 8’s out-of-domain rates.
| Model | Success ratio | Success count |
|---|---|---|
| (%, out of 100 samples) | ||
| Claude 3.5 Sonnet | 55.00 | 55 |
| gpt-3.5-turbo | 45.00 | 45 |
| gpt-4o | 31.00 | 31 |
| gpt-4o-mini | 41.00 | 41 |
| o1 | 36.00 | 36 |
| o1-mini | 33.00 | 33 |
| o3-mini | 37.00 | 37 |
| gemini-2.0-flash | 38.00 | 38 |
B.2 More details about Results
Tables 9–10 provide syntax, verification and SSR for each size and training configuration; Figures 15–17 retain training trajectories and a contract comparison.
B.2.1 SFT Results
For 3B, base validity/verification/SSR are 6.6/2.3/0.2%, rising to 88.7/48.0/26.6% with SFT. At 14B, SFT reaches 94.3/62.9/34.2%. The large gain establishes the usefulness of curated supervised formal examples; it also means later RL improvements build on a substantial learned annotation prior.
| Model | Validation | Verificaion | Spec Superiority |
|---|---|---|---|
| Rate (%) | Rate (%) | Rate (%) | |
| GPT-4o | 47.7 | 12.1 | 7.0 |
| Qwen-Coder-0.5B | 3.5 | 1.6 | 0.0 |
| Qwen-Coder-1.5B | 5.5 | 1.2 | 0.0 |
| Qwen-Coder-3B | 6.6 | 2.3 | 0.2 |
| Qwen-Coder-7B | 17.6 | 3.7 | 0.0 |
| Qwen-Coder-14B | 5.9 | 2.5 | 0.4 |
| 0.5B SFT | 80.1 | 33.6 | 18.0 |
| 1.5B SFT | 84.2 | 41.6 | 22.1 |
| 3B SFT | 88.7 | 48.0 | 26.6 |
| 7B SFT | 90.8 | 53.3 | 27.9 |
| 14B SFT | 94.3 | 62.9 | 34.2 |
B.2.2 RL Result Table
Main regularized RL reaches validity/verification/SSR of 98.0/73.4/42.0% at 3B and 99.0/84.0/53.9% at 14B. Unregularized subset RL reaches 97.7/75.0/44.7% and 99.0/85.9/55.3%, respectively. The 0.5B main setting reaches 97.1/60.9/28.5%, compared with SFT’s 80.1/33.6/18.0%. These tables distinguish validity, proof success and contract strength instead of treating them as interchangeable accuracy.
| Model Size | Training Method | Validity | Verification | Spec Superiority |
|---|---|---|---|---|
| Rate (%) | Rate (%) | Rate (%) | ||
| 0.5B | Verification Reward | 99.2 | 92.8 | 20.7 |
| 0.5B | Subset Reward | 96.3 | 65.8 | 30.1 |
| 0.5B | +Entropy& KL | 97.1 | 60.9 | 28.5 |
| 1.5B | Verification Reward | 98.8 | 86.0 | 27.0 |
| 1.5B | Subset Reward | 97.5 | 72.4 | 40.4 |
| 1.5B | +Entropy& KL | 94.3 | 59.0 | 31.8 |
| 3B | Verification Reward | 98.8 | 85.2 | 30.7 |
| 3B | Subset Reward | 97.7 | 75.0 | 44.7 |
| 3B | +Entropy& KL | 98.0 | 73.4 | 42.0 |
| 7B | Verification Reward | 99.6 | 89.1 | 30.7 |
| 7B | Subset Reward | 98.4 | 78.1 | 49.8 |
| 7B | +Entropy& KL | 98.2 | 74.0 | 44.1 |
| 14B | Verification Reward | 99.4 | 92.6 | 37.3 |
| 14B | Subset Reward | 99.0 | 85.9 | 55.3 |
| 14B | +Entropy& KL | 99.0 | 84.0 | 53.9 |
B.2.3 RL Training Curves
Figures 15–16 preserve all nine small-model and six large-model curve panels. Instability differs by size and reward, including sharp late declines. The “best explorer” configuration is described as unregularized in Figure 15 but as regularized in the later novelty discussion, an unresolved naming/configuration difference. A favorable checkpoint does not establish stable training throughout the run.
B.2.4 Qualitative Analysis
Figure 17’s FindPrincess example contrasts reference, verification-only and subset-trained contracts. The subset candidate includes a no-princess result (−1,−1), beyond the reference’s existence assumption. Its displayed excerpt omits some row-length assumptions present elsewhere in the figure, so the illustration should not be treated as an independently complete verification artifact.
Complete original code panels
Code
method findPrincess(n: int, grid: seq<seq<char>>) returns (position: (int, int))
{
var i := 0;
position := (-1, -1);
while i < n
{
var j := 0;
while j < n
{
if grid[i][j] == 'p' {
position := (i, j);
return;
}
j := j + 1;
}
i := i + 1;
}
}Ground truth
requires n > 0 requires |grid| == n requires forall i :: 0 <= i < n ==> |grid[i]| == n requires exists i, j :: 0 <= i < n && 0 <= j < n && grid[i][j] == 'p' ensures 0 <= position.0 < n && 0 <= position.1 < n ensures grid[position.0][position.1] == 'p'
Verification reward generation
requires n > 0 requires |grid| == n && forall i :: 0 <= i < n ==> |grid[i]| == n ensures -1 <= position.0 < n && -1 <= position.1 < n
Subset reward generation
requires n >= 0
requires |grid| == n && n >= 0
ensures position.0 == -1 && position.1 == -1 ==>
forall i, j :: 0 <= i < n && 0 <= j < n ==> grid[i][j] != 'p'
ensures position.0 != -1 && position.1 != -1 ==>
0 <= position.0 < n && 0 <= position.1 < n &&
grid[position.0][position.1] == 'p'B.3 More Exploration Analysis
Novelty and embedding diversity supplement SSR. They describe finite sampled outputs and depend on their formal-checking and pooling definitions.
B.3.1 Novel Spec Rate
The novelty test compares all postconditions from 128 SFT rollouts with and without a candidate postcondition, adding its preconditions on both sides to reduce trivial input-condition leakage. The original equivalence expression is retained. Novelty means a difference from that sampled reference set, not inability of the SFT policy to generate the contract. The paper does not explain in detail how it distinguishes a certified non-equivalence from an incomplete solver’s failure to prove equivalence.
Equation
B.3.2 Diversity Score
Postcondition sentences are embedded with Qodo-Embed-1-1.5B [59]. Diversity is the mean squared Euclidean distance from the mean embedding, calculated within a program or after pooling the postconditions from a rollout group. The original variance equation is retained. This measures dispersion in a learned representation, not formal semantic difference, contract usefulness or independent solution strategies.
Equation
B.3.3 Quantitative Results
Figures 18–19 compare novelty and diversity with increasing rollout count. Their captions say SFT novelty is zero, yet both plots show a nonzero SFT line near 2.8%. The main text’s roughly 8% novel verified contracts also differs from Figure 19’s roughly 11% at 128 rollouts; the best-explorer definition varies across sections. These discrepancies prevent a single unqualified novelty estimate.
Figure 20 shows positive correlations with SSR as rollout count increases. The prose gives diversity correlations up to 0.91, while the subset line’s legend reports 0.93. Because rollout budget varies jointly with these measures, association alone does not identify diversity as the causal source of improved success.
B.3.4 Discussion about Diversity Score
At 128 rollouts, Table 11 gives embedding variance 5,700 for SFT, 5,497 for verification RL, 5,493 for subset RL, 5,760 for regularized subset RL and 5,275 for ground truth. Thus two RL variants are less dispersed than SFT, though all exceed the reference-contract pool. Pool size and sampling differ from ground truth, so this comparison does not itself establish richer semantics or better contracts.
| Model | SFT | Verification | Subset | Subset+KL+entropy | Ground Truth |
|---|---|---|---|---|---|
| Diversity Score | 5700 | 5497 | 5493 | 5760 | 5275 |
B.4 Examples Before and After
The full code examples illustrate why verifier success, novelty relative to sampled outputs and meaningful specification quality are different properties.
B.4.1 Trivial Specifications
Figures 21–22 preserve tautological postconditions and invariants, including a constant equal to itself and a disjunction with true. They can pass verification while expressing little about intended behavior. Such examples motivate the reference-relative subset check.
ensures -1.111 == -1.111
ensures forall i :: 0 <= i < | rpn | ==> rpn[i].Number? || true
class Board {
var cells: array<int>;
ghost var Valid: bool;
constructor Init()
//////// ↓ These postconditions are trivially true
ensures -1.111 == -1.111
ensures 500 <= 5000
ensures 0 <= 30
ensures "abc" == "abc"
ensures 11 > 10
ensures forall u,v :: u>=0 && v>0 ==> u+v!=u+v
ensures 123 > 122
ensures "abc" == "abc"
ensures forall w,x,y :: w>=0 && x>0 && y>0 ==> w*x*y>=0
ensures 456 > 455
ensures 789 > 788
ensures forall u,v,w :: u>=0 && v>0 && w>0 ==> u*v*w>=0
ensures -2.23 == -2.23
ensures -0.321 == -0.321
ensures 500 <= 5000
//////// ↑
{
cells := new int[9];
Valid := true;
}
}
datatype Token = Number(value: int) | Operator(op: char)
method ConvertToRPN(tokens: seq<Token>) returns (rpn: seq<Token>)
ensures |rpn| >= 0
//////// ↓ These postconditions are trivially true
ensures forall i :: 0 <= i < |rpn| ==> rpn[i].Number? || true
ensures |rpn| == 0 ==> true
ensures |rpn| >= 0 ==> true
ensures forall i :: 0 <= i < |rpn| ==> rpn[i].Number? || true
ensures |rpn| >= 0 ==> true
ensures |rpn| == 0 ==> true
//////// ↑
{
var stack: seq<Token> := [];
rpn := [];
var i := 0;
while i < |tokens|
invariant 0 <= i <= |tokens|
invariant |rpn| >= 0
invariant |stack| >= 0
invariant |rpn| >= 0
//////// ↓ These invariants are trivially true
invariant forall j :: 0 <= j < |rpn| ==> rpn[j].Number? || true
//////// ↑
{
var token := tokens[i];
if token.Number? {
rpn := rpn + [token];
} else {
while |stack| > 0 && Precedence(stack[|stack|-1]) >= Precedence(token)
invariant |stack| >= 0
invariant |rpn| >= 0
//////// ↓ This invariant is trivially true
invariant forall j :: 0 <= j < |rpn| ==> rpn[j].Number? || true
//////// ↑
{
rpn := rpn + [stack[|stack|-1]];
stack := stack[..|stack|-1];
}
stack := stack + [token];
}
i := i + 1;
}
while |stack| > 0
invariant |stack| >= 0
invariant |rpn| >= 0
//////// ↓ This invariant is trivially true
invariant forall j :: 0 <= j < |rpn| ==> rpn[j].Number? || true
//////// ↑
{
rpn := rpn + [stack[|stack|-1]];
stack := stack[..|stack|-1];
}
}
B.4.2 Novel Spectifications
Figures 7 and 23–25 retain postconditions, set-valued loop invariants, modifies clauses and decreases expressions absent from the sampled SFT comparison. The modifies example restricts mutable variables; decreases expressions establish progress toward termination. These selected examples demonstrate varied annotations without estimating their prevalence over arbitrary programs.
datatype Enrollment = Enrollment(accountKey: string, joinDate: int, cancelDate: int)
method CountProblemStudents(
enrollments: seq<Enrollment>,
engagedStudents: set<string>
) returns (problemCount: int)
ensures problemCount >= 0
ensures problemCount <= |enrollments|
{
problemCount := 0;
var processedStudents := {};
var i := 0;
var problemStudents := {};
while i < |enrollments|
invariant 0 <= i <= |enrollments|
invariant problemCount <= i
invariant problemCount >= 0
//////// ↓ The novel specification
invariant processedStudents == set x | 0 <= x < i :: enrollments[x].accountKey
//////// ↑
decreases |enrollments| - i
{
var enrollment := enrollments[i];
var student := enrollment.accountKey;
if student !in engagedStudents &&
enrollment.joinDate != enrollment.cancelDate &&
student !in problemStudents
{
problemStudents := problemStudents + {student};
problemCount := problemCount + 1;
}
processedStudents := processedStudents + {student};
i := i + 1;
}
}
ensures forall i :: 0 <= i < |input| ==>
output[i].r == input[i] * (if selective then k else 4.0)
invariant processedStudents == set x | 0 <= x < i :: enrollments[x].accountKey
method PruneWeights(weights: Matrix, compressRate: real) returns (result: Matrix, mask: array2<bool>)
requires weights.rows > 0 && weights.cols > 0
requires weights.values.Length0 == weights.rows
requires weights.values.Length1 == weights.cols
requires 0.0 <= compressRate <= 1.0
ensures fresh(mask)
ensures mask.Length0 == weights.rows
ensures mask.Length1 == weights.cols
ensures result.rows == weights.rows
ensures result.cols == weights.cols
ensures result.values.Length0 == weights.rows
ensures result.values.Length1 == weights.cols
{
mask := new bool[weights.rows, weights.cols];
var prunedValues := new real[weights.rows, weights.cols];
var threshold := 0.0;
var i := 0;
while i < weights.rows
invariant 0 <= i <= weights.rows
invariant mask.Length0 == weights.rows && mask.Length1 == weights.cols
invariant prunedValues.Length0 == weights.rows
invariant prunedValues.Length1 == weights.cols
//////// ↓ The novel specification
modifies mask, prunedValues
//////// ↑
{
var j := 0;
while j < weights.cols
invariant 0 <= j <= weights.cols
invariant 0 <= i < weights.rows
//////// ↓ The novel specification
modifies mask, prunedValues
//////// ↑
{
if abs(weights.values[i,j]) > threshold {
mask[i,j] := true;
prunedValues[i,j] := weights.values[i,j];
} else {
mask[i,j] := false;
prunedValues[i,j] := 0.0;
}
j := j + 1;
}
i := i + 1;
}
result := Matrix(weights.rows, weights.cols, prunedValues);
}
modifies mask, prunedValues
decreases nK_s - k decreases hatk - i,
method ARSEngine(nK_s: int, nT: int, K_g: int, sigma: real) returns (pattern: array<int>)
requires nK_s > 0
requires nT > 0
requires K_g > 0
requires sigma >= 0.0
ensures fresh(pattern)
ensures pattern.Length >= 1
{
var tempPattern := new int[2 * nK_s];
var hatk := 0;
var n_hatk := 0;
var k := 0;
while k < nK_s
invariant 0 <= k <= nK_s
invariant 0 <= hatk <= 2 * nK_s
//////// ↓ The novel specification
decreases nK_s - k
//////// ↑
{
var x_k: real := GaussianRandom();
var nstar_hatk := n_hatk + nT + (x_k * RealSqrt(sigma) * (nT as real)).Floor;
if (0 < nstar_hatk <= K_g) {
n_hatk := nstar_hatk;
if hatk < tempPattern.Length {
tempPattern[hatk] := n_hatk - 1;
hatk := hatk + 1;
}
}
k := k + 1;
}
if hatk == 0 {
pattern := new int[1];
pattern[0] := 0;
} else {
pattern := new int[hatk];
var i := 0;
while i < hatk
invariant 0 <= i <= hatk
invariant pattern.Length == hatk
//////// ↓ The novel specification
decreases hatk - i
//////// ↑
{
pattern[i] := tempPattern[i];
i := i + 1;
}
}
}
B.5 Qualitative Real-World Connection Examples
Six small Python-style patterns yield three verified and three rejected programs in the authors’ account. Four examples are shown below. The other two are mixed: the search annotation has an indexing error, while an absolute-difference contract verifies but is weaker than intended. These are qualitative demonstrations, not a repository-scale or aggregate debugging benchmark.
Hidden non-empty-input assumption
Figure 26 makes the hidden non-empty-sequence assumption explicit with |xs| > 0 and ensures that the output is the first element. The generated contract matches this small operation under the stated assumption.
method Head(xs: seq<int>) returns (y: int)
requires |xs| > 0
ensures y == xs[0]
{
y := xs[0];
}
Hidden arithmetic precondition
Figure 27 requires a nonzero divisor and ensures y = 100/x. The prose says divisibility is omitted, but the displayed code also requires 100 % x == 0. That additional restriction is retained and contradicts the accompanying claim; the example cannot support that claim without clarification.
method Divide100(x: int) returns (y: int)
requires x != 0
requires x != 0 && 100 % x == 0
ensures y == 100 / x
{
y := 100 / x;
}
Bug exposure through semantic postconditions
Figure 28’s absolute-value implementation incorrectly returns x in both branches. The generated nonnegativity and absolute-value postconditions reveal the mismatch for negative inputs, causing verification failure.
method Abs(x: int) returns (y: int)
ensures y >= 0
ensures y == x || y == -x
{
if x > 0 {
y := x;
} else {
y := x;
}
}
Bug exposure in a max-style function
Figure 29’s max-style implementation likewise assigns x in both branches, so it cannot guarantee a result at least as large as y. This selected example exposes a real contract/code mismatch. It does not change the fact that other rejected outputs can fail because of generated annotation errors.
method Max(x: int, y: int) returns (z: int)
ensures z >= x
ensures z >= y
{
if x >= y {
z := x;
} else {
z := x;
}
}
B.6 Failure Analysis
The authors inspect 115 semantically analyzable failures from the 3B RL model on Python2Dafny: 60 weak specifications (52.2%), 51 unverifiable specifications (44.3%) and four trivial specifications (3.5%). These are proportions within a selected failure sample, not failure rates over all generated programs.
Unverifiable Specifications.
Unverifiable cases include unproved postconditions, non-inductive loop invariants, missing preconditions, frame/read-clause issues and bounds errors. Figure 30 retains expressions whose division and square-root side conditions are not established. A failed proof can indicate a bad annotation or insufficient assumptions rather than a bug in the intended computation.
var mean := sum / n as real; var denominator := Sqrt(denomTrue) * Sqrt(denomPred);
Weak Specifications.
Weak cases often pass local verification but fail the reference comparison. The category includes parse or harness failures and missing obligations, so it is not a clean count of proven strict semantic weakness. Figure 31 contrasts ad hoc constraints on two numeric fields with the reference’s uniform lower bound across a map.
Generated Specification:
requires "humidity" in data.main requires "pressure" in data.main requires "temp" in data.main requires "temp_max" in data.main requires "temp_min" in data.main requires data.main["humidity"] >= 0.0 requires data.main["pressure"] >= 0.0
Ground Truth Specification:
requires data.main.Keys >= {"humidity", "pressure", "temp", "temp_max", "temp_min"}
requires forall key :: key in data.main.Keys ==> data.main[key] >= -273.15Trivial Specifications.
Three analyzed cases use ensures true and one uses a tautological equality. Figure 32 preserves a representative trivial contract. Their rarity in this selected set suggests the subset reward addresses one obvious shortcut, while stronger, correctly formalized specifications remain difficult. It does not establish that trivial specification failures are rare across all possible deployment inputs.
method Total(x: int, y: int)
ensures true
{
var z := x + y;
print z, "\n";
}