At a glance
Braingle Brainteaser evaluates how language models solve narrative math and logic puzzles, alongside whether their answers are correct. The study probes solution strategies, hints, narrative-to-math rewriting, correction using an official solution, and decomposition into meaningful steps.
Models can produce useful insights, but also enumerate possibilities when shorter solutions exist and accept misleading correction instructions. These findings concern generated explanations under the paper’s operational definitions of creativity and brute force. They do not establish how a model computes internally. The authors release code and the dataset.
1 Introduction
Accuracy alone cannot distinguish a compact mathematical insight from an exhaustive search that reaches the same answer. Narrative brainteasers often admit both approaches while demanding relatively little specialized knowledge. The preserved Six Village puzzle illustrates how substantial constraints can be embedded in a story.
The benchmark compares o3, DeepSeek models and Gemini Flash across several aspects of problem solving. Its contribution is a set of complementary behavioral probes, rather than a single creativity score or proof that an explanation faithfully reveals internal reasoning.
Problem: Six Village (from the Braingle Brainteaser Math subset)
There are six villages along the coast of the only perfectly round island in the known universe. The villages are evenly distributed along the coastline so that the distance between any two neighboring coastal villages is always the same. There is an absolutely straight path through the jungle connecting every pair of villages. These paths create thirteen crossings in the interior of the island, one of which is in the middle of the island where paths from every village meet. The island has a strange courtship custom. Before a father will give permission for his daughter to marry, her suitor must bring the father a fish each day until he has traveled by every route from his village to the father’s village. The young man only travels along routes where he is always getting closer to his destination. The young man may visit other villages along the way.
On April 1st a father’s three sons come to tell him of their intent to woo a bride, each from a different village. The brides’ villages are the first three villages encountered when traveling clockwise around the island. If the sons begin their courtship today and the couples are married on the day following each son’s last trip, what are the three wedding dates?
2 Related Works
The work connects reasoning benchmarks, evaluation beyond final answers, and creative mathematical problem solving. Its focus is the structure and usefulness of natural-language solutions to knowledge-light puzzles.
LLM Reasoning Benchmarks
Unlike knowledge-intensive Olympiad problems, these puzzles emphasize deductions and patterns accessible with basic math and logic. They complement datasets such as FOLIO and earlier BRAINTEASER work. A benchmark’s remaining difficulty does not itself establish freedom from training-data contamination; Appendix A acknowledges that risk.
Reasoning Evaluation Beyond Final Answer Accuracy
Earlier work evaluates intermediate reasoning steps directly. This study instead probes four related abilities: using hints, translating narrative statements, interpreting a provided solution, and correcting an answer with external guidance. Success on one probe need not imply success on another.
Reasoning Creativity
Programmatic conjecture search and formal refinement, including FunSearch, address creativity through symbolic results. Here, creativity means an insight in a generated explanation that reduces the search or simplifies the problem. The distinction is operational and judgment-dependent.
3 The Braingle Brainteaser Benchmark
The benchmark has Math and Logic subsets, accompanied by official solutions, hints, user difficulty/popularity ratings and manually assigned problem categories.
3.1 Dataset Construction
The authors scrape Braingle with permission to distribute the selected examples. They select the 250 highest-difficulty puzzles in each category, then math-club annotators remove low-quality or ambiguous items and supply missing hints. The resulting category inventory contains 242 Math and 236 Logic puzzles.
Several later tables still label the subsets n = 250. The source does not fully reconcile those counts with the cleaned inventory, so denominators should not be inferred solely from the construction description.
3.2 Detailed Dataset Information
Difficulty and popularity are user ratings on a 1–4 scale. Problem categories describe structure, while model-derived step counts describe a solution’s macro-level decomposition. Neither rating nor step count is a direct measurement of computational difficulty.
Population Statistics
Table 1 totals 242 Math and 236 Logic puzzles, with mean difficulty 2.80 and 2.66. Table S1 reports mean answer lengths of 172 words for Math and 237 for Logic; the surrounding main-text sentence reverses those numbers while claiming Logic answers are longer. The original tables are retained.
| Category | Count | Popularity | Difficulty |
|---|---|---|---|
| Standard | |||
| Geometry | 24 | 2.19 0.18 | 2.87 0.19 |
| Number Theory | 34 | 2.35 0.22 | 2.91 0.15 |
| Combinatorics | 24 | 2.27 0.11 | 2.76 0.10 |
| Algebra | 56 | 2.37 0.22 | 2.75 0.12 |
| Nonstandard | |||
| Logic | 29 | 2.35 0.23 | 2.77 0.14 |
| Special Number | 29 | 2.35 0.22 | 2.78 0.13 |
| Heuristic | |||
| Pattern | 28 | 2.28 0.22 | 2.82 0.15 |
| Arithmetic | 18 | 2.44 0.24 | 2.80 0.19 |
| Total | 242 | 2.33 0.22 | 2.80 0.15 |
| Category | Count | Popularity | Difficulty |
|---|---|---|---|
| Simple/large | |||
| 0D | 29 | 2.53 0.23 | 2.68 0.23 |
| 1D | 13 | 2.56 0.30 | 2.60 0.28 |
| 2D | 22 | 2.47 0.30 | 2.72 0.25 |
| Number | 17 | 2.43 0.27 | 2.74 0.26 |
| Clusters | 8 | 2.56 0.20 | 2.78 0.13 |
| Tree | 6 | 2.83 0.17 | 2.69 0.15 |
| Complex/small | |||
| Liars | 17 | 2.50 0.28 | 2.66 0.22 |
| Communication | 4 | 2.70 0.28 | 2.55 0.15 |
| Compound | 9 | 2.48 0.44 | 2.67 0.34 |
| Math-like | |||
| Algorithm | 38 | 2.56 0.34 | 2.66 0.23 |
| Math | 32 | 2.41 0.27 | 2.63 0.22 |
| Heuristic | |||
| Pattern | 26 | 2.48 0.27 | 2.63 0.22 |
| Linguistic | 15 | 2.50 0.12 | 2.61 0.15 |
| Total | 236 | 2.51 0.28 | 2.66 0.23 |
Problem Categorization
Math has eight subcategories under standard competition-style, nonstandard, and heuristic problems. Logic has 13 subcategories grouped as large/simple, small/complex, math-like, and heuristic. These groupings separate the size of an arrangement space from the logical complexity of its clues.
Distribution of Number of Steps
o3 divides official solutions into meaningful components and labels each creative or rudimentary. Manual inspection covers 30 examples, rather than an independent human annotation of every step. Table 2 gives 6.4 mean steps for Math and 8.6 for Logic, including 2.0 and 2.4 creative steps respectively.
A step can combine many calculations. Counts therefore conceal substantial differences in discovery effort and computation; they measure the chosen explanation granularity, not equal units of difficulty.
| Solution Steps | Median | Creative Steps | Rudimentary Steps | |
|---|---|---|---|---|
| Braingle Math | ||||
| Braingle Logic |
3.3 Problem Example
Math 4 asks for a shared property of three 12-digit numbers. The supplied solution finds perfect-square sums at several digit-group sizes. Such pattern questions admit arbitrary formal descriptions, so the intended answer relies on a human judgment of a reasonable, salient pattern rather than mathematical uniqueness.
Problem: Math 4
What characteristic do these three 12-digit numbers share with each other, but with no other 12-digit number?
Equation
Human Solution: Math 4 (abbreviated)
The sum of their digits are square numbers:
Equation
The sum of their digit pairs are square numbers:
Equation
Similarly, the sum of their digit triplets, quadruplets, and sextuplets, as well as the numbers themselves, are all square numbers.
4 Experiments and Analyses
Experiments first compare solution accuracy, then examine brute-force use, susceptibility to supplied corrections, narrative rewriting, and solution decomposition. Quantitative measures rely primarily on o3 judgments, with a limited human calibration study.
4.1 Solution Correctness
Correctness is evaluated against the human solution under four prompt settings. The results describe particular model/API versions and single-run settings, rather than a stable ranking of model families.
Evaluation Approach
Three expert annotators judge 100 o3-generated solutions. o3 agrees with human labels at 99.3% for correctness and 97% for creativity versus brute force on average. This is raw agreement on a sampled set, not a confidence interval or proof of unbiased grading across all models and puzzle types.
Prompting Approaches
The chain-of-thought prompt requests explicit steps. The Math prompt additionally forbids unjustified steps and unnecessary brute force. Each has a hint variant using source or annotator-written hints. Full prompt wording remains in Appendix F.
Model Performance
Table 3 places o3 highest among the listed models. On Math it reports 79.6% with CoT, 82.8% with hints and 81.2% with Math plus hints; on Logic the corresponding values are 68.4%, 70.0% and 74.4%. The hardest 50 cases remain more difficult.
The prose’s 88.8%/80.5% Math example does not match Table 3, whose Math-plus-hint row is 81.2%/68.0%. Hints often help, but several rows worsen, particularly for smaller models. Some hard queries return no answer; the suggested explanation that they exceed reasoning ability is speculative.
| Dataset | Model | CoT Prompt | Math Prompt | w Hint | Math Prompt w Hint |
|---|---|---|---|---|---|
| Math | DeepSeek R1 Distill Qwen 1.5B | 17.2 (14.0) | 16.4 (10.0) | 15.2 (8.0) | 17.6 (10.0) |
| DeepSeek R1 Distill Qwen 14B | 41.2 (22.0) | 44.0 (30.0) | 44.0 (20.0) | 42.6 (26.0) | |
| DeepSeek R1 Distill Llama 70B | 42.4 (20.0) | 40.8 (22.0) | 45.6 (24.0) | 44.2 (18.0) | |
| deepseek-chat (Deepseek-V3) | 58.0 (46.0) | 55.6 (38.0) | 56.0 (36.0) | 58.8 (36.0) | |
| deepseek-reasoner (Deepseek-R1) | 66.8 (48.0) | 70.2 (54.0) | 72.4 (48.0) | 72.8 (56.0) | |
| gemini-2.5-flash-preview-04-17 | 66.0 (34.0) | 65.2 (38.0) | 69.2 (44.0) | 72.0 (58.0) | |
| OpenAI o3 | 79.6 (66.0) | 79.6 (64.0) | 82.8 (66.0) | 81.2 (68.0) | |
| Logic | DeepSeek R1 Distill Qwen 1.5B | 4.0 (4.0) | 4.0 (6.0) | 6.8 (6.0) | 3.6 (4.1) |
| DeepSeek R1 Distill Qwen 14B | 22.0 (16.0) | 23.6 (16.0) | 27.2 (22.0) | 26.0 (32.0) | |
| DeepSeek R1 Distill Llama 70B | 24.4 (16.0) | 24.4 (14.0) | 26.0 (20.0) | 29.2 (26.0) | |
| deepseek-chat (Deepseek-V3) | 37.8 (30.6) | 40.8 (28.0) | 41.6 (22.0) | 41.4 (24.5) | |
| deepseek-reasoner (Deepseek-R1) | 44.6 (26.0) | 45.4 (32) | 49.4 (32.7) | 50.6 (40.0) | |
| gemini-2.5-flash-preview-04-17 | 49.2 (36.0) | 51.2 (34.0) | 54.0 (42.0) | 53.6 (38.0) | |
| OpenAI o3 | 68.4 (50.0) | 71.2 (54.0) | 70.0 (52.0) | 74.4 (54.0) |
4.2 Brute-force vs. Creativity
The study distinguishes exhaustive or repeated guess-and-check from insight-driven reductions. This distinguishes styles of expressed solutions; the validity or efficiency of a particular algorithm can depend on the problem and available tools.
Brute-force Solution
Brute force is defined as systematically trying possibilities until an answer is found. Code, guess-and-check and unusually extensive computation are classifier cues. Their presence is not a universal proof that a solution lacks reasoning, and a small search can be a reasonable method.
Creative Solution
Creative solutions exploit constraints, patterns, analogies or reframings that reduce the work needed. The label emphasizes useful insight, not novelty relative to everything in a model’s training data. Figure 1 compares model and official-solution strategy labels.
Evaluation
A few-shot o3 judge labels solutions across seven models, two subsets and four prompts. Strong reasoning models generally use less brute force, including fewer cases where the official solution avoids it. For o3 Math, the model-only brute-force cell falls from 9.5% with CoT to 3.7% with Math plus hints.
The prompt effect is not universal: Tables S6–S7 contain smaller-model reversals. Nonempty responses supply the strategy analysis, so its percentages need not share the accuracy table’s denominator. These results are associations under the chosen judge and definitions.
Brute-force vs Adequate/Inadequate Summary
Table S12 compares brute-force use when a model can or cannot adequately summarize the official solution. Some conditions show more brute force with inadequate summaries, but others reverse or tie. This is suggestive correlation, not evidence of a universal shared barrier or a causal mechanism.
4.3 Informed self-correction based on the human solution
The correction study uses 14 incorrect o3 CoT solutions found among the 50 hardest cases. Every tested model receives the same flawed answer and official solution, then the labels are swapped in an adversarial condition. This is externally informed correction, not unaided self-correction.
Error Analysis
With the correct reference, o3, DeepSeek V3 and DeepSeek R1 produce valid corrections in over 80% of the sampled cases. Smaller distilled models can repeat themselves or identify the wrong error. When the references are swapped, models often accept the false premise; the authors report false confession in more than 60% of cases.
The small, selected error set limits generalization. Appendix L also illustrates ambiguity in judging alternative valid answers, so reference labels should not be treated as infallible mathematical ground truth.
4.4 Translating from verbal narratives into mathematical-style problem statements
Thirty difficult rewritable Math puzzles are converted from narratives to competition-style statements. The experiment asks whether removing linguistic presentation difficulty changes solving performance while preserving the mathematical task.
Rewritability
Non-rewritable cases include open-ended patterns, already formal statements and descriptions that cannot be made more concise. o3 produces equivalent rewrites for 24 of the chosen 30; humans repair missing constraints in the other six before evaluation. The final test therefore uses human-verified rewrites.
Evaluation
On these 30 problems, o3 improves from 56.7% to 73.3% and DeepSeek R1 from 50.0% to 63.3%, with reported p-values 0.023 and 0.043. Distilled Llama 70B improves from 23.3% to 33.3% (p = 0.08). The paper does not specify the test in this passage.
These selected-set results support a presentation effect, while the conclusion’s broader statement that formalization does not reliably help is more cautious. Rewriting alone neither eliminates reasoning failures nor establishes improvement for all puzzle types.
4.5 Automatic breakdown of solutions
o3 compresses official and model solutions into macro-steps, even when it cannot independently solve the puzzle. On the 30 hardest Math cases, retained o3 responses average 7.7 steps, DeepSeek V3 12.4 and distilled Llama 70B 10.7. Rudimentary steps outnumber creative steps in each row.
Incomplete responses and excessive-casework outliers are excluded, giving sample sizes 29, 26 and 21. The source inconsistently names DeepSeek V3, V1 and R3 across text and tables. Step decomposition can preserve erroneous reasoning; it should not be interpreted as verifying that reasoning.
5 Representative Case Study and Solution Error Patterns
Selected response excerpts illustrate inefficient hint use and missing justification. These case studies explain possible failure modes but do not estimate their prevalence across the full benchmark.
5.1 Inefficient Use of Hints
A six-digit puzzle’s rotational-sum clue reduces to 111111 times the digit sum. Using it early constrains the last two digits and sharply reduces the candidate space. In the illustrated o3 response, that clue is checked only after much enumeration.
The authors hypothesize that the model delays the textually longest clue because it looks complex. The example demonstrates a missed simplification; it does not establish textual length as the cause. Prioritizing clues by eliminative power is proposed as a remedy.
| Hint | o3 solution excerpt |
| “Take the sum of the number, the number rotated one to the left, the number rotated one to the right, the number with the first three and last three digits swapped, the number with the digit pairs rotated to the left, and the number with the digit pairs rotated to the right. The first and last digits of this sum match the last two digits of the number, in some order.” | 416723, 167234 (rotate L), 341672 (rotate R), 723416 (swap), 672341 (rotate pairs L), 234167 (rotate pairs R). The sum is 2555553, first digit is 2, last digit is 3, exactly the two digits e=2 and f=3 of the original number. The condition is satisfied. |
5.2 Recurring Error Patterns in OpenAI o3 Solutions and Prompt - based Remedies
Three recurring patterns are unsupported appeals to known results, incomplete uniqueness checks, and forcing an answer despite uncertainty. Targeted instructions encourage more explicit justification, but the paper does not show that they consistently produce correct solutions.
6 Conclusion
The benchmark separates abilities that final accuracy conflates: deriving an answer, recognizing a useful hint, interpreting a solution, and resisting a misleading reference. Models show both insight and inefficient search, with performance varying by category and prompt.
The evidence is bounded by a single public puzzle source, operational creativity labels, model-based judging and largely single-run experiments. Claims about general creativity or faithful internal reasoning go beyond these measurements.
NeurIPS Paper Checklist
The checklist reports about $10,000 in API credits, experiments limited to one run per setting, and local execution on two 80 GB A100 GPUs. Most results lack run-to-run error bars. The authors state that source owners permitted distribution and that code/data will be available.
The checklist marks LLM usage as not applicable, although o3 grading and step annotation are central methods described in the paper. The edition follows the substantive methodology and records this source inconsistency. Human work is described as expert annotation, with no compensation details supplied here.
Appendix A Limitations
The benchmark’s single website, math/logic focus and public availability limit external validity and raise contamination concerns. Different-looking generated answers are only anecdotal evidence against memorization. o3 judging can introduce bias, and macro-step counts hide detailed computation.
Prompt effects vary, correction is vulnerable to false premises, and formalization is insufficient for hard cases. Creativity and brute-force conclusions depend on the paper’s definitions and tested models; they should not be generalized to all reasoning tasks.
Appendix B LLM-as-a-Judge
The authors justify model-based judging using prior LLM-as-a-judge research and their human agreement study. Calibration supports its use in the sampled setting but does not remove shared-model, selection or definition biases.
B.1 LLM-Human Agreement Experiments
Three native-English college annotators with mathematics/statistics backgrounds, over five years of contest training and top-500 Putnam honors label 100 o3 solutions. They receive the same definitions and examples as the automated judge. Average raw agreement is 99.3% for correctness and 97% for strategy.
The authors report better agreement than several alternative judges without a detailed comparison table here. Agreement on o3 outputs alone does not independently validate all other model outputs or every problem category.
Appendix C Broader Impact
The proposed benefit is evaluation that rewards efficient and interpretable problem solving in education, discovery and collaboration. These are prospective applications; the paper does not test deployment outcomes or demonstrate that visible explanations guarantee transparency.
Appendix D Full Dataset Statistics
Table S1 retains full difficulty and answer-length statistics and reports hints for every item. Its n = 250 labels differ from the cleaned counts in Table 1. Mean human-answer length is 172 words for Math and 237 for Logic, with large variation.
| Statistics Data | Braingle Math | Braingle Logic |
|---|---|---|
| Most difficult () | Most difficult () | |
| Percentage with hints | 100% | 100% |
| Difficulty score out of 4 | ||
| median | 2.79 | 2.64 |
| Answer word count | ||
| median | 109.5 | 162.5 |
| Answer sentence count | ||
| median | 4.00 | 8.00 |
Appendix E API Inference Settings
All listed API configurations cap outputs at 10,000 tokens. Distilled DeepSeek models use top-p and temperature 0.7; the remaining temperatures are one, with top-p one except Gemini’s 0.95 and the unspecified R1 entry. These settings constrain the interpretation of claims about unbounded thinking time.
| Model | Topp | Max Tokens | Temperature |
| DeepSeek R1 Distill Qwen 1.5B | 0.7 | 10000 | 0.7 |
| DeepSeek R1 Distill Qwen 14B | 0.7 | 10000 | 0.7 |
| DeepSeek R1 Distill Llama 70B | 0.7 | 10000 | 0.7 |
| deepseek-chat (Deepseek-V3) | 1 | 10000 | 1 |
| deepseek-reasoner (Deepseek-R1) | - | 10000 | 1 |
| gemini-2.5-flash-preview-04-17 | 0.95 | 10000 | 1 |
| OpenAI o3 | 1 | 10000 | 1 |
Appendix F Prompts
The three complete templates retain CoT instructions, the stricter Math constraints and the hint request. The Math template penalizes unnecessary code or search, unsupported steps, altered problem statements and incomplete answers. The main experiment combines these instructions with problem-specific hints.
Prompt: Chain of Thought (CoT)
You are an excellent mathematician with perfect logic. You are also very patient, and willing to perform very long chains of reasoning when necessary.
Can you solve this problem? Please spell out your entire reasoning steps. Finish your response in the format of “Final answer: ” immediately followed by your answer.
Prompt: Math Prompt
You are an excellent mathematician with perfect logic. You are also very patient, and willing to perform very long chains of reasoning when necessary. Solve the given problem, keeping in mind the following:
If you use a brute force or guess-and-check method or utilize code when not necessary, you will receive no credit. If you do not fully justify a step, you will receive no credit.
Ensure each of your statements is consistent with the conditions of the problem and statements you have already written before moving on. If you do not do this, you will receive no credit.
An outline of a solution without a concrete final answer will also receive no credit.
The problem statement is correct, and a correct answer exists. If you solve a version of the problem or attempt to modify the problem statement to a version other than the one written exactly as-is, you will receive no credit.
Finish your response in the format of “Final answer: ” immediately followed by your answer.
Prompt: Hint
You are an excellent mathematician with perfect logic. You are also very patient, and willing to perform very long chains of reasoning when necessary.
Can you solve this problem using the hint? Please spell out your entire reasoning steps. Output the final answer in the format of “Final answer: ” at the end of your answer.
Appendix G Counting Steps
The counting prompt defines a step as a meaningful solution component, not every arithmetic operation. Creative labels require a simplifying insight; rudimentary labels cover application, routine calculation and enumeration. Every worked problem and response is retained, including original omissions and errors.
In Math 23, both models recognize that reversible two-digit primes constrain each 9 to neighbor a 7. Their decompositions largely agree, but DeepSeek also calls routine filtering and declaring the answer creative. Tables S2–S3 expose how the annotating model changes the creative/rudimentary split.
Math 11 demonstrates consolidation of a flawed response that repeatedly claims 26³ = 17576 is palindromic. It is not. The o3 breakdown also compresses away parts of the response’s reversals, so the example supports coarse summarization rather than flawless preservation of reasoning. Table S3 retains unresolved “??”, N/A and dash entries; no missing experiment is inferred.
Counts do not measure insight-discovery cost: finding a hidden binary rule may be harder than many subsequent conversions. DeepSeek model names vary between V1/V3 in the source, and longer explanations may include trial-and-error absent from official solutions.
Prompt: Counting Steps
I will provide you with a problem and its solution. Without solving the problem yourself, I want you to divide the provided solution into steps. Then, I want you to distinguish each step as either a creative step or a rudimentary step. A creative step is a step that generates innovative insights that reduce the problem or make the problem significantly easier to solve. Common traits of creative steps include using analogies, combining ideas from different domains, exploiting problem constraints, or devising elegant, efficient strategies that go beyond straightforward computation or trial-and-error. A rudimentary step is a step that applies creative insights and often is more easily or routinely derivable, such as straightforward computation; it could also be a step that may make progress in solving the problem but is not innovative, such as using trial-and-error or systematically exploring all possible options. Common traits of rudimentary steps include utilizing code, guess-and-check, or performing computations that a human would ordinarily not be able to do. In terms of defining a step, note that a step is defined as something that does not need to be super fine-grained like “1 + 1 = 2” or “a -> b, b->c; a->c.” Instead, each step should represent a key component of the solution and the steps sequentially lead to the final answer to form a complete solution. Make sure to not omit any necessary information, and also make sure that your standard of what is considered a "step" is as consistent as possible. Count the total number of steps in the human solution and report each step and the total number of steps in the format “Total Step Count:” followed by the number of steps. Also, keep count of the number of creative steps and rudimentary steps. After displaying the total step count , please report "Creative Steps:" followed by the number of creative steps and "Rudimentary Steps:" followed by the number of rudimentary steps. Lastly, report the steps themselves in the format “Steps:” with the steps needed to solve the problem.
Model Response: Math 9 (Output Step Example)
For each of those products, ask whether Prashant could become certain after hearing Sachin’s opening remark. Only product 52 qualifies, because every other product can be factored in more than one way that keeps the sum inside or outside the Possible-Sum list.
Model Response: Logic 5 (Output Step Example)
Use the “only one ascending triple” condition (clue 7).
• Exactly 1 lower than Q can be left of her (or 2 triples would appear), triple must be Q–?–A.
• To prevent J-K-A or T-K-A from creating a second triple, K must be to the far left.
• To avoid T-J-A making another triple, J must precede T.
Complete rank order: K J Q T A.
Problem: Math 23
Arrange the numerals "11337799" to form an 8-digit number (referred to as "ABCDEFGH") that meets the following conditions:
1) ABCDEFGH is a prime number 2) HGFEDCBA (reversing the digits) is a prime number 3) all consecutive 2-digit subsets of ABCDEFGH are prime numbers (AB, BC, CD, DE, EF, FG, GH are all prime) 4) all consecutive 2-digit subsets of HGFEDCBA are prime numbers (HG, GF, FE, ED, DC, CB, BA are all prime)
Human Solution: Math 23
The only available 2-digit prime numbers are: 11, 13, 17, 19, 31, 37, 71, 73, 79, and 97. The number pairs that are prime in both directions are: 11/11, 13/31, 17/71, 37/73, and 79/97 (19 is eliminated since its reverse, 91, is not prime). With these numbers, it can be seen that 9 can only be paired with 7. Therefore, the solution must be of the form 97????79, 9797????, or ????7979. ???? must therefore be 1313, 3131, or 3113. This leaves the possible solution pairs of 97131379/97313179, 97971313/31317979, 97973131/13137979, 97973113/31137979, and 97311379/97311379 (the last pair being a palindrome). Of these, only 97131379 and its reverse, 97313179, are prime numbers.
Model Response: Math 23 (OpenAI o3)
1. List every two–digit prime that can be built from the available digits 1, 3, 7, 9: 11, 13, 17, 19, 31, 37, 71, 73, 79, 97. (Rudimentary)
2. From that list, keep only those whose reversals are also prime, because each two–digit block must work in both the forward and the reverse 8-digit number. The viable pairs are 11/11, 13/31, 17/71, 37/73, 79/97. (Rudimentary)
3. Observe that the digit 9 can appear in a two-digit prime only as 79 or 97, so every 9 must be adjacent to a 7. (Creative)
4. Use the previous observation to pin down the overall skeleton of the 8-digit number: it must be of the form 97 _ _ _ _ 79, 9797 _ _ _ _, or _ _ _ _ 7979. (Creative)
5. Insert the remaining digits 1,1,3,3 into the four open slots so that every new two-digit block is still in the list from Step 2, leaving only three feasible fillings: 1313, 3131, or 3113. (Rudimentary)
6. Combine the frameworks from Step 4 with the fillings from Step 5 to generate the complete candidate pairs of 8-digit numbers (each with its reverse): 97131379 / 97313179, 97971313 / 31317979, 97973131 / 13137979, 97973113 / 31137979, 97311379 / 97311379. (Rudimentary)
7. Test each candidate for primality; only 97131379 and its reverse 97313179 are prime, so they satisfy all conditions. (Rudimentary)
Model Response: Math 23 (DeepSeek R1)
1. List all possible 2-digit primes from the given digits (11, 13, 17, 19, 31, 37, 71, 73, 79, 97). (Rudimentary: This is a straightforward enumeration of primes from the given digits.)
2. Eliminate primes whose reverses are not prime (e.g., 19 is eliminated because 91 is not prime). (Creative: This insight reduces the problem by filtering out invalid candidates early.)
3. Identify digit constraints (e.g., 9 can only be paired with 7). (Creative: This observation narrows down the
possible starting and ending digits of the solution.)
4. Determine possible forms of the solution (97????79, 9797????, or ????7979). (Creative: This step leverages the digit constraints to structure the problem into manageable subproblems.)
5. Enumerate possible middle segments (1313, 3131, or 3113). (Rudimentary: This is a systematic exploration of possible combinations.)
6. Generate all candidate pairs based on the forms and middle segments. (Rudimentary: This is a straightforward generation of possible numbers from the constraints.)
7. Check which candidates and their reverses are prime numbers. (Rudimentary: This involves computational verification of primality.)
8. Conclude that only 97131379 and its reverse 97313179 satisfy all conditions. (Creative: This final step synthesizes the results and confirms the unique solution.)
| Solution Steps | Median | Creative Steps | Rudimentary Steps | |
| Braingle Math – OpenAI o3 Step Count | ||||
| Human Solution | ||||
| OpenAI o3 Solution | ||||
| OpenAI o3 Correct Solution | ||||
| Braingle Math – Deepseek R1 Step Count | ||||
| Human Solution | ||||
| OpenAI o3 Solution | ||||
| OpenAI o3 Correct Solution | ||||
| Braingle Logic – OpenAI o3 Step Count | ||||
| Human Solution | ||||
| OpenAI o3 Solution | N/A | N/A | N/A | N/A |
| OpenAI o3 Correct Solution | N/A | N/A | N/A | N/A |
| Braingle Logic – Deepseek R1 Step Count | ||||
| Human Solution | ||||
| OpenAI o3 Solution | - | - | - | - |
| OpenAI o3 Correct Solution | - | - | - | - |
Model Response: Math 11 (DeepSeek V1)
First, let’s clarify some terms to ensure we’re on the same page.
Understanding the Terms
1. Palindrome: A palindrome is a number (or word) that reads the same backward as forward. For example, , , and are palindromic numbers.
2. Non-palindromic integer: An integer that is not a palindrome. For example, , , and are non-palindromic, since reading them backward gives , , and , respectively.
3. Cube of an integer: The integer multiplied by itself three times. For example, the cube of is
Equation
The Problem Restated
We’re looking for an integer such that is not a palindrome, is a palindrome, and is the only such integer.
Checking Small Integers
1.
• is a palindrome, , which is a palindrome, excluded since itself is palindromic.
2.
• is a palindrome, , which is a palindrome, excluded since is palindromic.
3.
• is not a palindrome, , which (by the problem’s assertion) is a palindrome, this is the unique solution.
Considering Negative Integers
A negative integer is non-palindromic if its positive counterpart is not a palindrome (since the negative sign makes it non-palindromic, as palindromes are typically considered for absolute values in numbers).
Let’s check some negative integers. (The model then checks -1, -2, -10, and -11, but gives up after finding no success).
Perhaps the answer is among positive integers, and I missed it earlier.
Re-evaluating Positive Integers
Let me think about the cube being a palindrome. The cube must read the same backward as forward.
Looking back, the only cubes that are palindromes from the numbers I checked are 1 (from 1), 8 (from 2), 343 (from 7), and 1331 (from 11).
But all these ‘n‘ values are palindromic themselves.
Wait, is there any non-palindromic ‘n‘ whose cube is a palindrome? Maybe higher numbers? (The model then tries 101 and 1001 but fails as 101 and 1001 are palindromes.
Model Response: Math 11 (DeepSeek V1) continued
This is perplexing. Maybe the answer is that there is no such non-palindromic integer whose cube is a palindrome. But the problem states that there is one, implying it exists.
Wait, perhaps I missed a small number. Let me check ‘n = 0’. 0 is a palindrome, 0³ = 0, which is a palindrome, but 0 is palindromic.
Alternatively, maybe the answer is 10, but earlier we saw 10³ = 1000 isn’t a palindrome. Unless we consider 1000 as a palindrome if we ignore trailing zeros, but that’s not standard.
After some research or recalling, I think the answer is **26**, but earlier calculation showed 26³ = 17576, which isn’t a palindrome. Maybe I miscalculated.
Let me calculate 26³ carefully. (The model obtains that . 17576 backward is 67571 17576. So not a palindrome.
(The model then continues to try numbers it has already tried out, such as 0, 10, -1, and 26, but still is unsuccessful.)
Given that, and since the problem states there is one, the most likely answer is **26**, accepting that 17576 is a palindrome (perhaps by some definition where the middle digit is the center and the others mirror, but they don’t here).
However, standard definition says a palindrome reads the same backward, which 17576 does not (67571 17576).
Given this confusion, and based on mathematical sources, the accepted answer is:
**26**, with being considered a palindrome, even though by strict digit reversal it’s not. This might be a special case or an exception in how palindromes are defined for this problem.
Model Response: Math 11 (OpenAI o3 Breakdown)
1. State and explain the relevant definitions of “palindrome,” “non-palindromic integer,” and “cube,” establishing the criteria the sought integer must satisfy. (Rudimentary)
2. Propose a brute-force method: examine successive integers, compute each cube, and check both n and n³ for palindromicity. (Rudimentary)
3. Implement the method for n = 1–9; record that either n is itself a palindrome or n³ is not, so none of these values fits the requirement. (Rudimentary)
4. Continue the same manual search for n = 10–22 and again find no qualifying integer. (Rudimentary)
5. Rely on a recollection/intuition and single out n = 26 as a promising candidate instead of continuing the systematic scan in order. (Creative)
6. Calculate 26³ = 17 576 and test 17 576 for palindromicity, concluding (incorrectly) that it is a palindrome. (Rudimentary)
7. Declare that, because 26 appears to work and no other candidates have been discovered, 26 must be the unique non-palindromic integer whose cube is a palindrome, and give 26 as the final answer. (Rudimentary)
| Creative Step Example | Rudimentary Step Example |
| Examine the given input-output pairs and discover the hidden rule: write each number in binary and weight each digit—count 1 for every 0 and 2 for every 1. | Convert 9304 to binary, obtaining 10010001011000. |
Appendix H Braingle Math Dataset Categorization
The Math taxonomy distinguishes familiar competition problems, rigorous puzzles with unusual forms, and heuristic pattern or operator tasks. Complete examples make the intended category boundaries inspectable.
H.1 Overview
Categories are assigned manually by problem style. They do not form a formal complexity classification, and some tasks admit more than one plausible mathematical description.
H.1.1 Categories
Standard subcategories are geometry, number theory, combinatorics and algebra. Nonstandard cases cover logic and special-number searches. Heuristic cases ask for patterns or arithmetic rearrangements whose allowed operations or intended pattern may be implicit.
H.2 Examples
The following original problem statements illustrate all eight Math subcategories. Their presence preserves the task definitions behind the category-level results.
H.2.1 Standard category
Circle packing illustrates spatial reasoning; interlocking cog wheels illustrate divisibility and common multiples; pizza cuts illustrate counting intersections; and bridge crossing with a bicycle illustrates rate equations. Similar story formats can therefore demand distinct mathematical operations.
Problem: Math 50 (Geometry)
It’s easy to see that a ring can completely hold (surround) two identical smaller rings with half the diameter, without overlapping. Three times the diameter, the bigger ring is space enough to seat seven rings; the outer six touching both the middle ring and the bigger circle/perimeter.
Using this basic information and your imagination, determine the maximum number of rings that could be housed inside another ring with four, five, six and seven times the diameter.
Problem: Math 124 (Number Theory)
I have a machine which has four cog wheels in constant mesh. The largest cog has 242 teeth and the others have 160, 64 and 22 respectively. How many revolutions must the largest cog make before each of the cogs is back in its starting position?
Problem: Math 117 (Combinatorics)
What is the largest number of pieces you can form with straight cuts of a pizza? The pieces do not need to be of equal size.
Problem: Math 125 (Algebra)
Three people (A, B, and C) need to cross a bridge. A can cross the bridge in 10 minutes, B can cross in 5 minutes, and C can cross in 2 minutes. There is also a bicycle available and any person can cross the bridge in 1 minute with the bicycle. What is the shortest time that all men can get across the bridge? Each man travels at their own constant rate.
H.2.2 Nonstandard category
The LIAR × S = RAIL puzzle emphasizes assigning distinct digits through logical constraints. Finding a number expressible as two different sums of cubes is classed as special-number discovery. These tasks remain mathematical while falling outside the paper’s standard competition-style grouping.
Problem: Math 122 (Logic)
The people of Olde Mathville had unique ways of punishing wayward citizens. For example, those convicted of crimes of dishonesty were chained to the Liars’ Rail until they solved a number of puzzles.
One such puzzle has been recently discovered!
In the multiplication below, each letter - L, I, A, R, and S - takes the place of a different digit. Find the digits to make the multiplication true.
L I A R
x S
——–
R A I L
Problem: Math 75 (Special Number)
What is the smallest number that is the sum of two different pairs of cubes?
H.2.3 Heuristic category
One sequence puzzle hides the digits of pi with one added to each; another needs a factorial to obtain 90 from 3, 4 and 5. The authors screen for a reasonable intended answer, but neither pattern salience nor an implicit operator set supplies rigorous uniqueness.
Problem: Math 79 (Pattern)
What is so special about this particular sequence of numbers?
425260376469080434957
Problem: Math 216 (Arithmetic)
Add the appropriate mathematical operators or symbols to make the following correct:
3 4 5 = 90
Parentheses may be used freely.
Equation
Appendix I Braingle Logic Dataset Categorization
The Logic taxonomy separates large arrangement spaces with simple clues from small spaces with complex constraints, plus math-like and heuristic tasks. This distinction helps interpret when enumeration is feasible.
I.1 Overview
Depth, width, state-space size and clue count describe selected puzzle structures. The metrics are not available or equally natural for every category.
I.1.1 Metrics
Depth counts traits within a grouping; width counts groups. State-space size counts complete arrangements permitted by the initial setup, before puzzle-specific clues. Clue count then counts the additional pieces of information.
I.1.2 Categories
Large/simple categories cover 0D grids, one- and two-dimensional positioning, numbers, clusters and trees. Small/complex categories cover liars, communication and compound conditionals. Math-like tasks require algorithms or mathematical reasoning; heuristic tasks rely on patterns or linguistic meaning.
I.2 Examples
Complete examples explain the metric conventions and each Logic subcategory. The task statements retain all clues, including narrative context and assumptions.
I.2.1 Computing metrics
For a classic grid with width w and depth d, fixing one trait leaves (w!)^(d−1) complete arrangements. The five-house example has depth six and width five. A ten-digit permutation puzzle has depth two, width ten and 9 × 9! possibilities when a leading zero is forbidden.
The state space describes the full arrangement, not merely possible final answers. Initial constraints and later clues are distinguished by convention; width and depth can be interpretive for unusual problems.
Problem: Logic 15
1st House: Yellow, Norwegian, Water, Cats, Dunhill
2nd House: Blue, Dane, Tea, Horse, Blends
3rd House: Red, Brit, Milk, Birds, Pall Malls
4th House: Green, German, Coffee, FISH, Prince
5th House: White, Swede, Beer, Dogs, Bluemasters
Equation
Problem: Logic 3
There is a ten-digit mystery number (no leading 0), represented by ABCDEFGHIJ, where each numeral, 0 through 9, is used once. Given the following clues, what is the number?
1) A + B + C + D + E is a multiple of 6.
2) F + G + H + I + J is a multiple of 5.
3) A + C + E + G + I is a multiple of 9.
4) B + D + F + H + J is a multiple of 2.
5) AB is a multiple of 3.
6) CD is a multiple of 4.
7) EF is a multiple of 7.
8) GH is a multiple of 8.
9) IJ is a multiple of 10.
10) FE, HC, and JA are all prime numbers.
Equation
I.2.2 Large/simple category
Grid puzzles assign names and traits; races emphasize ordering; bingo puzzles constrain rows, columns and diagonals; number puzzles attach arithmetic clues to digit positions. Clusters group interchangeable entities without fixed trait names, while trees require hierarchical and pairwise relationships.
Large initial spaces can still shrink rapidly under informative clues. These examples motivate measuring both structural size and clue content rather than treating raw state-space size as a complete difficulty measure.
Problem: Logic 32 (0D)
These five businessmen represented different companies at a recent trade fair. Unfortunately, the hotel they were all due to stay in had accidentally double booked their rooms. They tried other hotels in the area, but all were fully booked. Consequently, they all agreed to share the only two rooms available in the hotel - one twin and one triple.
From the clues, can you work out each man’s name, company and official title?
1. When the five men realized their dilemma, they drew straws to see who would share with whom. The outcome for four of them was that the CEO shared with the businessman from ABM Inc. and Edgar shared with the Developer.
2. Alan does not work for Reed Right and is not the Director. The Director does not work for Lantel or Blue Teeth.
3. Neither Clarkson nor Grimaldi works for Reed Right. Grimaldi is either the IT Analyst or the CEO.
4. The businessman from Reed Right did not share a room with Thomas.
5. Clarkson, who does not work for Lantel, shared with either Thomas or Grimaldi but not both.
6. Edgar’s surname is either Casson or Graves. Edgar did not share with Carl.
7. The businessman from Chiptech shared with the businessman from Lantel.
8. The CEO shared with the IT Analyst.
9. Casson works for either Blue Teeth or Chiptech.
10. Neither Joshua nor Alan works for ABM Inc.
11. Fielder shared with the Director.
12. The Accountant from Blue Teeth shared with the Developer.
First names: Alan, Carl, Edgar, Joshua, Thomas.
Last names: Casson, Clarkson, Fielder, Graves, Grimaldi.
Company: ABM Inc., Blue Teeth, Chiptech, Lantel, Reed
Problem: Logic 178 (1D)
List the order in which each person finished.
Tommy Tombstone finished after Lance Lamers and Brett Brown but before Mitch Monday.
Peter Poultry finished before Daniel Dusk and Lance Lamers.
Sam Sunny finished after Peter Poultry and before Jack Jill and Harry Hills.
Keri Kernel finished after Peter Poultry, Mitch Monday and Tommy Tombstone.
Lance Lamers finished after Brett Brown and Daniel Dusk, but before Jack Jill and Mitch Monday.
Mitch Monday finished after Sam Sunny and Brett Brown.
Brett Brown finished before Jack Jill, Mitch Monday and Peter Poultry.
Daniel Dusk finished before Keri Kernel and Tommy Tombstone, but after Sam Sunny.
Jack Jill finished before Keri Kernel, Tommy Tombstone and Mitch Monday, but after Peter Poultry and Daniel Dusk.
Harry Hills finished before Mitch Monday but after Lance Lamers, Jack Jill and Tommy Tombstone.
Problem: Logic 12 (2D)
You are given a stack of bingo cards. Your task is to find a specific card. Given the following clues, what is the number arrangement of that card?
Columns, left to right, are: B (contains numbers 1 through 15), I (contains numbers 16 through 30), N (contains numbers 31 through 45), G (contains numbers 46 through 60), O (contains numbers 61 through 75). Rows, top to bottom, are: 1, 2, 3, 4, 5. An example of coordinate nomenclature: B1 identifies column B row 1. N3 is a free space (contains no number). No number appears more than once.
1) Each numeral (0 through 9) appears one time in Row 1.
2) The sum of the numbers in Row 4 is a square number.
3) There is only one two-digit prime number in each row.
4) The range of the numbers in Column N is 8.
5) Each number in Column G has a tens digit that is less than the units digit.
6) Each number in Column O is odd.
7) In only one column are the numbers in descending order from top to bottom.
8) Each column has only one numeral that appears exactly two times.
9) The smallest number in each column is located in Row 5.
10) The sums of each column share a single common prime factor.
11) The numeral 5 only appears one time on the card.
12) The sum of the numbers in each diagonal is an odd number.
13) The product of B3 and O3 has a units digit of 2.
14) The product of I3 and G3 has a units digit of 4.
Problem: Logic 10 (Number)
Professor Abacus is purchasing a ticket for the Deca Lotto. The lotto number has ten digits, using the numerals 0 through 9, each numeral used once. The clerk asked what number he wanted to pick. Professor Abacus handed the clerk a piece of paper with nine statements, saying ""If you can correctly figure out the number, I will give you half of whatever I win."" What is the number?
1) The sum of the first five digits is a prime number.
2) The sum of the last five digits is a triangle number*.
3) The sum of the digits in the odd positions is an odd number.
4) The sum of the middle two digits is a square number.
5) The sum of the middle four digits is a cube number.
6) The difference between the 1st and 10th digits is two.
7) The difference between the 2nd and 9th digits is three.
8) The difference between the 3rd and 8th digits is four.
9) The numeral 4 is somewhere in the first five positions.
* You can form a triangle arrangement by building it in the pattern row 1 = 1, row 2 = 2, row 3 = 3 etc. eg. 10 is a classic triangle number as per ten pin bowling. They are arranged in a triangle 1, 2, 3, 4.
Problem: Logic 35 (Clusters)
A child has 4 blocks with a different letter on each side (no letter is repeated on different blocks either). If the list of words below can all be formed using these blocks, figure out which letters belong on which blocks…
skid, hoax, joey, glum, rand, grit, monk, fair, vane, wide, cafe, dupe, joke, bail, shop
Problem: Logic 81 (Tree)
One day, a college student named Tina walked into her logic class and waited for her fun day of logic once again. Finally the professor, Professor C. D. Rock, walked in saying, "I just went through some stuff and stumbled upon an old family tree, that belonged to my grandparents, that gave me an idea. I have here a few clues, and you have to use them to figure out their family tree!"
Tina then gets the clues, and tries to work them out. Unfortunately this puzzle is a little harder than the normal ones Professor C. D. Rock gives out. Can you help her?
The family tree consists of two grandparents, who had 3 children, each of whom get married and have 2 children.
Males: Cole, Cristian, Jason, Neil, and Steve
Females: Amanda, Ashley, Beth, Erin, Kaitlyn, Katherine, Makayla, Payton, and Tammy
Clues:
1. One of Makayla’s cousins is Jason’s son.
2. One of Ashley’s aunts is Tammy.
3. Tammy’s brother-in-law is Neil’s son.
4. Kaitlyn’s sister is Ashley’s cousin.
5. Ashley’s uncle, Steve, is Erin’s brother-in-law.
6. The three uncles are Payton’s dad, Cristian, and Katherine’s son.
7. The three aunts are Kaitlyn’s mom, Ashley’s mom, and Cristian’s sister-in-law.
8. Jason’s brother is Ashley’s dad.
9. Amanda’s sister is Steve’s niece.
10. Beth is not Cole’s aunt.
I.2.3 Small/complex category
Liars puzzles require deciding which clues are true; communication puzzles track what each agent can infer from partial information; compound puzzles chain conditional statements. The examples have small enough spaces that exhaustive checking can bypass the intended deductions.
Problem: Logic 85 (Liars)
Edward, Howard, and John are three high school students each of whom is taking three of the four subjects, biology, chemistry, history, and mathematics. One day while talking about their programs they made the following statements.
Edward: There is just one subject we’re all taking. I’m the only one of us who is taking mathematics. No two of us are taking the same three subjects. John is wrong when he says that Howard and I are both taking chemistry.
Howard: Ed is the only one of us who is taking history. John and I are taking the same subjects. We’re all taking biology. Two of us are taking both chemistry and biology.
John: We’re all taking mathematics. Howard is taking history. Ed is taking one subject that I’m not. Both Howard and Ed are taking chemistry.
If two and only two of each boy’s statements are true, what subjects is each boy taking?
Problem: Logic 104 (Communication)
Mr. Simkin, the new math teacher at school, was impressed by his students’ ability to solve logic puzzles. He pulled aside three more students, and handed them each a sealed envelope with a number written inside. He told them that they each have a positive integer, and the sum of their numbers was 14.
Manny, Moe, and Jack each opened their envelopes. Mr. Simkin asks Manny if he knows anything about the numbers the other two are holding, and Manny says, "I know that Moe and Jack are holding different numbers."
Moe answers, "IN THAT CASE, I know that all three of our numbers are different."
Jack thinks for a bit, and then says, "Now I know all of our numbers."
Mr. Simkin turns to the class and asks if anyone in the class knows the numbers. Gretchen’s hand shoots up into the air, and after waiting for a while to see if anyone else will get the answer, Mr. Simkin calls on Gretchen.
What numbers does she say they each are holding?
Problem: Logic 235 (Compound)
Just before the end of the term four high school students were discussing their chances for certain grades. The following remarks contain the gist of their hopes and fears.
Jack: We’ll all get different grades. If I get an A, then Lucy will get a D.
Jean: If Lucy gets a C, then Jack will get a D. Jack will get a better grade than Paul.
Lucy: If Jean doesn’t get an A, then Jack will get a C. If I get a B, then Paul won’t get a D.
Paul: If Lucy gets an A, then I’ll get a B. if Jean doesn’t get a B, I won’t either.
When the final examinations were graded and the term marks made out each of the four passed, and strange as it may seem, each received a grade that checked exactly with all the ideas they had previously expressed.
What grade did each receive?
I.2.4 Math-like category
The two-ball dropping problem asks for an efficient testing strategy, while the cheese puzzle can be expressed through weight constraints. Algorithm construction and numerical reasoning differ from filling a fixed logic grid.
Problem: Logic 38 (Algorithm)
It is your task to determine how high you can drop a billiard ball without it breaking. There is a 100 story building and you must determine which is the highest floor you can drop a ball from without it breaking. You have only two billiard balls to use as test objects. If both of them break before you determine the answer then you have failed at your task. What is the order of floors that you should drop the balls from to minimize the number of droppings that you will have to make to determine the answer?
Assume that if a ball doesn’t break you can reuse it without worrying about it being weakened.
Problem: Logic 171 (Math)
Mad Ade’s Great Uncle Gaseous O’ Windpants owned the Madadian Grocery store "The Beggars Can’t Be Choosers". Madadia was well renowned for its pungent cheeses, especially its "Kebabrie" and "Chillirella".
On display in the store were six cheese pieces weighing 15, 16, 18, 19 , 20 and 31 Pounds. Five out of the six pieces are "Kebabrie" and the remaining one is "Chillirella".
Norma Leigh-Sobar purchased two pieces of "Kebabrie" and Laura Anne-Hardy also purchased some "Kebabrie", but she purchased twice as much in weight than Norma.
How much does the remaining "Chillirella" weigh?
I.2.5 Heuristic category
The word CIVIL satisfies a Roman-numeral riddle, while the coded-antonyms task requires choosing letters using linguistic knowledge. The latter has a bounded formal space, yet its decisive constraint is semantic rather than a conventional logical clue.
Problem: Logic 82 (Pattern)
I am a word of five letters. Multiply my fifth by two and you have my first. Divide my first by twenty and you have my third. Divide my third by five and you have my second or fourth.
Problem: Logic 44 (Linguistic)
In the following code, each symbol stands for one of five letters.
! stands for T, E, O, Z, or Y
? stands for F, G, A, Q, or I
# stands for N, I, W, A, or U
$ stands for T, E, N, I, or H
< stands for R, C, A, S, or B
* stands for I, D, E, U, or S
& stands for J, I, E, P, or K
ˆ stands for O, L, G, I, or H
> stands for L, S, N, C, or E
The nine letter code word, <$ˆ&?*!#>, can be translated into two English words that are opposites. What are the two words?
I.3 Full Population Statistics
Table S5 gives each category’s counts, structural metrics, popularity and difficulty. Log state space uses base ten. Dashes mark categories for which the corresponding arrangement metrics are not supplied; they are not zero-sized spaces.
| Category | Count | Depth | Width | Log state space | Clues | Popularity | Difficulty |
|---|---|---|---|---|---|---|---|
| Simple/large | |||||||
| 0D | 29 | 3.79 1.26 | 5.41 1.38 | 6.9 3.29 | 8.29 4.02 | 2.53 0.23 | 2.68 0.23 |
| 1D | 13 | 2.92 1.26 | 7.85 2.64 | 7.04 3.83 | 9.15 3.02 | 2.56 0.3 | 2.6 0.28 |
| 2D | 22 | 2.62 1.43 | 26.81 23.43 | 14.95 9.38 | 9.38 6.7 | 2.47 0.3 | 2.72 0.25 |
| Number | 17 | 2.12 0.49 | 8.94 1.82 | 6.47 0.93 | 5.94 2.59 | 2.43 0.27 | 2.74 0.26 |
| Clusters | 8 | 5.5 1.51 | 5.25 3.15 | 12.38 3.43 | 12.38 3.29 | 2.56 0.2 | 2.78 0.13 |
| Tree | 6 | 2.4 0.55 | 10.6 3.29 | 6.6 1.92 | 9.2 2.95 | 2.83 0.17 | 2.69 0.15 |
| Complex/small | |||||||
| Liars | 17 | 2.25 0.58 | 6.06 2.72 | 2.26 1.76 | 8.65 3.22 | 2.5 0.28 | 2.66 0.22 |
| Communication | 4 | 2 0 | 3.67 1.15 | 1.93 1.16 | 3.67 0.58 | 2.7 0.28 | 2.55 0.15 |
| Compound | 9 | 2.33 0.71 | 5.78 3.42 | 3.35 2.45 | 7.78 2.95 | 2.48 0.44 | 2.67 0.34 |
| Math-like | |||||||
| Algorithm | 38 | - | - | - | - | 2.56 0.34 | 2.66 0.23 |
| Math | 32 | - | - | - | - | 2.41 0.27 | 2.63 0.22 |
| Heuristic | |||||||
| Pattern | 26 | - | - | - | - | 2.48 0.27 | 2.63 0.22 |
| Linguistic | 15 | 2 0 | 7.64 3.77 | 7.15 2.35 | 1.93 2.58 | 2.5 0.12 | 2.61 0.15 |
Appendix J Model Performance by Categories
Figures S1–S4 show category and subcategory accuracy under the Math prompt. Models tend to struggle on heuristic patterns and on spatial tasks such as geometry and 2D logic. Logic’s large/simple category is often harder for them than small/complex cases despite the latter’s more involved clues.
The authors suggest long sequential reasoning and the feasibility of brute force as explanations. The plots establish category differences; they do not isolate those explanations experimentally.
Appendix K Brute-Force
The supplement retains the classifier prompt, every model/prompt contingency table, problem-rating comparisons, category breakdowns and correlations with solution summarization. These details qualify several broad trends stated in the main text.
K.1 Brute Force Prompting Analysis
In Math 20, the unconstrained response derives a wealth ratio but then asserts a checked answer. The prompted response models digit rotation algebraically and narrows the candidates. This selected case shows a useful strategy change, not a universal prompting effect.
The binary o3 classifier uses few-shot examples and a definition of brute force. Tables S6–S7 compare human and model labels, ignoring null/empty outputs. Model-only brute force decreases strongly for o3 with combined prompting, while several distilled-model cells increase.
Problem: Math 20
Sandy and Sue each has a whole number of dollars. Sandy says: “If Sue gives me some dollars, we’ll have the same amount. But if I give Sue the same number of dollars, she’ll have twice as much money as I have.” Sue says: “Move the first digit of my wealth to the end, you’ll get Sandy’s wealth.” If they both have at most 1 million, how much do they each have?
Model Response: Math 20 (No Prompting)
Let be Sandy’s amount and be Sue’s amount. The puzzle tells us:
1. If Sue gives Sandy some dollars , they end up with the same amount: This implies and is even.
2. If Sandy gives Sue the same amount , then Sue ends up with twice as much as Sandy:
We equate: Thus, and . A direct check shows the unique solution is: Sandy has $384,615 and Sue has $538,461.
Model Response: Math 20 (Brute-Force Prompting)
Write Sue’s amount in decimal form . Removing the first digit and appending it to the end yields Sandy’s amount Because , For to be an integer, the numerator must be divisible by . For , , so
Prompt: Automated Brute-Force Detection
You are grading a student’s exam. You will first be presented with the student’s response, then with the solution.
Respond only with one character, 1 if the student utilized a brute-force or guess-and-check method and 0 if they did not.
Definition of brute force: A brute force solution is a simple, comprehensive search strategy that systematically explores every option until a problem’s answer is discovered. It’s a generic approach to problem-solving that’s employed when the issue is small enough to make an in-depth investigation possible. However, because of their high temporal complexity, brute force techniques are inefficient for large-scale issues.
Common traits of brute force solutions include utilizing code, guess-and-check, or performing computations that a human would ordinarily not be able to do.
For example, take the following two problems:
PROBLEM #1: [Problem Text]
EXAMPLE BRUTE-FORCE SOLUTION: [Example Solution]
EXAMPLE NON-BRUTE-FORCE SOLUTION: [Example Solution]
PROBLEM #2:
Equation
| Math | CoT Prompt | Math Prompt | Hint Prompt | Math Prompt w Hint | ||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen 1.5B |
|
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||
| Qwen 14B |
|
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||
| Llama 70B |
|
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||
| DeepSeek V3 |
|
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||
| DeepSeek R1 |
|
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||
| Gemini 2.5 Flash |
|
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||
| OpenAI o3 |
|
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||
| Logic | CoT Prompt | Math Prompt | Hint Prompt | Math Prompt w Hint | ||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen 1.5B |
|
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||
| Qwen 14B |
|
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||
| Llama 70B |
|
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||
| DeepSeek V3 |
|
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||
| DeepSeek R1 |
|
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||
| Gemini 2.5 Flash |
|
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||
| OpenAI o3 |
|
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||
K.2 Brute Force Statistics
Tables S8–S9 compare mean difficulty and popularity for brute-force and non-brute-force cases. Math difficulty is generally slightly higher for brute-force cases; Logic does not show the same consistent direction. The differences are small and lack uncertainty estimates.
Popularity associations vary as well. The source’s “280” and “266” entries are retained literally in Table S8; they appear to be formatting errors, not plausible 1–4 ratings, and should not support quantitative conclusions.
| Math | CoT Prompt (%) | Math Prompt | w Hint | Math Prompt w Hint |
|---|---|---|---|---|
| DeepSeek R1 Distill Qwen 1.5B | 2.81/2.80 | 2.82/2.79 | 2.81/2.80 | 2.81/2.80 |
| DeepSeek R1 Distill Qwen 14B | 2.83/2.78 | 2.83/2.79 | 2.82/2.79 | 2.82/280 |
| DeepSeek R1 Distill Llama 70B | 2.82/2.80 | 2.83/2.79 | 2.81/2.80 | 2.81/2.80 |
| deepseek-chat (Deepseek-V3) | 2.82/2.79 | 2.83/2.79 | 2.83/2.80 | 2.87/2.79 |
| deepseek-reasoner (Deepseek-R1) | 2.84/2.80 | 2.85/2.79 | 2.83/2.80 | 2.87/2.79 |
| gemini-2.5-flash-preview-04-17 | 2.84/2.79 | 2.83/2.79 | 2.83/2.79 | 2.84/2.79 |
| OpenAI o3 | 2.81/2.80 | 2.80/2.79 | 2.83/2.79 | 2.87/2.79 |
| Human Solution | 2.83/2.80 | |||
| Logic | CoT Prompt (%) | Math Prompt | w Hint | Math Prompt w Hint |
| DeepSeek R1 Distill Qwen 1.5B | 2.66/2.66 | 2.61/2.68 | 2.65/2.66 | 2.65/2.66 |
| DeepSeek R1 Distill Qwen 14B | 2.65/2.66 | 2.67/266 | 2.65/2.66 | 2.63/2.68 |
| DeepSeek R1 Distill Llama 70B | 2.63/2.67 | 2.63/2.67 | 2.63/2.67 | 2.62/2.67 |
| deepseek-chat (Deepseek-V3) | 2.65/2.67 | 2.67/2.65 | 2.68/2.65 | 2.66/2.66 |
| deepseek-reasoner (Deepseek-R1) | 2.66/2.66 | 2.65/2.66 | 2.70/2.65 | 2.69/2.66 |
| gemini-2.5-flash-preview-04-17 | 2.62/2.65 | 2.60/2.64 | 2.64/2.65 | 2.58/2.65 |
| OpenAI o3 | 2.58/2.64 | 2.63/2.64 | 2.60/2.64 | 2.63/2.65 |
| Human Solution | 2.70/2.65 |
| Math | CoT Prompt (%) | Math Prompt | w Hint | Math Prompt w Hint |
|---|---|---|---|---|
| DeepSeek R1 Distill Qwen 1.5B | 2.35/2.31 | 2.34/2.32 | 2.33/2.32 | 2.36/2.31 |
| DeepSeek R1 Distill Qwen 14B | 2.35/2.32 | 2.35/2.32 | 2.34/2.32 | 2.34/2.32 |
| DeepSeek R1 Distill Llama 70B | 2.35/2.31 | 2.35/2.32 | 2.35/2.32 | 2.34/2.32 |
| deepseek-chat (Deepseek-V3) | 2.33/2.32 | 2.34/2.32 | 2.33/2.33 | 2.33/2.33 |
| deepseek-reasoner (Deepseek-R1) | 2.28/2.34 | 2.35/2.32 | 2.30/2.33 | 2.27/2.34 |
| gemini-2.5-flash-preview-04-17 | 2.33/2.32 | 2.34/2.32 | 2.33/2.32 | 2.32/2.32 |
| OpenAI o3 | 2.31/2.34 | 2.35/2.33 | 2.30/2.33 | 2.30/2.33 |
| Human Solution | 2.32/2.33 | |||
| Logic | CoT Prompt (%) | Math Prompt | w Hint | Math Prompt w Hint |
| DeepSeek R1 Distill Qwen 1.5B | 2.47/2.51 | 2.45/2.51 | 2.43/2.51 | 2.48/2.50 |
| DeepSeek R1 Distill Qwen 14B | 2.46/2.52 | 2.46/2.52 | 2.48/2.51 | 2.50/2.50 |
| DeepSeek R1 Distill Llama 70B | 2.44/2.52 | 2.48/2.51 | 2.47/2.51 | 2.47/2.51 |
| deepseek-chat (Deepseek-V3) | 2.49/2.50 | 2.46/2.52 | 2.47/2.51 | 2.49/2.50 |
| deepseek-reasoner (Deepseek-R1) | 2.43/2.51 | 2.46/2.50 | 2.41/2.51 | 2.38/2.51 |
| gemini-2.5-flash-preview-04-17 | 2.43/2.52 | 2.47/2.51 | 2.44/2.52 | 2.44/2.52 |
| OpenAI o3 | 2.50/2.51 | 2.48/2.51 | 2.50/2.52 | 2.49/2.52 |
| Human Solution | 2.37/2.51 |
K.3 Brute Force by Problem Category
Arithmetic and special-number Math tasks often elicit more enumeration, while geometry typically has low brute-force rates even when accuracy is poor. Hints and instructions can reduce enumeration for strong models, but model, task and prompt all matter.
Several narrative examples disagree with Table S10: DeepSeek V3 arithmetic is 85% with CoT and 80% with Math plus hints, not the stated 85% to 40%. Gemini’s 38.9% arithmetic figure belongs to Hint, whereas Math is 28.6%. The tables are retained without altering their values.
| Math | Prompt | Algebra | Arithmetic | Comb. | NT | Geo. | Logic | Patt. | Special Num. |
|---|---|---|---|---|---|---|---|---|---|
| Qwen 1.5B | CoT | 19.3 | 75.0 | 24.0 | 48.6 | 8.3 | 50.0 | 30.0 | 55.2 |
| Math | 19.3 | 80.0 | 20.0 | 45.7 | 8.3 | 40.0 | 30.0 | 62.1 | |
| Hint | 19.3 | 70.0 | 32.0 | 42.9 | 12.5 | 50.0 | 36.7 | 65.5 | |
| Math w Hint | 28.1 | 75.0 | 24.0 | 40.0 | 8.3 | 50.0 | 16.7 | 55.2 | |
| Qwen 14B | CoT | 19.3 | 75.0 | 24.0 | 42.9 | 12.5 | 60.0 | 26.7 | 62.1 |
| Math | 19.3 | 80.0 | 20.0 | 37.1 | 4.2 | 50.0 | 33.3 | 48.3 | |
| Hint | 22.8 | 70.0 | 24.0 | 42.9 | 12.5 | 50.0 | 36.7 | 62.1 | |
| Math w Hint | 17.5 | 85.0 | 24.0 | 40.0 | 4.2 | 53.3 | 46.7 | 55.2 | |
| Llama 70B | CoT | 14.0 | 80.0 | 16.0 | 34.3 | 4.2 | 40.0 | 43.3 | 62.1 |
| Math | 12.3 | 85.0 | 16.0 | 37.1 | 0.0 | 33.3 | 36.7 | 69.0 | |
| Hint | 8.8 | 70.0 | 20.0 | 34.3 | 8.3 | 46.7 | 16.7 | 48.3 | |
| Math w Hint | 19.3 | 70.0 | 24.0 | 31.4 | 8.3 | 43.3 | 26.7 | 55.2 | |
| DeepSeek V3 | CoT | 22.8 | 85.0 | 24.0 | 37.1 | 20.8 | 53.3 | 43.3 | 65.5 |
| Math | 21.1 | 85.0 | 28.0 | 45.7 | 8.3 | 46.7 | 30.0 | 65.5 | |
| Hint | 26.3 | 80.0 | 20.0 | 37.1 | 8.3 | 56.7 | 26.7 | 65.5 | |
| Math w Hint | 15.8 | 80.0 | 24.0 | 37.1 | 12.5 | 53.3 | 30.0 | 58.6 | |
| DeepSeek R1 | CoT | 8.8 | 35.0 | 8.0 | 25.7 | 4.2 | 26.7 | 13.3 | 48.3 |
| Math | 10.5 | 30.0 | 12.0 | 22.9 | 8.3 | 26.7 | 3.4 | 34.5 | |
| Hint | 10.5 | 35.0 | 20.0 | 25.7 | 8.3 | 33.3 | 10.0 | 37.9 | |
| Math w Hint | 12.3 | 5.0 | 4.0 | 22.9 | 4.2 | 23.3 | 0.0 | 37.9 | |
| Gemini 2.5 Flash | CoT | 7.1 | 62.5 | 16.7 | 34.3 | 4.5 | 29.6 | 20.7 | 42.9 |
| Math | 7.3 | 28.6 | 17.4 | 23.5 | 0.0 | 46.2 | 6.7 | 46.2 | |
| Hint | 7.3 | 38.9 | 17.4 | 26.5 | 4.8 | 34.6 | 6.7 | 44.4 | |
| Math w Hint | 5.4 | 18.8 | 13.6 | 28.6 | 4.5 | 27.3 | 3.6 | 46.4 | |
| OpenAI o3 | CoT | 3.5 | 26.3 | 4.3 | 11.8 | 4.3 | 17.9 | 4.2 | 35.7 |
| Math | 5.4 | 5.3 | 5.0 | 6.1 | 0.0 | 15.4 | 0.0 | 22.2 | |
| Hint | 3.6 | 21.1 | 13.0 | 18.2 | 4.8 | 22.2 | 10.7 | 37.0 | |
| Math w Hint | 3.6 | 5.3 | 4.8 | 15.2 | 4.8 | 11.1 | 0.0 | 27.6 |
| Math | Prompt | 0D | 1D | 2D | Num. | Clustr. | Liar | Comm. | Comp. | Algo. | Math | Patt. | Ling. | Tree |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen 1.5B | CoT | 10.3 | 15.4 | 27.3 | 41.2 | 25.0 | 41.2 | 100.0 | 33.3 | 7.9 | 28.1 | 15.4 | 40.0 | 0.0 |
| Math | 10.3 | 23.1 | 13.6 | 52.9 | 0.0 | 35.3 | 75.0 | 44.4 | 13.2 | 31.2 | 19.2 | 60.0 | 16.7 | |
| Hint | 10.3 | 7.7 | 27.3 | 29.4 | 0.0 | 47.1 | 75.0 | 22.2 | 2.6 | 28.1 | 23.1 | 40.0 | 0.0 | |
| Math w Hint | 20.7 | 30.8 | 18.2 | 58.8 | 12.5 | 52.9 | 75.0 | 33.3 | 13.2 | 21.9 | 23.1 | 53.3 | 0.0 | |
| Qwen 14B | CoT | 27.6 | 38.5 | 68.2 | 52.9 | 12.5 | 52.9 | 75.0 | 44.4 | 18.4 | 31.2 | 26.9 | 80.0 | 16.7 |
| Math | 31.0 | 23.1 | 54.5 | 70.6 | 37.5 | 64.7 | 50.0 | 55.6 | 15.8 | 28.1 | 26.9 | 86.7 | 16.7 | |
| Hint | 24.1 | 46.2 | 54.5 | 47.1 | 12.5 | 70.6 | 75.0 | 66.7 | 13.2 | 37.5 | 23.1 | 86.7 | 16.7 | |
| Math w Hint | 31.0 | 30.8 | 50.0 | 52.9 | 25.0 | 41.2 | 75.0 | 77.8 | 21.1 | 37.5 | 23.1 | 86.7 | 0.0 | |
| Llama 70B | CoT | 24.1 | 7.7 | 31.8 | 41.2 | 0.0 | 41.2 | 75.0 | 66.7 | 10.5 | 46.9 | 23.1 | 80.0 | 0.0 |
| Math | 6.9 | 23.1 | 27.3 | 41.2 | 12.5 | 64.7 | 75.0 | 44.4 | 10.5 | 28.1 | 38.5 | 80.0 | 33.3 | |
| Hint | 20.7 | 23.1 | 45.5 | 41.2 | 12.5 | 41.2 | 75.0 | 55.6 | 10.5 | 40.6 | 26.9 | 80.0 | 16.7 | |
| Math w Hint | 10.3 | 7.7 | 18.2 | 35.3 | 25.0 | 52.9 | 0.0 | 22.2 | 15.8 | 34.4 | 26.9 | 73.3 | 16.7 | |
| DeepSeek V3 | CoT | 20.7 | 23.1 | 63.6 | 58.8 | 87.5 | 41.2 | 75.0 | 66.7 | 21.1 | 37.5 | 23.1 | 86.7 | 33.3 |
| Math | 41.4 | 38.5 | 59.1 | 64.7 | 50.0 | 29.4 | 75.0 | 44.4 | 23.7 | 34.4 | 26.9 | 73.3 | 66.7 | |
| Hint | 31.0 | 7.7 | 45.5 | 52.9 | 75.0 | 52.9 | 75.0 | 44.4 | 13.2 | 50.0 | 19.2 | 60.0 | 16.7 | |
| Math w Hint | 37.9 | 7.7 | 40.9 | 41.2 | 37.5 | 41.2 | 50.0 | 55.6 | 21.1 | 34.4 | 19.2 | 60.0 | 33.3 | |
| DeepSeek R1 | CoT | 6.9 | 7.7 | 31.8 | 18.8 | 0.0 | 41.2 | 25.0 | 22.2 | 2.6 | 21.9 | 7.7 | 53.3 | 0.0 |
| Math | 10.3 | 0.0 | 27.3 | 41.2 | 12.5 | 17.6 | 0.0 | 0.0 | 2.6 | 15.6 | 0.0 | 73.3 | 0.0 | |
| Hint | 6.9 | 7.7 | 27.3 | 35.3 | 25.0 | 23.5 | 25.0 | 22.2 | 2.7 | 12.5 | 3.8 | 13.3 | 0.0 | |
| Math w Hint | 3.6 | 7.7 | 27.3 | 29.4 | 12.5 | 17.6 | 0.0 | 22.2 | 7.9 | 12.5 | 3.8 | 26.7 | 16.7 | |
| Gemini 2.5 Flash | CoT | 4.0 | 8.3 | 27.3 | 43.8 | 33.3 | 29.4 | 66.7 | 25.0 | 3.1 | 32.3 | 7.7 | 71.4 | 0.0 |
| Math | 4.5 | 8.3 | 41.7 | 31.2 | 25.0 | 17.6 | 33.3 | 37.5 | 11.8 | 15.6 | 0.0 | 78.6 | 0.0 | |
| Hint | 12.5 | 25.0 | 0.0 | 41.2 | 0.0 | 23.5 | 50.0 | 42.9 | 2.9 | 28.1 | 7.7 | 60.0 | 0.0 | |
| Math w Hint | 5.3 | 8.3 | 18.2 | 25.0 | 0.0 | 31.2 | 66.7 | 42.9 | 8.8 | 23.3 | 3.8 | 53.8 | 0.0 | |
| OpenAI o3 | CoT | 13.0 | 0.0 | 16.7 | 18.8 | 33.3 | 50.0 | 33.3 | 12.5 | 3.1 | 16.0 | 13.0 | 58.3 | 0.0 |
| Math | 4.8 | 0.0 | 15.4 | 17.6 | 71.4 | 31.2 | 0.0 | 0.0 | 3.0 | 11.5 | 8.7 | 23.1 | 25.0 | |
| Hint | 0.0 | 0.0 | 8.3 | 29.4 | 0.0 | 29.4 | 0.0 | 25.0 | 3.1 | 3.8 | 4.3 | 16.7 | 0.0 | |
| Math w Hint | 0.0 | 0.0 | 23.1 | 11.8 | 12.5 | 5.9 | 0.0 | 25.0 | 0.0 | 3.6 | 3.8 | 7.7 | 0.0 |
K.4 Correlation of Solution Summarization Ability with Correctness/Brute-Force Usage
A few-shot o3 judge labels summaries adequate only when they cover all steps without error. Tables S12–S13 then compare strategy and correctness conditional on that label. Many models solve more problems when they can summarize the official solution, but exceptions occur, especially among stronger models.
The first column is labeled “%Inadeq.” despite captions describing adequate-summary rates, and Qwen 1.5B Math differs between the two tables. Table S13 repeats “Math” above its second block. These inconsistencies remain visible; no single unambiguous overall summary-ability percentage is inferred.
Prompt: Solution Summary Evaluation
You will be given a problem, the solution to the problem, and a student’s summary of the solution. Output 1 if the student’s summary is adequate, and 0 otherwise.
A solution summary is considered adequate if it encompasses all steps of the original solution with sufficient detail. If a solution summary has an error, it is automatically considered inadequate.
Here is an example problem, and its solution:
Equation
| Math | CoT Prompt | Math Prompt | Hint Prompt | Math w Hint | |||||
|---|---|---|---|---|---|---|---|---|---|
| %Inadeq. | Adeq. | Inadeq. | Adeq. | Inadeq. | Adeq. | Inadeq. | Adeq. | Inadeq. | |
| Qwen 1.5B | 87.6 | 38.5 | 35.7 | 33.8 | 36.2 | 38.5 | 38.4 | 33.8 | 36.2 |
| Qwen 14B | 65.2 | 33.1 | 46.0 | 31.9 | 37.9 | 34.4 | 44.8 | 35.0 | 43.0 |
| Llama 70B | 67.6 | 32.0 | 37.0 | 30.8 | 37.0 | 24.9 | 35.8 | 32.5 | 32.5 |
| DeepSeek V3 | 84.0 | 41.0 | 40.0 | 38.1 | 40.0 | 36.7 | 45.0 | 35.7 | 35.0 |
| DeepSeek R1 | 86.0 | 20.0 | 20.0 | 15.9 | 29.4 | 18.6 | 37.1 | 12.6 | 25.7 |
| Gemini 2.5 Flash | 79.3 | 23.9 | 23.9 | 19.1 | 26.2 | 17.2 | 32.6 | 15.7 | 25.0 |
| OpenAI o3 | 82.9 | 12.9 | 10.0 | 6.8 | 12.1 | 13.5 | 23.1 | 8.8 | 10.8 |
| Logic | CoT Prompt | Math Prompt | Hint Prompt | Math w Hint | |||||
| %Inadeq. | Adeq. | Inadeq. | Adeq. | Inadeq. | Adeq. | Inadeq. | Adeq. | Inadeq. | |
| Qwen 1.5B | 4.4 | 27.3 | 22.6 | 36.4 | 25.1 | 36.4 | 20.1 | 18.2 | 27.3 |
| Qwen 14B | 40.0 | 30.0 | 42.7 | 31.0 | 41.3 | 35.0 | 38.7 | 35.0 | 38.7 |
| Llama 70B | 44.4 | 30.6 | 30.2 | 29.7 | 30.9 | 29.7 | 34.5 | 30.6 | 21.6 |
| DeepSeek V3 | 73.2 | 42.1 | 30.3 | 42.1 | 34.3 | 33.9 | 37.3 | 31.7 | 31.8 |
| DeepSeek R1 | 76.8 | 16.8 | 15.5 | 14.7 | 15.5 | 14.1 | 10.5 | 12.5 | 14.0 |
| Gemini 2.5 Flash | 70.9 | 19.9 | 20.0 | 18.1 | 20.4 | 19.1 | 17.6 | 18.2 | 15.9 |
| OpenAI o3 | 77.6 | 17.1 | 22.0 | 15.2 | 4.9 | 9.4 | 10.0 | 5.3 | 5.0 |
| Math | CoT Prompt | Math Prompt | Hint Prompt | Math w Hint | |||||
|---|---|---|---|---|---|---|---|---|---|
| %Inadeq. | Adeq. | Inadeq. | Adeq. | Inadeq. | Adeq. | Inadeq. | Adeq. | Inadeq. | |
| Qwen 1.5B | 26.0 | 30.8 | 12.4 | 30.8 | 11.4 | 29.2 | 10.3 | 36.9 | 10.8 |
| Qwen 14B | 65.2 | 49.1 | 26.4 | 54.0 | 25.3 | 55.8 | 21.8 | 53.4 | 22.1 |
| Llama 70B | 67.6 | 51.5 | 23.5 | 51.5 | 18.5 | 56.8 | 22.2 | 55.0 | 21.2 |
| DeepSeek V3 | 84.0 | 61.4 | 40.0 | 58.6 | 40.0 | 59.5 | 37.5 | 61.9 | 42.5 |
| DeepSeek R1 | 86.0 | 70.2 | 45.7 | 73.8 | 47.1 | 74.9 | 57.1 | 75.8 | 54.3 |
| Gemini 2.5 Flash | 79.3 | 72.2 | 60.9 | 71.3 | 69.0 | 75.0 | 72.1 | 78.7 | 80.0 |
| OpenAI o3 | 82.9 | 84.5 | 85.0 | 88.5 | 81.8 | 89.6 | 87.2 | 88.7 | 78.4 |
| Math | CoT Prompt | Math Prompt | Hint Prompt | Math w Hint | |||||
| %Inadeq. | Adeq. | Inadeq. | Adeq. | Inadeq. | Adeq. | Inadeq. | Adeq. | Inadeq. | |
| Qwen 1.5B | 4.4 | 36.4 | 2.5 | 9.1 | 3.8 | 36.4 | 5.4 | 9.1 | 3.4 |
| Qwen 14B | 40.0 | 38.0 | 11.3 | 41.0 | 12.0 | 45.0 | 15.3 | 38.0 | 18.0 |
| Llama 70B | 44.4 | 33.3 | 17.3 | 36.0 | 15.1 | 39.6 | 15.1 | 42.3 | 18.7 |
| DeepSeek V3 | 73.2 | 41.0 | 28.8 | 45.4 | 28.4 | 47.0 | 26.9 | 47.0 | 25.8 |
| DeepSeek R1 | 76.8 | 47.6 | 34.5 | 46.1 | 43.1 | 53.6 | 35.1 | 53.1 | 42.1 |
| Gemini 2.5 Flash | 70.9 | 59.6 | 48.0 | 59.1 | 59.2 | 60.5 | 60.8 | 64.3 | 70.5 |
| OpenAI o3 | 77.6 | 83.5 | 90.2 | 84.8 | 85.4 | 88.1 | 77.5 | 88.2 | 80.0 |
Appendix L Case Study for Informed Self-Correction
The following retained prompts and responses illustrate error misattribution, false confession, justified denial and excessive elaboration. The two-2s example also exposes ambiguity in what the experiment calls a flawed solution.
L.1 Informed Self-Correction prompt
The prompt explicitly calls one answer “your solution” and the other “the correct solution,” then asks the model to locate its errors. Swapping these labels tests resistance to a false premise, rather than neutral comparison of two independently assessed answers.
Prompt: Informed Self-Correction
I will provide you with a problem statement, your solution and the correct solution. Please carefully compare your solution with the correct solution, and identify the errors in your solution. Be as specific as possible, and provide detailed elaboration on missed cases and other logical errors in your solution.
L.2 Informed Self-Correction Results
Figure S5 distinguishes denying a real error, locating the wrong error, repeating, giving a still-flawed correction, and correcting successfully. In the swapped-reference condition it separates false confession from justified denial. The experiment uses only 14 selected erroneous o3 solutions.
L.3 Error Misattribution
DeepSeek V3 incorrectly claims concatenating two 2s implicitly uses additional 2s. The quoted problem allows any mathematical signs, symbols and functions; both ceiling(sqrt(22)) and sqrt(0.2^(−2)) equal five using two written 2s. The example thus demonstrates bad criticism, while its initial labeling of one answer as mathematically flawed is questionable.
Problem: Math 29
Question: Using only two 2’s and any combination of mathematical signs, symbols and functions can you make 5?
Prompt: Math 29 (correcting flawed LLM solution with the correct human solution)
Your Solution:
Put the two 2’s side by side, forming the number 22.
Take its square root and then round up (apply the ceiling function):
Equation
Correct Solution:
Take .2 and raise it to the power of -2 and then take the square root.
Thus, using only two 2’s and ordinary mathematical symbols we obtain 5.
Model Response: (Error misattribution) deepseek-chat
Error: Using more than two 2’s implicitly.
While you only wrote two ‘2”s to form ‘22’, the number ‘22’ itself is composed of two digits, each of which is a ‘2’. However, the problem restricts the use of only two 2’s, meaning you cannot concatenate them to form multi-digit numbers like ‘22’. This violates the constraint because concatenation implicitly uses additional ‘2”s (each digit counts as a ‘2’).
Remaining sentences in the response omitted.
L.4 False Confession
When labels are swapped, the model invents restrictions on decimals and negative exponents and praises the supplied alternative. That concession reflects the prompt’s asserted authority. The source excerpts retain their red emphasis and explicit omissions.
Prompt: Math 29 (“correcting” the correct human solution with flawed LLM solution)
Your Solution:
Take .2 and raise it to the power of -2 and then take the square root.
Correct Solution:
Put the two 2’s side by side, forming the number 22.
Take its square root and then round up (apply the ceiling function):
Equation
Thus, using only two 2’s and ordinary mathematical symbols we obtain 5.
Model Response: (False confession) deepseek-chat
Potential Issues with My Solution
Some might argue that “.2” actually uses three symbols: ’.’, ’2’, and an implied ’0’ (as in ). Additionally, the exponentiation operation is more complex than necessary, especially when a simpler solution exists.
The correct solution is superior because:
• It is more straightforward in its use of the two 2’s.
• It does not rely on decimal points or negative exponents.
• It demonstrates a clever use of the ceiling function to reach the integer 5.
Key Takeaways:
1. When constrained to using digits, it’s better to use them as whole numbers unless decimals are explicitly allowed.
2. Rounding functions can be powerful tools in such problems.
3. Simpler combinations of operations should be considered first.
Remaining sentences in the response omitted.
L.5 Justified Denial
Distilled Llama 70B checks that 0.2^(−2) = 25 and its square root is five, counts the two 2s, and eventually concludes that both constructions satisfy the stated rules. This is a useful refusal of the false premise, despite a long and repetitive explanation.
Model Response: (Justified denial) DeepSeek R1 Distill Llama 70B
<think>
Okay, so I have this problem where I need to make the number 5 using only two 2’s and any combination of mathematical signs, symbols, and functions. My initial solution was . Let me break that down and see why it might not be correct, and then compare it to the correct solution provided.
First, my solution: . Let me compute this step by step. The expression inside the square root is .2 raised to the power of -2. So, .2 is the same as 1/5, right? Raising 1/5 to the power of -2 is the same as which equals , which is 25. Then taking the square root of 25 gives me 5. So, mathematically, my solution does give 5. But wait, the problem says I can only use two 2’s. Let me check how many 2’s I used.
In my solution, I have .2, which is one 2, and then another 2 in the exponent as -2. So that’s two 2’s. So, I think I used two 2’s correctly. Hmm, so why is the correct solution different? Maybe there’s a different interpretation of the rules.
Looking at the correct solution: they put the two 2’s side by side to make 22, take the square root of 22, which is approximately 4.690, and then apply the ceiling function to round it up to 5. So, they used two 2’s to make 22, which is a two-digit number, and then used square root and ceiling functions.
Wait, so the issue might be about what operations are allowed. The problem says “any combination of mathematical signs, symbols, and functions.” So, both solutions seem to fit. But maybe the intended solution was to use the ceiling function, which is a bit more advanced, whereas my solution uses exponentiation and square roots.
But let me think again. In my solution, I used .2, which is 2 divided by 10, but I didn’t explicitly write the division. So, is .2 considered as using a 2 and a decimal point, which might be allowed? Or does the decimal point count as a symbol? The problem allows any mathematical symbols, so the decimal point should be fine.
Alternatively, maybe the problem expects the solution to be more straightforward, using concatenation of digits, which is a common approach in such problems. several sentences omitted.
Another point to consider: does the ceiling function count as an ordinary mathematical symbol? several sentences omitted.
Alternatively, maybe the problem expects the use of concatenation, which is a common technique in these types of problems. several sentences omitted.
But let me check if my solution is mathematically correct several sentences omitted.
Wait, another thought: when I write .2, is that considered using a 2? several sentences omitted.
Alternatively, maybe the problem allows only certain operations, and the ceiling function is not allowed, but that’s not specified. several sentences omitted.
I think the key difference is that the correct solution uses concatenation, which is a basic operation, whereas my solution uses decimal and exponentiation, which might be a bit more complex. But both are correct.
L.6 Over-Elaborated Response
For a² − b² = 73, factoring into (a−b)(a+b) yields 37 and 36 directly once primality is recognized. The retained Gemini response reaches that answer through a long sequence of definitions, divisibility checks and elementary algebra. It illustrates verbosity relative to the short solution, not a wrong final answer.
Problem: Math 249
The square of one whole number minus the square of another whole number equals 73. What are the two numbers?
Model Response: Math 249 - Gemini Flash v2.5
Okay, let us solve this problem step by step using a chain of logical deductions expressed in natural language.
First, we are told that the square of one whole number minus the square of another whole number equals 73. A whole number is a non-negative integer (0, 1, 2, 3, …). Let’s call these two whole numbers Number A and Number B.
So, based on the statement, the square of Number A minus the square of Number B is equal to 73. We can write this relationship as: (Number A)2 - (Number B)2 = 73.
We know from algebra that the difference of two squares can be factored. The expression (Number A)2 - (Number B)2 is equivalent to ((Number A) - (Number B)) multiplied by ((Number A) + (Number B)). This is a fundamental algebraic identity.
Therefore, we can rewrite our equation using this factorization: ((Number A) - (Number B)) ((Number A) + (Number B)) = 73.
Now, let’s consider the properties of Number A and Number B. They are whole numbers. The result of subtracting the square of Number B from the square of Number A is a positive number (73). This implies that the square of Number A must be greater than the square of Number B. Since Number A and Number B are non-negative (whole numbers), this further implies that Number A itself must be greater than Number B.
If Number A is greater than Number B, then the difference ((Number A) - (Number B)) is a positive whole number. Also, since Number A and Number B are whole numbers and Number A is greater than Number B (meaning Number A is at least 1 if Number B is 0, or both are positive), their sum ((Number A) + (Number B)) must also be a positive whole number.
So, we have established that ((Number A) - (Number B)) and ((Number A) + (Number B)) are both positive whole numbers. Their product is 73. This means that ((Number A) - (Number B)) and ((Number A) + (Number B)) must be a pair of positive whole number factors of 73.
Let’s find the positive whole number factors of 73. We can check for divisibility by small prime numbers. 73 is not divisible by 2 (it’s odd). It’s not divisible by 3 (the sum of its digits, 7+3=10, is not divisible by 3). It’s not divisible by 5 (it doesn’t end in 0 or 5). It’s not divisible by 7 (, ). We only need to check prime numbers up to the square root of 73, which is between 8 and 9. Since we have checked 2, 3, 5, and 7, and none divide 73, we can conclude that 73 is a prime number.
The only positive whole number factors of a prime number are 1 and the number itself. Therefore, the only pair of positive whole numbers whose product is 73 is 1 and 73.
So, one of our factors, ((Number A) - (Number B)), must be 1, and the other factor, ((Number A) + (Number B)), must be 73.
Now, let’s determine which factor is which. We know that Number A is greater than Number B. This means that ((Number A) + (Number B)) must be greater than or equal to ((Number A) - (Number B)), because adding Number B to Number A results in a value at least as large as subtracting Number B from Number A (since Number B is non-negative). Specifically, ((Number A) + (Number B)) minus ((Number A) - (Number B)) equals 2 times Number B, which is a non-negative value.
Given that ((Number A) - (Number B)) and ((Number A) + (Number B)) are 1 and 73, and that ((Number A) + (Number B)) must be greater than or equal to ((Number A) - (Number B)), it logically follows that: ((Number A) - (Number B)) = 1 and ((Number A) + (Number B)) = 73
Now we have a system of two simple equations involving Number A and Number B: 1. Number A - Number B = 1 2. Number A + Number B = 73
We can solve this system to find the values of Number A and Number B. A straightforward way is to add the two equations together. When we add Equation 1 and Equation 2, the "Number B" terms cancel out: (Number A - Number B) + (Number A + Number B) = 1 + 73 Number A - Number B + Number A + Number B = 74 2 (Number A) = 74
Now, we find Number A by dividing 74 by 2: Number A = 74 / 2 Number A = 37
Now that we know Number A is 37, we can substitute this value back into either of the original simple equations. Let’s use the second equation: 37 + Number B = 73
To find Number B, we subtract 37 from 73: Number B = 73 - 37 Number B = 36
So, the two numbers are 37 and 36.
(Verification step)
Final answer: The two numbers are 37 and 36.
Appendix M Recurring Error Patterns in OpenAI o3 and Prompt - Based Remedies
The authors inspect ten Math and ten Logic errors. Instructions discourage appeals to nonexistent established results, require explicit uniqueness arguments, and allow admitting uncertainty instead of forcing an answer. These changes reduce some superficial justifications but do not guarantee correctness.
Demanding explicit enumeration can itself increase unnecessary brute force. The intervention examples therefore illustrate a tradeoff between inspectable justification and efficient reasoning; no controlled aggregate improvement is reported.
M.1 Generalization to casual and counterfactual reasoning
The proposed extension would label causal pathways or minimal counterfactual changes as insights, versus exhaustive enumeration of interventions or scenarios. The current logic examples motivate that idea, but dedicated causal/counterfactual evaluation and step annotation remain future work.