Literature
Display
← All papers

ROSETTA: Constructing Code-Based Reward from Unconstrained Language Preference

Sanjana Srivastava · Kangrui Wang · Yung-Chieh Chan · Tianyuan Dai · Manling Li · Ruohan Zhang · Mengdi Xu · Jiajun Wu · Li Fei-Fei

· ICLR 2026 ↗

Robotics16 min text · CondensedOriginal paper ↗
Contents

At a glance

ROSETTA turns a person’s changing language preferences into executable reward functions. A grounding module interprets feedback in context, a staging module plans dense intermediate rewards, and a coding module implements and revises the reward. Reinforcement learning then trains candidate policies, from which the person chooses a preferred rollout.

The study covers 116 preferences in 35 chains across five simulation environments, with a smaller real-robot demonstration. Headline success near 87% and satisfaction near 86% measure different things: generated-task optimizability and improvement perceived by the annotator. This edition retains the complete 67-page ICLR 2026 paper, all figures and tables, sixteen unnumbered trajectory strips, and every original prompt, code template and survey.

Complete original figure 1
Figure 1: ROSETTA’s three-module structure takes unconstrained, unseen natural language preference and generates reward code that incentivizes desired behavior. ROSETTA can adapt to diverse behavioral preferences, not only in position but orientation and speed. We validate ROSETTA’s performance in two short-horizon and three long-horizon task-agnostic environments.

1. Introduction

Natural feedback can be colloquial, refer to earlier actions, or change the desired task entirely. Optimizing a fixed hand-coded goal is therefore insufficient: the reward’s meaning must change along with the user’s preferences [3] [22].

The paper studies adaptation after each new preference, rather than repeated corrections toward a fixed goal. “Single-step” refers to one preference interaction; internally the pipeline uses multiple model calls, code variants, policy-training runs and human selection. It is not immediate motor adaptation without learning.

3. Problem formulation

At interaction t, a person observes the previous policy and supplies preference h(t). ROSETTA uses the current policy context and preference history to produce reward R(t); a learning algorithm A_M(R(t)) produces the next policy. The objective is the person’s unknown, time-varying satisfaction F(t)(A_M(R(t))).

Unlike a fixed fitness score, this objective is not available for automated reward search. Human judgments are therefore part of the procedure. Selecting the favorite among a finite set of shown candidates maximizes the observed choice within that set, not the unknown global satisfaction objective.

3.1 Evaluation framework

Three views of performance are complementary. Alignment measures the original annotator’s satisfaction with a rollout. Semantic match asks expert roboticists whether grounded language, the stage plan and code preserve the intended meaning. Optimizability measures whether reinforcement learning succeeds at the generated reward’s own success condition.

A policy can optimize a misinterpreted goal perfectly, or fail a strict generated success test while satisfying the person. None of the three measures can replace the others. Appendix E specifies their aggregation and selection rules.

4. Approach

The three modules form an ordered pipeline: grounding → staged reward plan → code and verification. Main experiments use GPT-4o for grounding/staging and o1-mini for coding [35] [36]. Figure 2 includes the simulator checks, candidate variants, policy training and human selection loop.

Complete original figure 2
Figure 2: ROSETTA diagram. A trajectory from the latest policy and preference history are given to the Preference Grounding module. The resulting grounded preference is given to the Staging module. The Coding module takes resulting staged reward plan and generates reward variants with various domain knowledge reasserted. Iterative error correction conducted.

4.1 Preference grounding

Images, symbolic object states, the current goal and preference history are converted into state/action descriptions, a demonstration summary, explicit feedback and an updated goal. This supplies context for phrases such as “the other one” and distinguishes a changed goal from dissatisfaction with execution.

The procedure uses privileged simulator state as well as images. It is not a demonstration of grounding arbitrary language solely from raw video. Errors in descriptions or inherited context can propagate into later stages.

4.2 Staging

The grounding output and environment code are turned into atomic intermediate outcomes with prerequisite structure. For example, placing an object in a bin requires approaching, grasping and transporting it with appropriately shaped rewards. Language-space planning lets the reward structure change with the requested task.

Long-horizon staging is restricted to available pick, place and push primitives. Short-horizon staging has explicit prohibitions on release and certain sequential motions. These domain constraints are important to the meaning of “unconstrained” preferences.

4.3 Coding

The coding prompt includes the staged plan, environment code and documentation, desired function headers, and a checklist for geometry, reward density, masking and scale. Continuous-control rewards are generated as code; primitive-based rewards fill a fixed task-independent template.

Verification revisits object-center offsets, target coordinates, penalties and prerequisites. These checks are prompted reviews, not formal proofs of semantic correctness, safety or learnability. The exact code examples, including apparent inconsistencies, remain in the appendix.

4.3 Domain-knowledge verification

Initial code and later verified variants are executed in simulation, and stack traces are passed back for correction. The paper reports 79.35% executable before correction and 95.38% afterward across its main preference set. Executable code can still encode the wrong goal or create an unlearnable reward.

Several variants are produced by reasserting domain knowledge. At most three variants per generation are trained. Consequently the final result includes candidate generation and selection, rather than one unreviewed reward-code sample.

4.4 Policy training and selection

For each trained variant, choose the checkpoint with highest generated-task success; if all checkpoints have zero success, choose by accumulated reward. Show candidate rollouts to the original annotator, who selects the policy used for the next preference.

The number of end-to-end generations is a hyperparameter, while verification determines variants per generation. Hyperparameters are chosen per environment rather than tuned individually for each reward. Reported best-policy success is therefore a selected result, not mean performance over all generated code or training runs.

5. Experimental questions

Experiments ask whether ROSETTA adapts across preference sequences, handles varied language and changing task content, and whether the three evaluation families expose different outcomes. These questions are examined in a small set of manipulation environments and human interactions.

5.1 Training and environments

The main dataset contains 80 preferences in twenty four-step short-horizon chains and 36 preferences in fifteen two- or three-step long-horizon chains. ManiSkill 3 simulates a Franka Panda; PPO trains continuous control and SAC trains a hierarchy over primitives [44] [40] [13].

Five paid annotators generate the chains and judge results. Figure 5 illustrates overlapping language and content labels. Preference count is not participant count, and category bars are not independent samples because one preference may have several labels.

Complete original figure 5
Figure 5: Visualization of diverse human preferences. Each column is one human preference, and each group of columns is a preference sequence. Colored cells indicate applicable categories: orange for language, blue for content. A full list of preference categories can be found in Sec. D.3.

5.1 Baselines

Eureka and Text2Reward are compared after one adaptation round, starting from the original ManiSkill reward. Both use o1-mini for code generation, and humans select from policy videos. Tests include the two main short-horizon scenes and two additional baseline environments.

This evaluates a task-changing, human-scored adaptation protocol rather than reproducing the baselines’ full iterative fixed-goal searches. In particular, the normal externally specified fitness signal is replaced by human assessment; Appendix G’s original-method cost formulas refer to a different iteration budget.

5.1 Ablations and evaluation metrics

Removing grounding passes raw feedback directly to staging; removing staging sends grounded feedback directly to coding; removing follow-up omits domain-knowledge review. Figure 4 preserves all category comparisons.

The no-grounding version can have high optimizability while ignoring the new preference and retaining an easy old reward. Its weaker human alignment supports the need to measure meaning separately. Corrective preferences often reduce differences because the existing success condition remains appropriate.

Complete original figure 4
Figure 4: ROSETTA vs ablations, compared as in Fig 3. ROSETTA outperforms on alignment metrics for all preferences other than corrective. For optimizability, ROSETTA-no-grounding appears superior because ungrounded preferences often result in minimal edits to the prior reward. If it is optimizable, the success rate will be high despite it being misaligned. See Sec. 5.2.4 for analysis.

5.2.1 Adaptation across preference sequences

Table 2 separates short and long horizons. Best-policy success is 89.2% and 82.2%; the fraction with success above 50% is 90.1% and 88.8%. More-satisfied responses are 84.8% and 88.8%, while normalized satisfaction scores are 76.4 and 92.4. These are distinct statistics, not interchangeable definitions of “86% satisfaction.”

The main text reports overall satisfaction score 79.0, which does not equal the simple preference-count-weighted average of the two Table 2 scores. Aggregation is not fully explained. Cascading semantic match is 78.1 and 78.6, with difficult-to-interpret preferences excluded; it is a product of normalized expert ratings, not a percentage of objectively correct programs.

Complete original table 2
Table 2: Experimental results evaluating semantic alignment, optimization potential, and user satisfaction. Semantic match metrics: expert Likert evaluation on grounding quality, staging quality, coding quality, cascading (product of all three). Optimizability metrics: Success rate: average success rate of best policy for a single preference, Pct Success >50%: percent runs with success rate above 50%. Alignment: Satisfaction: % preferences where annotator was satisfied, Satisfaction Score: preference annotator Likert evaluation on how much more/less satisfied (50 is neutral). Note: Cascading Match excludes cases where expert evaluators found the feedback difficult to interpret.

5.2.1 Baseline, model and prompt comparisons

Figure 3 shows the strongest advantage over baselines for preferences changing goals or behavior, with comparable or better baseline results on corrective feedback. The main text’s “33% better” satisfaction claim should be read with the plotted metric and protocol, rather than assumed to be a universal gain.

An exploratory model substitution reports mean best reward 0.741 for Qwen-based grounding/coding versus 0.623 for the proprietary-model configuration. Paraphrased prompts give 0.584 with Gemini and 0.658 with GPT-4o. These are best-reward results, not the main human-satisfaction measure, and the paper does not fully specify their uncertainty or evaluation subset.

Complete original figure 3
Figure 3: ROSETTA vs baselines, compared on optimizability and alignment for goal, behavior, preferential, corrective, and context-dependent preferences. ROSETTA outperforms on all preferences except corrective, as expected given Eureka and Text2Reward are built for the corrective setting.

5.2.2 Diverse language

The system handles several colloquial and contextual expressions, but categories overlap and differ in difficulty. The text reports success/satisfaction of 60%/92% for specific positions and 73%/76% for context-dependent preferences. These selected category results do not establish broad population-level language coverage.

Grounding and staging help convert ambiguous references into concrete reward plans. When the intended meaning itself is unclear, forcing a single interpretation can still create semantic errors rather than resolve the ambiguity.

5.2.3 Compounding preferences

Table 1 reports success 94.6%, 89.4%, 94.4% and 78.1% over four interaction positions, with more-satisfied responses at least 80%. Fourth-step success declines, and difficult requests or inherited code errors can accumulate.

Later positions contain different chain subsets because long-horizon chains have only two or three steps. These row averages should not be treated as a controlled degradation curve for a fixed population. The main text also points to Table 4 for preference ablations, but Table 4 actually contains SAC settings; Figure 4 carries those ablations.

Complete original table 1
Table 1: Breakdown of optimizability and alignment across iterations. Metrics are as described in Sec. 5.1.

5.2.4 Why the metrics differ

Only 66% of human-selected policies have the highest generated-task success among offered options; 73% have success at least 50%. Thus the best-success candidate and the preferred candidate frequently differ.

One example requests pushing a ball next to a bin. A gentle but incomplete push satisfies the person more than high-success pick-and-place behavior. The generated reward simultaneously over-specifies a difficult target point and lets a success bonus outweigh the closed-gripper penalty. This distinguishes semantic interpretation, shaping and human preference.

5.2.5 Real robot experiments

Four two-iteration ThreeCubes preference chains are transferred from simulation using FoundationPose-derived object state and the primitive interface [49]. Mean first- and second-iteration success is 72.5% and 80.0%; Figure 6 preserves the examples.

The authors report no observed policy transfer degradation, conditioned on correct state estimation. This small experiment does not establish zero sim-to-real gap generally, end-to-end visual robustness, or transfer of the continuous-control policies. Trial counts and matched uncertainty are not fully reported.

Complete original figure 6
Figure 6: Example real robot results. The first element in each color column represents a preference given to the policy trained on the original task. The second element represents another iteration of preference given on its parent. We see that simulation-trained policies transfer to real effectively.

6. Limitations and broader impacts

The system depends on simulator-specific state interfaces, code templates, available primitives and domain knowledge. Temporal dependencies, infeasible requests, some releases and multi-objective preferences remain restricted. Biased foundation-model coverage can produce uneven interpretation across users.

Expert code evaluation is costly and subjective; preference collection covers only five people. The study does not provide a formal guarantee against harmful or misleading generated rewards. Broader deployment and simulator transfer remain future work.

7. Conclusion

ROSETTA demonstrates a structured way to revise rewards as a person’s task changes. Its strongest methodological lesson is to separate whether reward code matches the request, whether a policy can optimize it, and whether the resulting behavior satisfies the person.

Acknowledgments

Support includes Stanford HAI, AFOSR YIP FA9550-23-1-0127, ONR N00014-23-1-2355, and ONR MURI grants N00014-22-1-2740 and N00014-24-1-2748.

A. Additional results

Figures 7 and 8 retain the full short- and long-horizon category plots. Some language categories are omitted for legibility, and rare or absent content categories are excluded. Their plotted satisfaction and best-success values support the distinction between alignment and optimizability.

Complete original figure 7
Figure 7: ROSETTA performance on various categories on short-horizon environments. For legibility reasons, we exclude some language categories that foundation models are able to handle without additional domain knowledge, such as 3rd-person POV, 2nd-person POV, no POV, curious tone, directive tone, and suggestive tone. We also exclude conditional and physically impossible feedbacks, which are rare.
Complete original figure 8
Figure 8: ROSETTA performance on various categories on long-horizon environments. For legibility reasons, we exclude some language categories that foundation models are able to handle without additional domain knowledge, such as 3rd-person POV, 2nd-person POV, no POV, curious tone, directive tone, and suggestive tone. We also exclude conditional, inadvertently long-horizon, physically difficult, distraction, multi-part, and physically impossible feedbacks, which didn’t occur in long-horizon experiments.

B. Examples

Appendix B includes successful and failed sequences. All sixteen original trajectory strips are retained with their scenario labels. The four hosted project videos remain available through the original links: introduction, preference sequence, simulation primitives, and real robot primitives. These links require the original host.

B.1 Short-horizon sequence

A four-step sequence changes from pushing rightward to spinning during movement and finally rotating in place. Satisfaction scores are 87.5, 75, 50 and 100. The third attempt illustrates how grounding can retain an unwanted travel goal; the fourth explicitly removes it. Every initial and revised rollout strip remains below.

Original trajectory strip: B.1 initial demonstration: push the ball to the goal
B.1 initial demonstration: push the ball to the goal. Original unnumbered appendix image.
Original trajectory strip: B.1 step 1: push the ball rightward
B.1 step 1: push the ball rightward. Original unnumbered appendix image.
Original trajectory strip: B.1 step 2: spinning motion
B.1 step 2: spinning motion. Original unnumbered appendix image.
Original trajectory strip: B.1 step 3: rotate toward the target
B.1 step 3: rotate toward the target. Original unnumbered appendix image.
Original trajectory strip: B.1 step 4: rotate in place
B.1 step 4: rotate in place. Original unnumbered appendix image.

B.2 Long-horizon example 1

A request to separate light and dark objects is grounded as fruit in one bin and balls in the other. The illustrated result has success 0.9 and satisfaction 100. This is a selected example; the grounding’s semantic grouping is not independently established as the only interpretation of color.

Original trajectory strip: B.2 example 1: previous demonstration
B.2 example 1: previous demonstration. Original unnumbered appendix image.
Original trajectory strip: B.2 example 1: revised sorting
B.2 example 1: revised sorting. Original unnumbered appendix image.

B.2.1 Long-horizon example 2

A request to collect “the other items” extends an apple-only drawer task to include a soup can. The selected example reports success 1.0 and satisfaction 100, with both before and after strips retained.

Original trajectory strip: B.2 example 2: previous apple-only demonstration
B.2 example 2: previous apple-only demonstration. Original unnumbered appendix image.
Original trajectory strip: B.2 example 2: apple and soup can
B.2 example 2: apple and soup can. Original unnumbered appendix image.

B.2.2 Long-horizon example 3

A stacking task changes to pushing one cube beside another and placing a third above them. The example reports success 0.6 and satisfaction 100, again showing that the generated success predicate and human judgment need not coincide.

Original trajectory strip: B.2 example 3: previous stacked cubes
B.2 example 3: previous stacked cubes. Original unnumbered appendix image.
Original trajectory strip: B.2 example 3: revised cube arrangement
B.2 example 3: revised cube arrangement. Original unnumbered appendix image.

B.3.1 Inherited reward-code error

An erroneous target 0.1 m above an apple is copied into the next reward. A previous favorable outcome had hidden the error; the next policy repeatedly fails to lower the gripper enough, with zero success and satisfaction. Both exact code snippets and both rollout strips are preserved.

Complete original code previous offset
Original previous reward-function snippet from Appendix B.3.1. The target is offset 0.1 m above the apple.
Complete original code inherited offset
Original next-step reward-function snippet from Appendix B.3.1. The erroneous target offset persists.
Original trajectory strip: B.3.1 previous successful drawer behavior
B.3.1 previous successful drawer behavior. Original unnumbered appendix image.
Original trajectory strip: B.3.1 failed grasp after inherited offset
B.3.1 failed grasp after inherited offset. Original unnumbered appendix image.

B.3.2 Release and temporal-dependency limits

Requests to release a sphere after positioning it can create a reward conflict when only current state is available: releasing removes the condition that enabled the reward. After repeated failure, the user asks for direct placement, and the system adapts. This is a representational limitation, not merely a wording problem.

Original trajectory strip: B.3.2 step 1: failed release
B.3.2 step 1: failed release. Original unnumbered appendix image.
Original trajectory strip: B.3.2 step 2: repeated release failure
B.3.2 step 2: repeated release failure. Original unnumbered appendix image.
Original trajectory strip: B.3.2 step 3: direct placement
B.3.2 step 3: direct placement. Original unnumbered appendix image.

B.4 Content constraints

Appendix B.4 prohibits history-dependent requirements, discourages long-horizon continuous-control tasks, multiple simultaneous objectives and release, and restricts primitive-based feedback to positions without pulling. Jitter must be addressed explicitly. Some violations were accepted to study failures.

The printed survey additionally says not to request speed changes, despite the main paper illustrating speed preferences. The manuscript does not reconcile these instruction versions. “Unconstrained” therefore describes relatively free language within substantial content and interface limits.

C.1 Short-horizon reward prompts

All six short-horizon prompt groups are retained, including continuations: staging, coding, geometry, target positions, reward design and masking. They specify atomic outcomes, cumulative staged rewards, success bonuses, batched tensors and restricted editable methods.

Original inconsistencies remain visible. One example uses tanh(distance) as a reward for approaching a target despite the earlier 1−tanh(distance) guidance; a later masking illustration discusses release despite an earlier prohibition. These excerpts are the paper’s prompts, not corrected executable implementations.

Original Short Horizon Staging Prompt continuation 1Original Short Horizon Staging Prompt continuation 2Original Short Horizon Staging Prompt continuation 3
Short Horizon Staging Prompt — complete original appendix box, including all continuation pages.
Original Short Horizon Coding Prompt continuation 1Original Short Horizon Coding Prompt continuation 2Original Short Horizon Coding Prompt continuation 3Original Short Horizon Coding Prompt continuation 4
Short Horizon Coding Prompt — complete original appendix box, including all continuation pages.
Original Short Horizon Geometry Review Prompt continuation 1
Short Horizon Geometry Review Prompt — complete original appendix box, including all continuation pages.
Original Short Horizon Target Position Review Prompt continuation 1
Short Horizon Target Position Review Prompt — complete original appendix box, including all continuation pages.
Original Short Horizon Reward Design Review Prompt continuation 1
Short Horizon Reward Design Review Prompt — complete original appendix box, including all continuation pages.
Original Short Horizon Masking Review Prompt continuation 1Original Short Horizon Masking Review Prompt continuation 2
Short Horizon Masking Review Prompt — complete original appendix box, including all continuation pages.

C.2 Long-horizon reward prompts

The six long-horizon groups preserve staging, coding, geometry, normalization, cleanup and the full reward-function template. They constrain actions to pick/place/push, generate target positions and success predicates, normalize position reward to [−1,0], and insert results into the stage-tracking template.

The template gives an affordance reward scaled by five and a success term of ten, with explicit stage advancement. These fixed structures are part of the system’s domain knowledge; the approach does not synthesize an unrestricted controller.

Original Long Horizon Staging Prompt continuation 1Original Long Horizon Staging Prompt continuation 2Original Long Horizon Staging Prompt continuation 3Original Long Horizon Staging Prompt continuation 4
Long Horizon Staging Prompt — complete original appendix box, including all continuation pages.
Original Long Horizon Coding Prompt continuation 1Original Long Horizon Coding Prompt continuation 2Original Long Horizon Coding Prompt continuation 3Original Long Horizon Coding Prompt continuation 4
Long Horizon Coding Prompt — complete original appendix box, including all continuation pages.
Original Long Horizon Reward Function Geometry Review Prompt continuation 1
Long Horizon Reward Function Geometry Review Prompt — complete original appendix box, including all continuation pages.
Original Long Horizon Reward Normalization Review Prompt continuation 1
Long Horizon Reward Normalization Review Prompt — complete original appendix box, including all continuation pages.
Original Long Horizon Reward Function Cleanup Prompt continuation 1
Long Horizon Reward Function Cleanup Prompt — complete original appendix box, including all continuation pages.
Original Long Horizon Reward Function Template continuation 1Original Long Horizon Reward Function Template continuation 2
Long Horizon Reward Function Template — complete original appendix box, including all continuation pages.

C.3.1 Demonstrations to language

Short-horizon videos are sampled every five frames; state and transition descriptions combine images with object coordinates and gripper states. Long-horizon descriptions use primitive start/end states, action type and parameters. All four original state/action description prompts are retained.

Original Short Horizon State Description Prompt continuation 1
Short Horizon State Description Prompt — complete original appendix box, including all continuation pages.
Original Long Horizon State Description Prompt continuation 1
Long Horizon State Description Prompt — complete original appendix box, including all continuation pages.
Original Short Horizon Action Description Prompt continuation 1Original Short Horizon Action Description Prompt continuation 2
Short Horizon Action Description Prompt — complete original appendix box, including all continuation pages.
Original Long Horizon Action Description Prompt continuation 1
Long Horizon Action Description Prompt — complete original appendix box, including all continuation pages.

C.3.2 Grounded preference and task-goal prompts

The grounding prompt summarizes the demonstration and rewrites feedback using explicit object names and contextual interpretation. A separate goal-update prompt changes the task description only when feedback changes the goal. Both full prompts and examples are retained.

The grounding checklist assumes that every referenced object is present in the video. That is a strong dataset assumption, not a general guarantee about natural user requests.

Original Grounded Preference Prompt continuation 1Original Grounded Preference Prompt continuation 2Original Grounded Preference Prompt continuation 3
Grounded Preference Prompt — complete original appendix box, including all continuation pages.
Original Task Goal Update Prompt continuation 1
Task Goal Update Prompt — complete original appendix box, including all continuation pages.

C.4 Environment descriptions and state keys

All five scene descriptions and three state-key lists are retained. They specify dimensions, object-center positions, coordinate directions, drawer geometry and grasp status. These values are essential to translating spatial language into rewards.

Coordinate descriptions differ between PushBall and the shared geometry checklist. The exact texts remain intact; reproducing the system requires checking the intended scene/view convention against the environment implementation.

Original PushBall Setup Description continuation 1Original PushBall Setup Description continuation 2
PushBall Setup Description — complete original appendix box, including all continuation pages.
Original PlaceSphere2BinWide Setup Description continuation 1
PlaceSphere2BinWide Setup Description — complete original appendix box, including all continuation pages.
Original Stack3Cube Setup Description continuation 1
Stack3Cube Setup Description — complete original appendix box, including all continuation pages.
Original Stack3Cube Information Keys continuation 1
Stack3Cube Information Keys — complete original appendix box, including all continuation pages.
Original PutObjectInDrawer Setup Description continuation 1Original PutObjectInDrawer Setup Description continuation 2
PutObjectInDrawer Setup Description — complete original appendix box, including all continuation pages.
Original PutObjectInDrawer Information Keys continuation 1
PutObjectInDrawer Information Keys — complete original appendix box, including all continuation pages.
Original ObjectToBin Setup Description continuation 1
ObjectToBin Setup Description — complete original appendix box, including all continuation pages.
Original ObjectToBin Information Keys continuation 1
ObjectToBin Information Keys — complete original appendix box, including all continuation pages.

C.5 Baseline prompts and documentation

The appendix includes Eureka, Text2Reward and LMPC prompt templates, Text2Reward class descriptions, shared environment documentation, and the complete example environment/reward code embedded in LMPC. All continuations are retained, including typos and placeholder names.

LMPC is included in the prompt appendix but is not a reported main baseline in Figure 3. The preserved prompt should not be interpreted as an additional evaluated method.

Original Eureka Prompt Template continuation 1Original Eureka Prompt Template continuation 2
Eureka Prompt Template — complete original appendix box, including all continuation pages.
Original Text2Reward Prompt Template continuation 1Original Text2Reward Prompt Template continuation 2
Text2Reward Prompt Template — complete original appendix box, including all continuation pages.
Original Text2Reward Environment Class Discription continuation 1Original Text2Reward Environment Class Discription continuation 2
Text2Reward Environment Class Discription — complete original appendix box, including all continuation pages.
Original Documentation For Eureka and Text2Reward continuation 1Original Documentation For Eureka and Text2Reward continuation 2Original Documentation For Eureka and Text2Reward continuation 3
Documentation For Eureka and Text2Reward — complete original appendix box, including all continuation pages.
Original LMPC Prompt Template continuation 1Original LMPC Prompt Template continuation 2Original LMPC Prompt Template continuation 3Original LMPC Prompt Template continuation 4Original LMPC Prompt Template continuation 5Original LMPC Prompt Template continuation 6
LMPC Prompt Template — complete original appendix box, including all continuation pages.

C.6 Error correction prompts

Both error-correction prompts supply generated code, stack trace and failing line, restrict editable functions, and request complete revised definitions. Long-horizon correction also documents tensor-to-array conversion and the skill-reward interface. These are runtime repair steps, distinct from semantic verification.

Original Short Horizon Error Correction Prompt continuation 1Original Short Horizon Error Correction Prompt continuation 2
Short Horizon Error Correction Prompt — complete original appendix box, including all continuation pages.
Original Long Horizon Error Correction Prompt continuation 1
Long Horizon Error Correction Prompt — complete original appendix box, including all continuation pages.

D.1 Human annotators

Five Upwork annotators each follow a preference chain. They watch two- to six-second videos and spend about one to three minutes per form; a four-step chain takes two to four days including turnaround. Payment is $20 for eight new preference/evaluation entries. Expert reviewers volunteer.

The repeated judgments are clustered by person and chain. The paper does not provide a population-level sampling study or a detailed inter-expert agreement analysis.

D.2.1 Preference and alignment surveys

All three complete forms are retained: natural preference collection, quality/satisfaction evaluation of a selected new rollout, and ranking across candidate videos. They ask about incorporated feedback, regressions, unwanted changes and satisfaction relative to the prior behavior.

The evaluator first chooses a favorite video and rates that selected result. This selection is part of the evaluation protocol. Form wording and branching are reproduced exactly, including inconsistencies, rather than silently repaired.

Original Robot Video Preference continuation 1Original Robot Video Preference continuation 2
Robot Video Preference — complete original appendix box, including all continuation pages.
Original Video Quality Evaluation continuation 1Original Video Quality Evaluation continuation 2Original Video Quality Evaluation continuation 3
Video Quality Evaluation — complete original appendix box, including all continuation pages.
Original Preference Evaluation continuation 1Original Preference Evaluation continuation 2
Preference Evaluation — complete original appendix box, including all continuation pages.

D.2.2 Semantic-match survey

The complete expert survey separately rates grounding against raw feedback, staging against grounded feedback, and coding assuming the stage plan is reasonable. It also asks about missing/unnecessary steps, logic errors, sparsity and common-sense mistakes.

Conditional ratings isolate stages but can conceal upstream mistakes unless combined. The cascading product is intended to reflect this dependency; excluding difficult-to-interpret feedback changes its evaluation population.

Original Expert evaluation continuation 1Original Expert evaluation continuation 2
Expert evaluation — complete original appendix box, including all continuation pages.

D.3 Preference taxonomy

Language labels cover point of view, verbosity, colloquiality, context dependence, ambiguity, tone, distraction and multi-part requests. Content labels distinguish correction from changed requirements, goal-related from behavioral preferences, affirmation, contradiction, specific positions/orientations and physical difficulty.

Additional categories mark temporal requirements, multi-objective requests and changes requiring prohibited code edits. Labels can overlap. The source repeats “preferential” in its taxonomy; this duplication does not define a new category.

E. Detailed metrics

“More satisfied” is a binary comparison with the previous rollout. The satisfaction score combines five-point satisfaction/dissatisfaction responses around a neutral midpoint and rescales them; figures/tables present a percentage-like 0–100 form, while Appendix E describes 0–10. A score of 50 is neutral, not 50% success.

Success is measured for the best policy across variants, and the above-50% statistic counts preferences meeting that threshold. Human-selected policy success is a separate result. Semantic scores normalize seven-point expert ratings; their product is the cascading score, typically restricted to interpretable preferences.

F.1 Short-horizon reinforcement learning

PPO uses 1,024 parallel environments, state observations, learning rate 3×10⁻⁴, discount 0.8, GAE 0.9, eight epochs, 32 minibatches and two 256-unit actor/critic layers. Tables 3 and 5 retain every setting. Budgets are 10 million steps for cube picking/pulling, 20 million for ball pushing and 100 million for sphere placement.

The main setup describes operational-space control, whereas Appendix F.1 specifies seven-dimensional pd_joint_delta actions. This controller discrepancy needs clarification for faithful reproduction. Training remains substantial even though preference collection is interactive.

Complete original table 3
Table 3: PPO Hyperparameters
Complete original table 5
Table 5: Short-horizon Environment Setups

F.2 Long-horizon reinforcement learning

SAC selects among pick/place/push plus continuous parameters using a nine-dimensional action representation. State includes object positions, grasp status, gripper position and stage. Tables 4, 6 and 7 preserve all hyperparameters, observations and primitive definitions.

Two 512-unit layers are trained with learning rate 3×10⁻³, discount 0.2 and the printed update coefficient 0.5. Training stops at evaluation success at least 0.9 or 200 million low-level steps. A balanced buffer samples stages evenly. Primitive execution usually takes fewer than 100 low-level actions, with a 250-action timeout.

Complete original table 4
Table 4: SAC Hyperparameters
Complete original table 6
Table 6: Observation Space for Long Horizon Environments
Complete original table 7
Table 7: Parameterized Primitive Skills

G. Cost analysis

Appendix G compares symbolic costs using language generation, code generation, policy training and reflection terms. ROSETTA generates seven variants and trains three; the displayed Text2Reward and Eureka formulas assume their original three- and five-iteration fixed-task protocols, including Eureka’s evolutionary branches. The exact expressions remain below.

These are accounting assumptions, not a measured matched-budget price comparison. Main baseline experiments use a single adaptation round and a common coding model. Reported policy training takes 20–90 minutes for short horizons and 8–12 hours for long horizons on the listed RTX-class hardware, before human turnaround.

Equation
Complete original cost expressions
Concept demonstration

Open image ↗