At a glance
ROSETTA turns a person’s changing language preferences into executable reward functions. A grounding module interprets feedback in context, a staging module plans dense intermediate rewards, and a coding module implements and revises the reward. Reinforcement learning then trains candidate policies, from which the person chooses a preferred rollout.
The study covers 116 preferences in 35 chains across five simulation environments, with a smaller real-robot demonstration. Headline success near 87% and satisfaction near 86% measure different things: generated-task optimizability and improvement perceived by the annotator. This edition retains the complete 67-page ICLR 2026 paper, all figures and tables, sixteen unnumbered trajectory strips, and every original prompt, code template and survey.
1. Introduction
Natural feedback can be colloquial, refer to earlier actions, or change the desired task entirely. Optimizing a fixed hand-coded goal is therefore insufficient: the reward’s meaning must change along with the user’s preferences [3] [22].
The paper studies adaptation after each new preference, rather than repeated corrections toward a fixed goal. “Single-step” refers to one preference interaction; internally the pipeline uses multiple model calls, code variants, policy-training runs and human selection. It is not immediate motor adaptation without learning.
3. Problem formulation
At interaction t, a person observes the previous policy and supplies preference h(t). ROSETTA uses the current policy context and preference history to produce reward R(t); a learning algorithm A_M(R(t)) produces the next policy. The objective is the person’s unknown, time-varying satisfaction F(t)(A_M(R(t))).
Unlike a fixed fitness score, this objective is not available for automated reward search. Human judgments are therefore part of the procedure. Selecting the favorite among a finite set of shown candidates maximizes the observed choice within that set, not the unknown global satisfaction objective.
3.1 Evaluation framework
Three views of performance are complementary. Alignment measures the original annotator’s satisfaction with a rollout. Semantic match asks expert roboticists whether grounded language, the stage plan and code preserve the intended meaning. Optimizability measures whether reinforcement learning succeeds at the generated reward’s own success condition.
A policy can optimize a misinterpreted goal perfectly, or fail a strict generated success test while satisfying the person. None of the three measures can replace the others. Appendix E specifies their aggregation and selection rules.
4. Approach
The three modules form an ordered pipeline: grounding → staged reward plan → code and verification. Main experiments use GPT-4o for grounding/staging and o1-mini for coding [35] [36]. Figure 2 includes the simulator checks, candidate variants, policy training and human selection loop.
4.1 Preference grounding
Images, symbolic object states, the current goal and preference history are converted into state/action descriptions, a demonstration summary, explicit feedback and an updated goal. This supplies context for phrases such as “the other one” and distinguishes a changed goal from dissatisfaction with execution.
The procedure uses privileged simulator state as well as images. It is not a demonstration of grounding arbitrary language solely from raw video. Errors in descriptions or inherited context can propagate into later stages.
4.2 Staging
The grounding output and environment code are turned into atomic intermediate outcomes with prerequisite structure. For example, placing an object in a bin requires approaching, grasping and transporting it with appropriately shaped rewards. Language-space planning lets the reward structure change with the requested task.
Long-horizon staging is restricted to available pick, place and push primitives. Short-horizon staging has explicit prohibitions on release and certain sequential motions. These domain constraints are important to the meaning of “unconstrained” preferences.
4.3 Coding
The coding prompt includes the staged plan, environment code and documentation, desired function headers, and a checklist for geometry, reward density, masking and scale. Continuous-control rewards are generated as code; primitive-based rewards fill a fixed task-independent template.
Verification revisits object-center offsets, target coordinates, penalties and prerequisites. These checks are prompted reviews, not formal proofs of semantic correctness, safety or learnability. The exact code examples, including apparent inconsistencies, remain in the appendix.
4.3 Domain-knowledge verification
Initial code and later verified variants are executed in simulation, and stack traces are passed back for correction. The paper reports 79.35% executable before correction and 95.38% afterward across its main preference set. Executable code can still encode the wrong goal or create an unlearnable reward.
Several variants are produced by reasserting domain knowledge. At most three variants per generation are trained. Consequently the final result includes candidate generation and selection, rather than one unreviewed reward-code sample.
4.4 Policy training and selection
For each trained variant, choose the checkpoint with highest generated-task success; if all checkpoints have zero success, choose by accumulated reward. Show candidate rollouts to the original annotator, who selects the policy used for the next preference.
The number of end-to-end generations is a hyperparameter, while verification determines variants per generation. Hyperparameters are chosen per environment rather than tuned individually for each reward. Reported best-policy success is therefore a selected result, not mean performance over all generated code or training runs.
5. Experimental questions
Experiments ask whether ROSETTA adapts across preference sequences, handles varied language and changing task content, and whether the three evaluation families expose different outcomes. These questions are examined in a small set of manipulation environments and human interactions.
5.1 Training and environments
The main dataset contains 80 preferences in twenty four-step short-horizon chains and 36 preferences in fifteen two- or three-step long-horizon chains. ManiSkill 3 simulates a Franka Panda; PPO trains continuous control and SAC trains a hierarchy over primitives [44] [40] [13].
Five paid annotators generate the chains and judge results. Figure 5 illustrates overlapping language and content labels. Preference count is not participant count, and category bars are not independent samples because one preference may have several labels.
5.1 Baselines
Eureka and Text2Reward are compared after one adaptation round, starting from the original ManiSkill reward. Both use o1-mini for code generation, and humans select from policy videos. Tests include the two main short-horizon scenes and two additional baseline environments.
This evaluates a task-changing, human-scored adaptation protocol rather than reproducing the baselines’ full iterative fixed-goal searches. In particular, the normal externally specified fitness signal is replaced by human assessment; Appendix G’s original-method cost formulas refer to a different iteration budget.
5.1 Ablations and evaluation metrics
Removing grounding passes raw feedback directly to staging; removing staging sends grounded feedback directly to coding; removing follow-up omits domain-knowledge review. Figure 4 preserves all category comparisons.
The no-grounding version can have high optimizability while ignoring the new preference and retaining an easy old reward. Its weaker human alignment supports the need to measure meaning separately. Corrective preferences often reduce differences because the existing success condition remains appropriate.
5.2.1 Adaptation across preference sequences
Table 2 separates short and long horizons. Best-policy success is 89.2% and 82.2%; the fraction with success above 50% is 90.1% and 88.8%. More-satisfied responses are 84.8% and 88.8%, while normalized satisfaction scores are 76.4 and 92.4. These are distinct statistics, not interchangeable definitions of “86% satisfaction.”
The main text reports overall satisfaction score 79.0, which does not equal the simple preference-count-weighted average of the two Table 2 scores. Aggregation is not fully explained. Cascading semantic match is 78.1 and 78.6, with difficult-to-interpret preferences excluded; it is a product of normalized expert ratings, not a percentage of objectively correct programs.
5.2.1 Baseline, model and prompt comparisons
Figure 3 shows the strongest advantage over baselines for preferences changing goals or behavior, with comparable or better baseline results on corrective feedback. The main text’s “33% better” satisfaction claim should be read with the plotted metric and protocol, rather than assumed to be a universal gain.
An exploratory model substitution reports mean best reward 0.741 for Qwen-based grounding/coding versus 0.623 for the proprietary-model configuration. Paraphrased prompts give 0.584 with Gemini and 0.658 with GPT-4o. These are best-reward results, not the main human-satisfaction measure, and the paper does not fully specify their uncertainty or evaluation subset.
5.2.2 Diverse language
The system handles several colloquial and contextual expressions, but categories overlap and differ in difficulty. The text reports success/satisfaction of 60%/92% for specific positions and 73%/76% for context-dependent preferences. These selected category results do not establish broad population-level language coverage.
Grounding and staging help convert ambiguous references into concrete reward plans. When the intended meaning itself is unclear, forcing a single interpretation can still create semantic errors rather than resolve the ambiguity.
5.2.3 Compounding preferences
Table 1 reports success 94.6%, 89.4%, 94.4% and 78.1% over four interaction positions, with more-satisfied responses at least 80%. Fourth-step success declines, and difficult requests or inherited code errors can accumulate.
Later positions contain different chain subsets because long-horizon chains have only two or three steps. These row averages should not be treated as a controlled degradation curve for a fixed population. The main text also points to Table 4 for preference ablations, but Table 4 actually contains SAC settings; Figure 4 carries those ablations.
5.2.4 Why the metrics differ
Only 66% of human-selected policies have the highest generated-task success among offered options; 73% have success at least 50%. Thus the best-success candidate and the preferred candidate frequently differ.
One example requests pushing a ball next to a bin. A gentle but incomplete push satisfies the person more than high-success pick-and-place behavior. The generated reward simultaneously over-specifies a difficult target point and lets a success bonus outweigh the closed-gripper penalty. This distinguishes semantic interpretation, shaping and human preference.
5.2.5 Real robot experiments
Four two-iteration ThreeCubes preference chains are transferred from simulation using FoundationPose-derived object state and the primitive interface [49]. Mean first- and second-iteration success is 72.5% and 80.0%; Figure 6 preserves the examples.
The authors report no observed policy transfer degradation, conditioned on correct state estimation. This small experiment does not establish zero sim-to-real gap generally, end-to-end visual robustness, or transfer of the continuous-control policies. Trial counts and matched uncertainty are not fully reported.
6. Limitations and broader impacts
The system depends on simulator-specific state interfaces, code templates, available primitives and domain knowledge. Temporal dependencies, infeasible requests, some releases and multi-objective preferences remain restricted. Biased foundation-model coverage can produce uneven interpretation across users.
Expert code evaluation is costly and subjective; preference collection covers only five people. The study does not provide a formal guarantee against harmful or misleading generated rewards. Broader deployment and simulator transfer remain future work.
7. Conclusion
ROSETTA demonstrates a structured way to revise rewards as a person’s task changes. Its strongest methodological lesson is to separate whether reward code matches the request, whether a policy can optimize it, and whether the resulting behavior satisfies the person.
Acknowledgments
Support includes Stanford HAI, AFOSR YIP FA9550-23-1-0127, ONR N00014-23-1-2355, and ONR MURI grants N00014-22-1-2740 and N00014-24-1-2748.
A. Additional results
Figures 7 and 8 retain the full short- and long-horizon category plots. Some language categories are omitted for legibility, and rare or absent content categories are excluded. Their plotted satisfaction and best-success values support the distinction between alignment and optimizability.
B. Examples
Appendix B includes successful and failed sequences. All sixteen original trajectory strips are retained with their scenario labels. The four hosted project videos remain available through the original links: introduction, preference sequence, simulation primitives, and real robot primitives. These links require the original host.
B.1 Short-horizon sequence
A four-step sequence changes from pushing rightward to spinning during movement and finally rotating in place. Satisfaction scores are 87.5, 75, 50 and 100. The third attempt illustrates how grounding can retain an unwanted travel goal; the fourth explicitly removes it. Every initial and revised rollout strip remains below.





B.2 Long-horizon example 1
A request to separate light and dark objects is grounded as fruit in one bin and balls in the other. The illustrated result has success 0.9 and satisfaction 100. This is a selected example; the grounding’s semantic grouping is not independently established as the only interpretation of color.


B.2.1 Long-horizon example 2
A request to collect “the other items” extends an apple-only drawer task to include a soup can. The selected example reports success 1.0 and satisfaction 100, with both before and after strips retained.


B.2.2 Long-horizon example 3
A stacking task changes to pushing one cube beside another and placing a third above them. The example reports success 0.6 and satisfaction 100, again showing that the generated success predicate and human judgment need not coincide.


B.3.1 Inherited reward-code error
An erroneous target 0.1 m above an apple is copied into the next reward. A previous favorable outcome had hidden the error; the next policy repeatedly fails to lower the gripper enough, with zero success and satisfaction. Both exact code snippets and both rollout strips are preserved.


B.3.2 Release and temporal-dependency limits
Requests to release a sphere after positioning it can create a reward conflict when only current state is available: releasing removes the condition that enabled the reward. After repeated failure, the user asks for direct placement, and the system adapts. This is a representational limitation, not merely a wording problem.



B.4 Content constraints
Appendix B.4 prohibits history-dependent requirements, discourages long-horizon continuous-control tasks, multiple simultaneous objectives and release, and restricts primitive-based feedback to positions without pulling. Jitter must be addressed explicitly. Some violations were accepted to study failures.
The printed survey additionally says not to request speed changes, despite the main paper illustrating speed preferences. The manuscript does not reconcile these instruction versions. “Unconstrained” therefore describes relatively free language within substantial content and interface limits.
C.1 Short-horizon reward prompts
All six short-horizon prompt groups are retained, including continuations: staging, coding, geometry, target positions, reward design and masking. They specify atomic outcomes, cumulative staged rewards, success bonuses, batched tensors and restricted editable methods.
Original inconsistencies remain visible. One example uses tanh(distance) as a reward for approaching a target despite the earlier 1−tanh(distance) guidance; a later masking illustration discusses release despite an earlier prohibition. These excerpts are the paper’s prompts, not corrected executable implementations.
C.2 Long-horizon reward prompts
The six long-horizon groups preserve staging, coding, geometry, normalization, cleanup and the full reward-function template. They constrain actions to pick/place/push, generate target positions and success predicates, normalize position reward to [−1,0], and insert results into the stage-tracking template.
The template gives an affordance reward scaled by five and a success term of ten, with explicit stage advancement. These fixed structures are part of the system’s domain knowledge; the approach does not synthesize an unrestricted controller.
C.3.1 Demonstrations to language
Short-horizon videos are sampled every five frames; state and transition descriptions combine images with object coordinates and gripper states. Long-horizon descriptions use primitive start/end states, action type and parameters. All four original state/action description prompts are retained.
C.3.2 Grounded preference and task-goal prompts
The grounding prompt summarizes the demonstration and rewrites feedback using explicit object names and contextual interpretation. A separate goal-update prompt changes the task description only when feedback changes the goal. Both full prompts and examples are retained.
The grounding checklist assumes that every referenced object is present in the video. That is a strong dataset assumption, not a general guarantee about natural user requests.
C.4 Environment descriptions and state keys
All five scene descriptions and three state-key lists are retained. They specify dimensions, object-center positions, coordinate directions, drawer geometry and grasp status. These values are essential to translating spatial language into rewards.
Coordinate descriptions differ between PushBall and the shared geometry checklist. The exact texts remain intact; reproducing the system requires checking the intended scene/view convention against the environment implementation.
C.5 Baseline prompts and documentation
The appendix includes Eureka, Text2Reward and LMPC prompt templates, Text2Reward class descriptions, shared environment documentation, and the complete example environment/reward code embedded in LMPC. All continuations are retained, including typos and placeholder names.
LMPC is included in the prompt appendix but is not a reported main baseline in Figure 3. The preserved prompt should not be interpreted as an additional evaluated method.
C.6 Error correction prompts
Both error-correction prompts supply generated code, stack trace and failing line, restrict editable functions, and request complete revised definitions. Long-horizon correction also documents tensor-to-array conversion and the skill-reward interface. These are runtime repair steps, distinct from semantic verification.
D.1 Human annotators
Five Upwork annotators each follow a preference chain. They watch two- to six-second videos and spend about one to three minutes per form; a four-step chain takes two to four days including turnaround. Payment is $20 for eight new preference/evaluation entries. Expert reviewers volunteer.
The repeated judgments are clustered by person and chain. The paper does not provide a population-level sampling study or a detailed inter-expert agreement analysis.
D.2.1 Preference and alignment surveys
All three complete forms are retained: natural preference collection, quality/satisfaction evaluation of a selected new rollout, and ranking across candidate videos. They ask about incorporated feedback, regressions, unwanted changes and satisfaction relative to the prior behavior.
The evaluator first chooses a favorite video and rates that selected result. This selection is part of the evaluation protocol. Form wording and branching are reproduced exactly, including inconsistencies, rather than silently repaired.
D.2.2 Semantic-match survey
The complete expert survey separately rates grounding against raw feedback, staging against grounded feedback, and coding assuming the stage plan is reasonable. It also asks about missing/unnecessary steps, logic errors, sparsity and common-sense mistakes.
Conditional ratings isolate stages but can conceal upstream mistakes unless combined. The cascading product is intended to reflect this dependency; excluding difficult-to-interpret feedback changes its evaluation population.
D.3 Preference taxonomy
Language labels cover point of view, verbosity, colloquiality, context dependence, ambiguity, tone, distraction and multi-part requests. Content labels distinguish correction from changed requirements, goal-related from behavioral preferences, affirmation, contradiction, specific positions/orientations and physical difficulty.
Additional categories mark temporal requirements, multi-objective requests and changes requiring prohibited code edits. Labels can overlap. The source repeats “preferential” in its taxonomy; this duplication does not define a new category.
E. Detailed metrics
“More satisfied” is a binary comparison with the previous rollout. The satisfaction score combines five-point satisfaction/dissatisfaction responses around a neutral midpoint and rescales them; figures/tables present a percentage-like 0–100 form, while Appendix E describes 0–10. A score of 50 is neutral, not 50% success.
Success is measured for the best policy across variants, and the above-50% statistic counts preferences meeting that threshold. Human-selected policy success is a separate result. Semantic scores normalize seven-point expert ratings; their product is the cascading score, typically restricted to interpretable preferences.
F.1 Short-horizon reinforcement learning
PPO uses 1,024 parallel environments, state observations, learning rate 3×10⁻⁴, discount 0.8, GAE 0.9, eight epochs, 32 minibatches and two 256-unit actor/critic layers. Tables 3 and 5 retain every setting. Budgets are 10 million steps for cube picking/pulling, 20 million for ball pushing and 100 million for sphere placement.
The main setup describes operational-space control, whereas Appendix F.1 specifies seven-dimensional pd_joint_delta actions. This controller discrepancy needs clarification for faithful reproduction. Training remains substantial even though preference collection is interactive.
F.2 Long-horizon reinforcement learning
SAC selects among pick/place/push plus continuous parameters using a nine-dimensional action representation. State includes object positions, grasp status, gripper position and stage. Tables 4, 6 and 7 preserve all hyperparameters, observations and primitive definitions.
Two 512-unit layers are trained with learning rate 3×10⁻³, discount 0.2 and the printed update coefficient 0.5. Training stops at evaluation success at least 0.9 or 200 million low-level steps. A balanced buffer samples stages evenly. Primitive execution usually takes fewer than 100 low-level actions, with a 250-action timeout.
G. Cost analysis
Appendix G compares symbolic costs using language generation, code generation, policy training and reflection terms. ROSETTA generates seven variants and trains three; the displayed Text2Reward and Eureka formulas assume their original three- and five-iteration fixed-task protocols, including Eureka’s evolutionary branches. The exact expressions remain below.
These are accounting assumptions, not a measured matched-budget price comparison. Main baseline experiments use a single adaptation round and a common coding model. Reported policy training takes 20–90 minutes for short horizons and 8–12 hours for long horizons on the listed RTX-class hardware, before human turnaround.