Overview
This edition covers all eight pages of arXiv v1 dated 17 March 2026. Pierre Krack, Tobias Jülg, Wolfram Burgard and Florian Walter are affiliated with the University of Technology Nuremberg; Walter also has a Technical University of Munich affiliation. There is no supplementary appendix in this version.
At a glance
Rewarding DINO learns to order robot states by analytic simulator reward using two camera images and a language goal. Frozen vision/text encoders feed a trained scalar head. Mixed expert/random data gives supervision outside successful demonstrations.
Experiments test familiar-task ranking, calibration, paraphrases, a held-out closing task and qualitative real-robot transfer. For policy learning, the score supplies potential-based shaping added to a sparse success reward; it does not replace every source of privileged task feedback.
I Introduction
Demonstration progress or goal-image similarity can reward a particular route rather than task completion. Analytic simulator rewards allow labeling random, failed and successful states, so the model can learn a broader task-conditioned ordering.
The contribution is a supervised reward-ranking recipe with compact pretrained representations and direct comparison against simulator labels. Generality to other preference sources is proposed by the loss formulation; coverage and physical validity remain limited by the chosen simulator rewards and sampled states.
II Related Work
Unlike inverse reinforcement learning from unlabeled demonstrations, this method has explicit reward supervision. Visual-representation methods such as VIP and LIV, success detectors, reward-code generation and video-ranking systems provide related signals but solve different supervision problems.
Rank2Reward, ReWiND, RoboMeter and RoboReward address demonstration coverage in different ways. The source prose calls Rank2Reward “RankNet” and cites RoboReward’s entry for ReWiND; the bibliography is retained unchanged, while these links identify the intended works.
III Method
Given two image embeddings and a goal embedding, predict a deterministic scalar whose ordering matches the analytic reward. The model observes only the current images. Changing the goal changes the scoring function; scores from different goals are not trained onto a common absolute scale.
Equation
III-A Model Architecture
Two 512×512 views reduce occlusion. Frozen DINOv3 ViT-S/16 supplies dense 384-dimensional patch tokens rather than its class token; frozen allMiniLM-L6-v2 supplies the sentence embedding. Four 1×1 projections reduce each image token to four values before flattening.
The reward head lists widths 4096, 512, 64 and 8, layer normalization and LeakyReLU. A goal-conditioned MLP generates FiLM scale/shift vectors for the first three layers. The final output is scalar, although the text does not explicitly specify the final eight-to-one readout. The quoted 21 million parameters describe the image encoder, not the complete trainable head or full model.
For a fixed goal, deterministic scalar scores define a total preorder, allowing ties. No temporal memory resolves visually ambiguous states.
Equation
III-B Pairwise Ranking Loss
Compare two unequal-reward samples, label the preferred one and minimize logistic loss on their score difference with temperature τ. Equal rewards are excluded, so no constraint forces tied states to share a prediction. The exact label and loss equations remain below.
This is RankNet-style preference learning, not regression to absolute rewards. Preserving instantaneous state order alone does not preserve discounted trajectory returns or optimal policies under arbitrary nonlinear score transformations. That is why the later experiment uses potential-based shaping rather than treating the score directly as reward.
Equations 1–2
| (1) |
| (2) |
III-C Data Generation and Sampling
Collect random and expert trajectories; switch an expert to random actions after success and back when the task becomes unsolved. Repeat each sampled random action n times to explore beyond small local motions.
Bin tool-center-point coordinates and normalized reward, retaining one sample per bin. The chosen thresholds are 1 cm and 1% of each task’s reward range, approximately halving the data. Coverage in these four quantities does not guarantee coverage of full scene state, object relations or occlusions.

IV Results
Evaluation measures ordering against newly collected simulator trajectories, pairwise calibration, limited generalization and PPO learning with shaped rewards. These are distinct outcomes: calibration does not establish task success or policy safety.
IV-A Experiment Setup
Data comes from Meta-World+ and a MuJoCo replica of a Franka Research 3 cube-picking setup built with Robot Control Stack and a ManiSkill-inspired reward. One view is gripper-mounted; the physical setup has four additional RealSense cameras.
Adjacent samples are 1/80 s apart in Meta-World+ and 1/30 s in cube picking. Close temporal spacing makes small reward differences hard to infer from images; the policy experiment still uses the benchmark’s original privileged state observations.
IV-A 1 Dataset
Training covers 24 Meta-World+ tasks plus simulated cube picking. Opening/closing variants encourage use of language; drawer-close is held out. Figure 4 preserves the full training-task list.
After binning, 770 trajectories contain 169,018 distinct steps and 809,886 images. More than 900 million step pairs and four billion image pairs are available combinatorially; these are not independent recorded transitions. The action-repeat count n is task-dependent and not tabulated for reproduction.
Equation
IV-A 2 Model Training
Randomize the wrist-camera position, sample a second view and choose prompts from predefined alternatives. Train only unequal-reward pairs with Adam and decoupled weight decay: learning rate 3×10⁻⁴, decay 0.03, temperature 2, batch size 512, and 3,000 epochs of 100,000 pairs each.
This is 300 million sampled pair presentations, potentially repeating observations. The claimed weak hyperparameter sensitivity is not supported here by a full sweep or measured inference-latency/parameter-count ablation.

IV-B Reward Quality Metrics
Because score magnitude is not supervised, assess relative ordering instead of raw reward error. The test trajectories are collected separately; camera/prompt selection and reward-gap filtering affect interpretation.
IV-B 1 Pairwise Accuracy
Pair samples from the same task, possibly different episodes, with matching cameras/prompt and normalized reward difference at least 0.01. The reported task results select the best camera configuration and prompt, so they are optimistic relative to arbitrary deployment choices.
Accuracy exceeds 80% for reward gaps from 0.06 to 0.7, then can fall for larger gaps. A post-success slip can change analytic reward sharply while barely changing appearance; two views do not eliminate this ambiguity.
IV-B 2 Rank Correlation
Figure 6 uses ten expert and ten random trajectories per environment, with tie-aware rank correlation. Expert trajectories are generally easier; random trajectories cluster at small rewards and expose subtle ordering failures.
Equation (3) sums over all i<j pairs, not merely consecutive frames as the surrounding prose incorrectly states. Without ties, τ=0.5 means 75% of all compared pairs are concordant. The printed equation is the no-tie form, while the text says the implementation corrects for ties; exact tie handling is not specified.
Equation 3
| (3) |
IV-C Probabilistic Interpretation
Turn a score difference into a pairwise preference probability using temperature scaling or a monotone isotonic map. Fit calibration on one evaluation split and assess another. Reported binned ECE is 0.043 for temperature scaling and 0.001 for isotonic regression.
These are average bin discrepancies of 4.3 and 0.1 percentage points in the evaluated distribution, not per-example error bounds, absolute-reward calibration or guaranteed uncertainty under domain shift. Figure 7 contains two reliability panels but its caption refers only to temperature scaling; the caption is preserved.
Equation
IV-D Generalization to New Prompts, Tasks, and Domains
The paper tests paraphrased prompts, one related held-out task and simulated-to-real cube-picking observations. These provide different strengths of evidence and should not be treated as broad unseen-task validation.
IV-D 1 Prompt Variation
New paraphrases reduce pairwise accuracy by 0.002 and per-trajectory Kendall correlation by 0.018. Training already includes multiple prompts and a pretrained sentence encoder, so this probes nearby linguistic variation.
IV-D 2 Task Variation
On held-out drawer-close, pairwise accuracy drops from 0.93 to 0.51 and Kendall correlation from 0.82 to 0.47; expert-only correlation drops from 0.94 to 0.72. The mostly binary target reward and many tied random states complicate comparison with dense drawer-open rewards.
The source attributes the decline to ties, but near-chance aggregate accuracy does not establish reliable unseen-task reward prediction. It reports a 0.34 correlation drop although the printed rounded endpoints differ by 0.35. Figure 10 says door-close, while the experiment, depicted drawer and source asset identify drawer-close; the original caption remains unchanged.
IV-D 3 Sim-to-Real
Train on the simulated lab task with varied views, then plot scores on real expert trajectories using a known prompt. The score broadly rises as the cube-picking demonstration progresses. There is no real-world reward ground truth, quantitative ranking evaluation or learned-policy deployment in this experiment.
The visual transfer is encouraging but does not by itself prove semantic invariance or rule out reliance on incidental visual cues.
IV-E Reinforcement Learning
Train PPO from scratch on eight familiar tasks for up to one million steps using Stable-Baselines3 defaults. Policies receive original proprioceptive/object-state observations and Cartesian actions. Evaluate every 50,000 steps; stop when ten deterministic rollouts succeed consecutively.
Use sparse success as base reward plus potential-based shaping, F=γφ(s′)−φ(s), with the image score as potential. Figure 9 compares this with analytic dense reward and sparse reward, including 95% bootstrap intervals. Shaping solves more runs than sparse feedback but less than analytic dense reward; assembly is unsolved. The number of repeated runs and bootstrap unit are not specified.
Policy-invariance claims require the shaping discount, potential/state construction and terminal treatment to meet the theorem’s assumptions. This experiment still depends on a success signal and privileged policy inputs; it is not entirely image-only reinforcement learning.
Equation
IV-F Limitations
Single-frame observations can alias states with different true rewards. The authors therefore caution about Markov/PBRS assumptions and suggest memory or belief-state methods. Partial observability alone does not prove an image-derived potential cannot be a function of full state; the observation process and terminal/discount handling must be assessed explicitly.
Reward ties are omitted and success detection is not trained. A scalar model can mathematically output equal values, but this objective provides no supervision to make it do so. An auxiliary success/comparison head is suggested; memory-based scaling and restored belief-state guarantees remain future work.
V Conclusion and Future Work
Frozen dense visual features support useful language-conditioned reward ranking with simulator supervision. The evidence is strongest for familiar tasks and nearby prompt changes, with weaker held-out-task results and qualitative real-world transfer. Potential shaping improves sparse-reward PPO while retaining success supervision; larger, sequential models are proposed extensions.
Acknowledgment
Support comes from Robotics Institute Germany through BMFTR grant 16ME1006 and Erlangen National HPC Center resources under BayernKI project v106be.