At a glance
FLaRe fine-tunes a pretrained mobile-manipulation transformer with sparse-reward reinforcement learning in simulation. Stable updates use on-policy optimization, small steps, no entropy bonus, and separate actor and critic networks. Across four simulated task families it averages 79.5% success. Real-robot policy success averages 80.7% across task types, but actual manipulation completion is lower when heuristic grasp failures are included. New tasks and embodiments are learned through additional task-specific simulation training rather than zero-shot instruction following.
I INTRODUCTION
Behavior cloning supplies useful representations and action priors, yet compounding errors can take a policy beyond its demonstrated states. Reinforcement learning can train recovery and task completion, but large pretrained policies can collapse under abrupt updates. FLaRe combines a multitask initialization, diverse simulation, and conservative policy optimization. The contribution is a practical fine-tuning recipe demonstrated on long-horizon household navigation and manipulation.
II Related Work
Related work covers pretrained robot policies and the use of pretrained behavior to initialize reinforcement learning.
II-A Foundation model for robotics
Large robot models commonly learn language-conditioned action prediction from multitask demonstrations. Such policies motivate FLaRe’s initialization strategy. The experiments fine-tune a modified SPOC model; applicability to every other foundation-policy architecture is a proposal rather than a demonstrated result.
II-B RL training and fine-tuning of robotics models
Earlier fine-tuning methods often assume small networks, single tasks, or continuing access to the demonstration dataset. FLaRe uses the pretrained policy without replaying its offline dataset. Its closest comparisons include PIRLNav and JSRL, evaluated here on the larger mobile-manipulation setting.
III Problem Formulation
Each task is a language-conditioned partially observable decision process. Episodes sample an instruction and initial state; a sparse success predicate provides the task reward. The objective is to train a separate task-specific policy from the shared initialization. Specifying a new capability therefore requires executable simulator task logic and a success criterion, even without demonstrations or a shaped reward.
IV Method
The method combines a broadly pretrained policy, large-scale simulated experience, and update choices that preserve useful behavior while learning from task outcomes.
IV-A Fine-tune from a multi-task robotics model
The base SPOC policy [7] is trained on shortest-path expert demonstrations across household tasks. Its representations, temporal architecture, and diverse behavior provide a reusable starting point for individual fine-tuning runs. Novel task and embodiment experiments reuse that initialization, then acquire new behavior through reinforcement learning.
| Success (SEL) | IL+RL: Sparse Reward | IL Only | RL Only | ||||
| FLaRe (Ours) | PIRLNav | JSRL | SPOC | Poliformer - Sparse | Poliformer - Dense | EmbSigLIP - Dense | |
| ObjectNav | 85.0 (67.6) | 20.0 (7.0) | 21.0 (15.6) | 55.0 (42.2) | 14.5 (10.4) | 85.5 (61.2) | 36.5 (24.5) |
| Fetch | 66.9 (54.7) | 0.0 (0.0) | 2.9 (2.8) | 14.0 (10.5) | 0.0 (0.0) | 0.0 (0.0) | 0.0 (0.0) |
| PickUp | 91.8 (90.4) | 0.0 (0.0) | 50.9 (47.7) | 90.1 (86.9) | 0.0 (0.0) | 90.1 (88.7) | 71.9 (52.9) |
| RoomVisit | 70.4 (67.1) | 12.5 (11.0) | 19.0 (18.6) | 40.5 (35.7) | 12.5 (12.5) | 12.5 (10.9) | 16.5 (11.9) |
IV-B Large-scale fine-tuning in simulation
AI2THOR supplies procedurally generated houses and varied objects. Color augmentation, cropping, posterization, and a frozen DINOv2 backbone support transfer to real camera images. Key–value caching reuses past transformer computations during rollout, making long episodes more affordable. Simulation diversity and pretrained visual features are central to the transfer strategy; fine-tuning remains entirely simulated.
IV-C Stabilize RL fine-tuning
FLaRe uses on-policy PPO [58] because simulation makes large rollout counts affordable. The policy takes small gradient steps, removes the entropy bonus that can initially disrupt pretrained behavior, and avoids actor–critic feature sharing. Both networks start from the pretrained transformer; the critic receives a new value head, so value-learning gradients cannot directly damage the actor’s features.
The text specifies a learning rate ten times below 2 × 10⁻⁴, and the ablation treats 2 × 10⁻⁴ as the larger setting. Appendix Table IV nevertheless lists 0.0002. This is an unresolved source inconsistency, retained rather than silently choosing a definitive configuration.
V Results
Experiments examine familiar task types in unseen houses, newly trained task types, physical transfer, embodiment and reward adaptation, and stability ablations. Main task training uses sparse completion rewards; the behavior-shaping experiment deliberately adds penalties.
V-A FLaRe on seen capabilities
On the CHORES simulation benchmark, FLaRe obtains 85.0% ObjectNav, 66.9% Fetch, 91.8% PickUp, and 70.4% RoomVisit success, averaging 79.5%. It exceeds the listed baselines on three task types; dense-reward Poliformer reaches slightly higher ObjectNav success, 85.5%, while FLaRe has better episode-length-weighted success there.
Sparse-reward baselines receive two or three times as many training steps. Dense-reward comparisons use task-specific reward engineering and longer training to convergence. The highlighted fifteen-fold efficiency comparison is 20 million versus 300 million ObjectNav training steps, not a controlled fifteen-fold reduction in total wall-clock cost including pretraining. These tasks were included in the base policy’s demonstrations, while evaluation houses are held out.
V-B FLaRe on novel capabilities
Three newly trained task types ask for a relatively specified object, a room type, or an object with a stated affordance. FLaRe reaches 71.0%, 91.6%, and 79.7% success, respectively. The SPOC++ comparison gets one million additional demonstration frames per new task, while FLaRe gets no new demonstrations but does receive 20–50 million simulator steps and success rewards. This demonstrates adaptation beyond pretraining tasks, not zero-shot mastery of new instructions.
| Success (SEL) | FLaRe (ours) | Poliformer (Sp) | SPOC++ | Poliformer (De) |
| ObjNavRelAttr | 71.0 (63.6) | 6.7 (6.7) | 54.5 (44.6) | 36.1 (32.4) |
| RoomNav | 91.6 (85.6) | 57.0 (51.8) | 74.5 (59.9) | 75.0 (62.4) |
| ObjNavAfford | 79.7 (70.6) | 35.5 (29.4) | 62.4 (50.6) | 53.8 (43.1) |
V-C FLaRe on real robots
Policies transfer directly to a Stretch RE-1 without real-world fine-tuning, with a separate heuristic grasping routine. Forty-six physical episodes cover four task types. Policy success is 94.4% for ObjectNav, 66.7% for Fetch, 86.7% for PickUp, and 75.0% for RoomVisit; their unweighted task-type average is the reported 80.7%.
The manipulation figures outside parentheses describe policy/proximity success. Including heuristic grasping lowers full Fetch success to 55.6% and PickUp to 66.7%. These are small trials in the specified apartment and RoboTHOR setups, rather than an extensive deployment study across homes.
| Success Rate | FLaRe (ours) | SPOC | Poliformer (Dense) |
| ObjectNav | 94.4 | 50.0 | 83.3 |
| Fetch | 66.7 (55.6) | 33.3 (11.1) | X |
| PickUp | 86.7 (66.7) | 66.7 (46.7) | X |
| RoomVisit | 75.0 | 50.0 | X |
V-D FLaRe for adaptation
Additional simulations alter either the robot’s embodiment or the reward objective, testing reuse of pretrained behavior under changed conditions.
V-D 1 Adaptation to New Embodiment
For Locobot, invalid Stretch actions are masked and two action slots are repurposed for camera control. After 30 million fine-tuning steps, ObjectNav success is 72.0% and length-weighted success 47.2, versus 44.0% and 29.7 for sparse-reward Poliformer. A pretrained Locobot Poliformer reaches 57.5% zero-shot, but its training houses and checkpoint budget differ. This is simulated adaptation with an explicitly modified action mapping.
V-D 2 Adaptation to New Behavior
Six hours of further training with a −0.01 step penalty reduces mean Fetch length from 258.2 to 222.8 steps while success changes from 66.9% to 65.7%. A −0.5 collision penalty reduces collisions from 10.0 to 3.1 with success 66.7%. The reward changes trade task efficiency or collision frequency against the original objective; they do not establish collision-free behavior.
| New Behaviors | Success Rate | Episode Length | # of Collisions |
| FLaRe | 66.9 | 258.2 | 10.0 |
| + Step Pen. | 65.7 | 222.8 | 10.0 |
| + Coll. Pen. | 66.7 | 251.2 | 3.1 |
V-E Ablation studies
On Fetch, replacing PPO with SAC, increasing the learning rate, sharing actor–critic transformers, or adding a 0.2 entropy coefficient causes the plotted success curves to collapse. These ablations support the combined recipe in this experiment. They do not imply that all off-policy algorithms or every nonzero entropy setting must fail after suitable tuning.
VI CONCLUSIONS
A capable pretrained policy can be improved and adapted through stable, task-specific simulated reinforcement learning. The principal limitation is dependence on a simulator that supports the task and its success predicate. Liquids, soft objects, and other poorly simulated interactions may require different training infrastructure or real-world fine-tuning.
APPENDIX
The appendix retains hyperparameters, training budgets, benchmark definitions, network details, and the physical evaluation protocol.
VI-A Results Visualization
The project website provides additional rollout videos, behavior examples, and failure analyses. The paper itself does not quantify a separate failure-mode benchmark in this subsection.
VI-B Hyperparameter
Table IV specifies 32 rollouts, 128 steps per PPO update, four update repeats, clipping 0.1, discount 0.99, and zero entropy loss. State encoder and causal decoder each use three layers, 512 hidden dimensions, and eight heads. Its learning-rate entry, 0.0002, conflicts with the main text’s tenfold smaller setting and is preserved as published.
| Training and Model Details | |
| Parameter | Value |
| Total Rollouts | 32 |
| Learning Rate | 0.0002 |
| Mini Batch per Update | 1 |
| Update Repeats | 4 |
| Max Gradient Norm | 0.5 |
| Discount Value Factor | 0.99 |
| GAE | 0.95 |
| PPO Surrogate Objective Clipping | 0.1 |
| Value Loss Weight | 0.5 |
| Entropy Loss Weight | 0.0 |
| Steps for PPO Update | 128 |
| Transformer State Encoder Layers | 3 |
| Transformer State Encoder Hidden Dims | 512 |
| Transformer State Encoder Heads | 8 |
| Causal Transformer Deocder Layers | 3 |
| Causal Transformer Deocder Hidden Dims | 512 |
| Causal Transformer Deocder Heads | 8 |
VI-C Number of Training Steps
The base policy receives 50,000 gradient updates on 100,000 demonstration episodes. FLaRe adds 20 million steps for ObjectNav/RoomVisit, 50 million for Fetch/PickUp, 50 million for relative-attribute and affordance tasks, 20 million for RoomNav, and 30 million for Locobot. These budgets make explicit the substantial online simulation behind each adapted policy.
VI-D CHORES Benchmark
CHORES defines shared RGB observations and discrete robot actions, with separate language instructions, completion conditions, and episode budgets for each task.
VI-D 1 Observation Space
Two orthogonally oriented egocentric RGB views, one for navigation and one for the arm, are resized to 384 × 224. Each episode supplies a natural-language instruction. The learned policy is described as RGB-based rather than consuming depth observations.
VI-D 2 Action Space
Twenty discrete commands move or rotate the base, move the arm, rotate the grasper, pick up or release an object, report a completed subtask, or terminate. This discretization and the grasp routine constrain the control interface; it is not continuous end-to-end dexterous action prediction.
VI-D 3 Tasks Specifications
ObjectNav finds an object category, PickUp grasps a visible object, Fetch combines search and pickup, and RoomVisit explores all rooms. Time limits are 600 steps for the first three and 1,000 for RoomVisit. The appendix states a 10:1 train/test split over 191,568 ProcTHOR houses, ensuring held-out scene configurations.
| Task | Description & Example | Max Steps |
| ObjectNav | Locate an object category: “find a mug” | 600 |
| PickUp | Pick up a specified object in agent line of sight: “pick up a mug” | 600 |
| Fetch | Find and pick up an object: “locate a mug and pick up that mug” | 600 |
| RoomVisit | Traverse the house. “Visit every room in this 5-room house. Indicate when you have seen a new room and when you are done.” | 1000 |
VI-E The SPOC Model
The implementation changes SPOC’s temporal decoder to a Llama 2 decoder for efficiency. Frozen DINOv2 image features and T5 instruction features enter a learned state encoder, followed by a causal temporal model and action head. Figure 7 preserves the complete architecture.
VI-E 1 Vision Transformer Model
DINOv2 [54] generates patch features for both cameras. Projection and a learned camera-type embedding distinguish navigation from manipulation views. The visual backbone stays frozen throughout training to preserve its transferable representation.
VI-E 2 Transformer State Encoder
A noncausal state transformer combines visual tokens, instruction features, and a learned STATE token. The output at that token summarizes the current language-conditioned observation.
VI-E 3 Causal Transformer Decoder
The causal decoder combines state summaries with temporal encodings and previous-action embeddings. Its belief representation integrates observation history to support long-horizon decisions under partial observability.
VI-F Real Robot Setup
The Stretch robot carries two RealSense 455 cameras tilted downward. Camera images are resized and cropped to match training. ObjectNav and Fetch use an unseen six-room apartment, PickUp uses RoboTHOR, and RoomVisit uses both. Despite RGB-D-capable cameras, the learned policy’s input remains the specified RGB images; physical grasping is provided separately.
VI-F 1 ObjectNav
Six object categories are tested from three starting positions, giving eighteen ObjectNav episodes. The targets are sofa, bed, chair, apple, vase, and houseplant.
VI-F 2 Fetch
Fetch uses apple, vase, and houseplant from the same three starts, giving nine episodes. Some instances and heights are adjusted to fit the robot’s grasping envelope. This intervention matters when interpreting manipulation transfer.
VI-F 3 PickUp
Five object categories are placed on three surfaces at different heights, giving fifteen pickup episodes: apple, houseplant, spray bottle, mug, and vase.
VI-F 4 RoomVisit
Exploration is tested in the full six-room apartment, two three-room subdivisions, and a three-room RoboTHOR/workroom arrangement. These four settings complete the forty-six physical evaluation episodes.