Literature
Display
← All papers

FLaRe: Achieving Masterful and Adaptive Robot Policies with Large-Scale Reinforcement Learning Fine-Tuning

Jiaheng Hu · Rose Hendrix · Ali Farhadi · Aniruddha Kembhavi · Roberto Martín-Martín · Peter Stone · Kuo-Hao Zeng · Kiana Ehsani

· ICRA 2025 ↗

Robotics8 min text · CondensedOriginal paper ↗
Contents

At a glance

FLaRe fine-tunes a pretrained mobile-manipulation transformer with sparse-reward reinforcement learning in simulation. Stable updates use on-policy optimization, small steps, no entropy bonus, and separate actor and critic networks. Across four simulated task families it averages 79.5% success. Real-robot policy success averages 80.7% across task types, but actual manipulation completion is lower when heuristic grasp failures are included. New tasks and embodiments are learned through additional task-specific simulation training rather than zero-shot instruction following.

I INTRODUCTION

Behavior cloning supplies useful representations and action priors, yet compounding errors can take a policy beyond its demonstrated states. Reinforcement learning can train recovery and task completion, but large pretrained policies can collapse under abrupt updates. FLaRe combines a multitask initialization, diverse simulation, and conservative policy optimization. The contribution is a practical fine-tuning recipe demonstrated on long-horizon household navigation and manipulation.

Refer to caption
Fig. 1: FLaRe is a simple but effective approach for large-scale fine-tuning of robotic policies. FLaRe achieves SoTA performance on simulation (+23.6%) and real-world (+30.7%) benchmarks, can generalize to unseen tasks, and adapts to new behaviors and embodiments.

II Related Work

Related work covers pretrained robot policies and the use of pretrained behavior to initialize reinforcement learning.

II-A Foundation model for robotics

Large robot models commonly learn language-conditioned action prediction from multitask demonstrations. Such policies motivate FLaRe’s initialization strategy. The experiments fine-tune a modified SPOC model; applicability to every other foundation-policy architecture is a proposal rather than a demonstrated result.

II-B RL training and fine-tuning of robotics models

Earlier fine-tuning methods often assume small networks, single tasks, or continuing access to the demonstration dataset. FLaRe uses the pretrained policy without replaying its offline dataset. Its closest comparisons include PIRLNav and JSRL, evaluated here on the larger mobile-manipulation setting.

III Problem Formulation

Each task is a language-conditioned partially observable decision process. Episodes sample an instruction and initial state; a sparse success predicate provides the task reward. The objective is to train a separate task-specific policy from the shared initialization. Specifying a new capability therefore requires executable simulator task logic and a success criterion, even without demonstrations or a shaped reward.

Refer to caption
Fig. 2: FLaRe introduces a series of design choices that help stabilize the RL training process, including 1) fine-tuning from a multi-task robotics policy, 2) large-scale fine-tuning in simulation, 3) using an on-policy algorithm as opposed to off-policy methods, 4) utilizing smaller learning rate than when performing RL from scratch, 5) disabling the entropy bonus objective that can potentially distort the policy at the start of the training, and 6) separating the actor and the critic network, so that the critic update will not influence the policy prediction.

IV Method

The method combines a broadly pretrained policy, large-scale simulated experience, and update choices that preserve useful behavior while learning from task outcomes.

Refer to caption
Fig. 3: FLaRe can efficiently fine-tune large transformer policies through large-scale Reinforcement Learning, using a sparse reward function that requires minimal human effort.

IV-A Fine-tune from a multi-task robotics model

The base SPOC policy [7] is trained on shortest-path expert demonstrations across household tasks. Its representations, temporal architecture, and diverse behavior provide a reusable starting point for individual fine-tuning runs. Novel task and embodiment experiments reuse that initialization, then acquire new behavior through reinforcement learning.

TABLE I: Success and Episode-length weighted Success (SEL) against baseline methods on the CHORES[7] benchmark. Baselines with privileged information are marked in blue. FLaRe significantly outperforms the previous SoTA methods.
Success (SEL) IL+RL: Sparse Reward IL Only RL Only
FLaRe (Ours) PIRLNav JSRL SPOC Poliformer - Sparse Poliformer - Dense EmbSigLIP - Dense
ObjectNav 85.0 (67.6) 20.0 (7.0) 21.0 (15.6) 55.0 (42.2) 14.5 (10.4) 85.5 (61.2) 36.5 (24.5)
Fetch 66.9 (54.7) 0.0 (0.0) 2.9 (2.8) 14.0 (10.5) 0.0 (0.0) 0.0 (0.0) 0.0 (0.0)
PickUp 91.8 (90.4) 0.0 (0.0) 50.9 (47.7) 90.1 (86.9) 0.0 (0.0) 90.1 (88.7) 71.9 (52.9)
RoomVisit 70.4 (67.1) 12.5 (11.0) 19.0 (18.6) 40.5 (35.7) 12.5 (12.5) 12.5 (10.9) 16.5 (11.9)

IV-B Large-scale fine-tuning in simulation

AI2THOR supplies procedurally generated houses and varied objects. Color augmentation, cropping, posterization, and a frozen DINOv2 backbone support transfer to real camera images. Key–value caching reuses past transformer computations during rollout, making long episodes more affordable. Simulation diversity and pretrained visual features are central to the transfer strategy; fine-tuning remains entirely simulated.

IV-C Stabilize RL fine-tuning

FLaRe uses on-policy PPO [58] because simulation makes large rollout counts affordable. The policy takes small gradient steps, removes the entropy bonus that can initially disrupt pretrained behavior, and avoids actor–critic feature sharing. Both networks start from the pretrained transformer; the critic receives a new value head, so value-learning gradients cannot directly damage the actor’s features.

The text specifies a learning rate ten times below 2 × 10⁻⁴, and the ablation treats 2 × 10⁻⁴ as the larger setting. Appendix Table IV nevertheless lists 0.0002. This is an unresolved source inconsistency, retained rather than silently choosing a definitive configuration.

Refer to caption
(a) Fetch Task
Refer to caption
(b) RoomVisit Task
Refer to caption
(c) ObjNavRelAttr Task
Refer to caption
(d) ObjNavAfford Task
Fig. 4: We evaluate FLaRe on mobile manipulation tasks. (a, b) In-distribution tasks, in unseen environments. (c, d) Novel tasks that require unseen capabilities from pretraining, in unseen environments. FLaRe excels in long-horizon tasks, showing strong object recognition, relational reasoning, and exploration abilities.

V Results

Experiments examine familiar task types in unseen houses, newly trained task types, physical transfer, embodiment and reward adaptation, and stability ablations. Main task training uses sparse completion rewards; the behavior-shaping experiment deliberately adds penalties.

V-A FLaRe on seen capabilities

On the CHORES simulation benchmark, FLaRe obtains 85.0% ObjectNav, 66.9% Fetch, 91.8% PickUp, and 70.4% RoomVisit success, averaging 79.5%. It exceeds the listed baselines on three task types; dense-reward Poliformer reaches slightly higher ObjectNav success, 85.5%, while FLaRe has better episode-length-weighted success there.

Sparse-reward baselines receive two or three times as many training steps. Dense-reward comparisons use task-specific reward engineering and longer training to convergence. The highlighted fifteen-fold efficiency comparison is 20 million versus 300 million ObjectNav training steps, not a controlled fifteen-fold reduction in total wall-clock cost including pretraining. These tasks were included in the base policy’s demonstrations, while evaluation houses are held out.

V-B FLaRe on novel capabilities

Three newly trained task types ask for a relatively specified object, a room type, or an object with a stated affordance. FLaRe reaches 71.0%, 91.6%, and 79.7% success, respectively. The SPOC++ comparison gets one million additional demonstration frames per new task, while FLaRe gets no new demonstrations but does receive 20–50 million simulator steps and success rewards. This demonstrates adaptation beyond pretraining tasks, not zero-shot mastery of new instructions.

TABLE II: FLaRe can fine-tune for tasks that are never seen by the base model, and achieve state-of-the-art performance. Baselines with privileged information are marked in blue.
Success (SEL) FLaRe (ours) Poliformer (Sp) SPOC++ Poliformer (De)
ObjNavRelAttr 71.0 (63.6) 6.7 (6.7) 54.5 (44.6) 36.1 (32.4)
RoomNav 91.6 (85.6) 57.0 (51.8) 74.5 (59.9) 75.0 (62.4)
ObjNavAfford 79.7 (70.6) 35.5 (29.4) 62.4 (50.6) 53.8 (43.1)

V-C FLaRe on real robots

Policies transfer directly to a Stretch RE-1 without real-world fine-tuning, with a separate heuristic grasping routine. Forty-six physical episodes cover four task types. Policy success is 94.4% for ObjectNav, 66.7% for Fetch, 86.7% for PickUp, and 75.0% for RoomVisit; their unweighted task-type average is the reported 80.7%.

The manipulation figures outside parentheses describe policy/proximity success. Including heuristic grasping lowers full Fetch success to 55.6% and PickUp to 66.7%. These are small trials in the specified apartment and RoboTHOR setups, rather than an extensive deployment study across homes.

TABLE III: Real-world results (total of 46 tasks). For manipulation tasks, we report both full success (policy and heuristic grasping) and policy success (proximity) following [7].
Success Rate FLaRe (ours) SPOC Poliformer (Dense)
ObjectNav 94.4 50.0 83.3
Fetch 66.7 (55.6) 33.3 (11.1) X
PickUp 86.7 (66.7) 66.7 (46.7) X
RoomVisit 75.0 50.0 X
Refer to caption
Fig. 5: The real-world layout that we tested upon

V-D FLaRe for adaptation

Additional simulations alter either the robot’s embodiment or the reward objective, testing reuse of pretrained behavior under changed conditions.

V-D 1 Adaptation to New Embodiment

For Locobot, invalid Stretch actions are masked and two action slots are repurposed for camera control. After 30 million fine-tuning steps, ObjectNav success is 72.0% and length-weighted success 47.2, versus 44.0% and 29.7 for sparse-reward Poliformer. A pretrained Locobot Poliformer reaches 57.5% zero-shot, but its training houses and checkpoint budget differ. This is simulated adaptation with an explicitly modified action mapping.

New Embodiment Success Rate ↑\uparrow SEL ↑\uparrow
FLaRe 72.0 47.2
Poliformer zero-shot22 2 We zero-shot evaluate Poliformer [10] 400400M ckpt trained with LoCoBot. This baseline was trained in ProcTHOR-10k, instead of ObjaTHOR houses. 57.5 30.1
Poliformer (Sparse Reward) 44.0 29.7

V-D 2 Adaptation to New Behavior

Six hours of further training with a −0.01 step penalty reduces mean Fetch length from 258.2 to 222.8 steps while success changes from 66.9% to 65.7%. A −0.5 collision penalty reduces collisions from 10.0 to 3.1 with success 66.7%. The reward changes trade task efficiency or collision frequency against the original objective; they do not establish collision-free behavior.

Original figure
(a) Baselines
Original figure
(b) Ablations
Fig. 6: Baseline performances and ablation studies on the Fetch task. FLaRe is the only method that can achieve good performance on this challenging task.
New Behaviors Success Rate ↑\uparrow Episode Length ↓\downarrow # of Collisions ↓\downarrow
FLaRe 66.9 258.2 10.0
+ Step Pen. 65.7 222.8 10.0
+ Coll. Pen. 66.7 251.2 3.1

V-E Ablation studies

On Fetch, replacing PPO with SAC, increasing the learning rate, sharing actor–critic transformers, or adding a 0.2 entropy coefficient causes the plotted success curves to collapse. These ablations support the combined recipe in this experiment. They do not imply that all off-policy algorithms or every nonzero entropy setting must fail after suitable tuning.

VI CONCLUSIONS

A capable pretrained policy can be improved and adapted through stable, task-specific simulated reinforcement learning. The principal limitation is dependence on a simulator that supports the task and its success predicate. Liquids, soft objects, and other poorly simulated interactions may require different training infrastructure or real-world fine-tuning.

APPENDIX

The appendix retains hyperparameters, training budgets, benchmark definitions, network details, and the physical evaluation protocol.

VI-A Results Visualization

The project website provides additional rollout videos, behavior examples, and failure analyses. The paper itself does not quantify a separate failure-mode benchmark in this subsection.

VI-B Hyperparameter

Table IV specifies 32 rollouts, 128 steps per PPO update, four update repeats, clipping 0.1, discount 0.99, and zero entropy loss. State encoder and causal decoder each use three layers, 512 hidden dimensions, and eight heads. Its learning-rate entry, 0.0002, conflicts with the main text’s tenfold smaller setting and is preserved as published.

Training and Model Details
Parameter Value
Total Rollouts 32
Learning Rate 0.0002
Mini Batch per Update 1
Update Repeats 4
Max Gradient Norm 0.5
Discount Value Factor γ\gamma 0.99
GAE λ\lambda 0.95
PPO Surrogate Objective Clipping 0.1
Value Loss Weight 0.5
Entropy Loss Weight 0.0
Steps for PPO Update 128
Transformer State Encoder Layers 3
Transformer State Encoder Hidden Dims 512
Transformer State Encoder Heads 8
Causal Transformer Deocder Layers 3
Causal Transformer Deocder Hidden Dims 512
Causal Transformer Deocder Heads 8
TABLE IV: Hyperparameters for training and model architecture. We use AllenAct [65] to implement our models and conduct experiments.

VI-C Number of Training Steps

The base policy receives 50,000 gradient updates on 100,000 demonstration episodes. FLaRe adds 20 million steps for ObjectNav/RoomVisit, 50 million for Fetch/PickUp, 50 million for relative-attribute and affordance tasks, 20 million for RoomNav, and 30 million for Locobot. These budgets make explicit the substantial online simulation behind each adapted policy.

VI-D CHORES Benchmark

CHORES defines shared RGB observations and discrete robot actions, with separate language instructions, completion conditions, and episode budgets for each task.

VI-D 1 Observation Space

Two orthogonally oriented egocentric RGB views, one for navigation and one for the arm, are resized to 384 × 224. Each episode supplies a natural-language instruction. The learned policy is described as RGB-based rather than consuming depth observations.

VI-D 2 Action Space

Twenty discrete commands move or rotate the base, move the arm, rotate the grasper, pick up or release an object, report a completed subtask, or terminate. This discretization and the grasp routine constrain the control interface; it is not continuous end-to-end dexterous action prediction.

VI-D 3 Tasks Specifications

ObjectNav finds an object category, PickUp grasps a visible object, Fetch combines search and pickup, and RoomVisit explores all rooms. Time limits are 600 steps for the first three and 1,000 for RoomVisit. The appendix states a 10:1 train/test split over 191,568 ProcTHOR houses, ensuring held-out scene configurations.

Task Description & Example Max Steps
ObjectNav Locate an object category: “find a mug” 600
PickUp Pick up a specified object in agent line of sight: “pick up a mug” 600
Fetch Find and pick up an object: “locate a mug and pick up that mug” 600
RoomVisit Traverse the house. “Visit every room in this 5-room house. Indicate when you have seen a new room and when you are done.” 1000
TABLE V: CHORES tasks.
Refer to caption
Fig. 7: A visualization of the network architecture of the transformer-based SPOC model that FLaRe fine-tunes upon.

VI-E The SPOC Model

The implementation changes SPOC’s temporal decoder to a Llama 2 decoder for efficiency. Frozen DINOv2 image features and T5 instruction features enter a learned state encoder, followed by a causal temporal model and action head. Figure 7 preserves the complete architecture.

VI-E 1 Vision Transformer Model

DINOv2 [54] generates patch features for both cameras. Projection and a learned camera-type embedding distinguish navigation from manipulation views. The visual backbone stays frozen throughout training to preserve its transferable representation.

VI-E 2 Transformer State Encoder

A noncausal state transformer combines visual tokens, instruction features, and a learned STATE token. The output at that token summarizes the current language-conditioned observation.

VI-E 3 Causal Transformer Decoder

The causal decoder combines state summaries with temporal encodings and previous-action embeddings. Its belief representation integrates observation history to support long-horizon decisions under partial observability.

VI-F Real Robot Setup

The Stretch robot carries two RealSense 455 cameras tilted downward. Camera images are resized and cropped to match training. ObjectNav and Fetch use an unseen six-room apartment, PickUp uses RoboTHOR, and RoomVisit uses both. Despite RGB-D-capable cameras, the learned policy’s input remains the specified RGB images; physical grasping is provided separately.

VI-F 1 ObjectNav

Six object categories are tested from three starting positions, giving eighteen ObjectNav episodes. The targets are sofa, bed, chair, apple, vase, and houseplant.

VI-F 2 Fetch

Fetch uses apple, vase, and houseplant from the same three starts, giving nine episodes. Some instances and heights are adjusted to fit the robot’s grasping envelope. This intervention matters when interpreting manipulation transfer.

VI-F 3 PickUp

Five object categories are placed on three surfaces at different heights, giving fifteen pickup episodes: apple, houseplant, spray bottle, mug, and vase.

VI-F 4 RoomVisit

Exploration is tested in the full six-room apartment, two three-room subdivisions, and a three-room RoboTHOR/workroom arrangement. These four settings complete the forty-six physical evaluation episodes.

Concept demonstration

Open image ↗