Overview
π₀.₇ combines a generalist action policy with detailed language, episode-quality metadata and optional generated image goals. This edition follows arXiv v2, including its complete 87-author byline, supplementary system details and evaluation rubrics.
At a glance
The central proposal is to label heterogeneous robot experience with context describing both the task and how it was performed. This lets one policy learn from demonstrations, imperfect autonomous episodes and non-robot data, then be prompted toward desirable behavior. Evaluations show strong performance on trained dexterous tasks, improved instruction following, transfer to new task–robot combinations and selected new tasks. Human coaching and separately trained high-level policies play an explicit role in the longer-horizon generalization results.
I Introduction
Simply pooling diverse trajectories risks mixing incompatible strategies and quality levels. π₀.₇ instead conditions on subtask descriptions, quality/speed/mistake labels and visual subgoals, with dropout allowing different prompt combinations at deployment. The claimed contribution is improved use of heterogeneous experience and evidence of skill recombination. The evaluation separates specialist-level performance on tasks present in training, instruction generalization in new environments, cross-embodiment transfer and new task compositions; these are different forms of generalization.
II Related Work
The model combines established components: a vision-language backbone, action generation, MEM history encoding, high-level language control and goal-image conditioning. Human video, web supervision and autonomous robot experience broaden its training mixture. The authors position detailed context conditioning as the main methodological contribution, rather than claiming invention of goal-conditioned control or world-model prompting.
III Flow-Based Vision-Language-Action Models
A VLA observes camera views and robot state, then predicts a short chunk of future actions through a smaller flow-matching action expert. Knowledge insulation trains the backbone with discrete FAST action tokens while blocking continuous-action gradients from entering it. Context ordinarily contains a task instruction; here it also disambiguates strategy and episode quality. The displayed likelihood objective is approximate because the flow-matching expert optimizes a bound rather than a closed-form action likelihood.
Equation 1
| (1) |
IV π 0.7 Overview
The action model has approximately 5B parameters: a Gemma3 4B backbone, including its 400M vision encoder, plus an 860M action expert. The memory encoder compresses multiple observation frames into a fixed token budget. A separate high-level policy can supply semantic subtasks, and a separate BAGEL-derived world model can supply image goals. Their parameters and compute should not be conflated with the 5B action model alone.
V Diversifying the Prompt
Prompt components carry complementary information: language describes the intended operation, images depict a desired state, and metadata identifies the quality or execution mode represented by an episode. Component dropout makes these inputs optional at test time.
V-A Subtask instructions
Subtask text specifies the next semantic step in addition to the overall goal. It can come from a trained high-level policy, live human coaching, or be omitted. Coaching a new task supplies timed language instructions without new teleoperated action demonstrations. Turning those episodes into an autonomous system then requires fine-tuning a high-level language policy to produce the instructions from observations and history.
V-B Subgoal images
Multi-view goal images specify near-future scene and gripper configurations that can be difficult to express in words. A separate world model receives current views, subtask language and metadata; it is initialized from BAGEL, a 14B multimodal model, and trained to predict segment-end views. Robot, human-video, image-editing and other video data supply semantic and physical context. Better performance with goal images can therefore reflect additional world-model knowledge and compute, as well as a richer control prompt.
Equation
V-C Episode metadata
Episode metadata includes length discretized in 500-step bins, human-rated quality from one to five, and segment-level mistake labels. These labels tell the policy which behavior mode a training example represents, allowing lower-quality and failed episodes to remain useful. At deployment, requesting high quality and no mistakes steers the conditional policy; those labels describe the desired mode rather than guaranteeing an error-free execution.
V-D Control mode
A textual control-mode identifier selects joint-space or end-effector actions. Both appear during training, and the identifier is always present. Runtime selection changes the action representation used by the controller.
V-E Full prompt and training details
Only 25% of training examples receive image goals, because those examples make action prediction an easier inverse-dynamics problem. Among them, subtask text is omitted 30% of the time. All metadata is dropped 15% of the time, with an additional independent 5% dropout per metadata component. Control mode is never dropped. The original example prompt is retained as a diagram.
VI The π 0.7 Model and Training Recipe
The recipe combines heterogeneous data, memory-aware observation encoding and goal-image training that anticipates imperfect runtime generations.
VI-A Training datasets
Training mixes demonstrations across robot types and environments, autonomous policy evaluations, human interventions, public robot data, egocentric human video and web-based vision/language supervision. It also includes failed or imperfect episodes and rollouts from earlier RL specialists, effectively distilling their behavior. The authors explicitly exclude autonomous data from generalization-focused evaluation tasks. Thus “out of the box” on trained tasks means no task-specific post-training of π₀.₇, not absence of task experience or specialist-generated training data.
VI-B Model architecture
Inputs include up to four camera views with six frames each, sampled one second apart, and up to three goal views. Images are resized to 448×448; history and the optional rear view each have 0.3 dropout. Proprioception uses learned linear projections, with dropped history states masked consistently. Block-causal attention separates observation, goals, text and action groups. The expert generates 50-step chunks, while training-time real-time chunking simulates delays of up to twelve steps, equivalent to 240 ms at 50 Hz.
VI-C Training with subgoal images
For real-image goals, a quarter of examples use the end of the annotated segment and the rest use a future frame sampled zero to four seconds ahead. Additional examples use goals generated by the world model. This exposure reduces the mismatch between clean future observations during training and imperfect or delayed synthesized goals during deployment.
VII Prompting π 0.7 at runtime
Runtime prompts always include control mode and metadata: quality five, no mistake, and a task-specific speed target based on the 15th percentile of episode length. Subtasks come from coaching or the high-level policy. Optional image goals refresh when the subtask changes or four seconds elapse, with generation running asynchronously. The action expert uses five denoising steps and executes fifteen or twenty-five steps of each fifty-step chunk. Metadata classifier-free guidance uses weights 1.3, 1.7 or 2.2 for selected dexterous tasks; the original workflow and guidance equation are retained.
Equation
VIII Robot system details
Experiments use mobile bimanual robots, lightweight static bimanual robots, single-arm systems and a larger UR5e platform with different reach, inertia and grippers. All use parallel-jaw grippers and front/wrist cameras; mobile robots add a rear camera. UR5e runs at 20 Hz and the other platforms at 50 Hz. Joint targets use a PD controller; end-effector targets are converted through numerical inverse kinematics. Cross-embodiment evaluation transfers tasks to robots without examples of those tasks, rather than claiming the target robot itself is entirely unseen.
IX Experimental Evaluation
The evaluation tests five questions: performance on trained dexterous tasks, instruction following, transfer across embodiments, composition of unfamiliar tasks, and how context changes the value of mixed-quality or diverse training data. Figures use several distinct metrics, including binary success, partial task progress and normalized successes per hour.
IX-A Out-of-the-box performance on challenging tasks
One π₀.₇ policy approximately matches task-specialized RL or supervised policies on trained tasks such as espresso preparation, laundry and box building, and can exceed specialist throughput on some tasks. Other evaluations include sandwich preparation, shirt inversion, door traversal, cutting, peeling and trash replacement. Removing episode metadata or autonomous evaluation data lowers performance, especially throughput, supporting the value of contextualized specialist experience. Memory tasks include mug sequencing, finding a hidden object, counting scoops and window cleaning; the generalist is comparable to memory-equipped specialists. These results are not zero-shot task learning.
IX-B Instruction following
Four unseen kitchens and two unseen bedrooms provide fourteen scenarios with three-to-six instructed steps each; the metric is the fraction of individual instructions followed, not whole-sequence success. π₀.₇ improves over π₀.₅/π₀.₆, especially for unusual referential commands, with further gains from generated goals. Reverse-bussing and microwave-to-fridge tasks deliberately contradict common training patterns. Goal images are particularly important for the reversed appliance sequence. Novelty is controlled by scenario design but cannot be exhaustively established against the entire large training mixture.
IX-C Cross-embodiment transfer
Older models also transfer simpler rearrangement tasks, while π₀.₇ has larger gains for demanding embodiment shifts. Examples include a single UR5e arm bagging shirts and two UR5e arms folding cloth despite no UR5e laundry demonstrations. Qualitative rollouts show different grasp and insertion strategies suited to the target geometry. The goal-conditioned model reaches 85.6% shirt-folding progress and 80% success; the reported expert-human comparison is 90.9% and 80.6%, respectively. This compares experienced operators making their initial attempts on this task–robot combination, not unrestricted human performance. Joint-space control is used in the main comparisons.
IX-D Compositional task generalization
Selected new short tasks include pressing a French press, scooping rice, wiping objects and spinning articulated items. Longer appliance tasks require step-by-step human coaching: loading/unloading an air fryer and toasting a bagel. No targeted robot action data were collected for those tasks, although related appliances appear in human and external data. Recorded coaching episodes then train a high-level instruction policy; its autonomous performance is close to coached performance on five tested tasks. This removes the need for additional teleoperation demonstrations but still uses collected coaching data and high-level training.
IX-E Can π 0.7 learn effectively from diverse and mixed-quality data?
Eight laundry models compare four nested data subsets—the top 30%, 50%, 80% and all episodes by quality/speed—with and without metadata. More mixed-quality data improves the metadata-conditioned models, whereas the unconditioned versions can degrade. A second ablation removes either the most task-diverse 20% or a random 20%; removing diverse data harms short-task generalization more at matched quantity. The results support complementary roles for context and diversity, while the authors acknowledge that slicing diversity cleanly remains difficult.
X Discussion
Detailed context makes heterogeneous experience more useful and allows a single action policy to express multiple strategies. Generalization remains weaker than trained-task performance: the discussion describes unseen task or task–robot success around 60–80%, versus often above 90% on seen tasks. Related behaviors may occur incidentally somewhere in the training mixture, so “unseen” means no deliberate targeted collection in the relevant evaluation, not proven absence of every constituent skill. Future work includes improving new-task reliability through coaching or autonomous learning; the reported system already relies on substantial annotation and auxiliary-model compute.
Acknowledgements
The authors acknowledge the operators responsible for collection, evaluation and recording, and the technicians maintaining the robots. Appendix A provides contributions by activity.
Appendix A
The appendix documents contributions, attention layouts, world-model training, inference costs, the human comparison and task scoring criteria. These details delimit the headline generalization claims.
A-A Contributions
The Physical Intelligence team divides contributions across data collection and operations, annotation and supplemental data, policy research and training, policy infrastructure, hardware, robot infrastructure, and writing/illustration. The edition’s byline uses all 87 names in the reviewed v2 source; earlier imported name variants remain available as search aliases.
A-B Attention pattern
Figure 19 retains the full attention masks. Observation views share bidirectional attention; goal views add a separate block. Training FAST tokens and continuous flow actions do not attend to each other. Classifier-free guidance packs positive and negative branches into one sequence without mutual attention. The world model uses ViT observation tokens plus VAE encodings of current and noisy goal images, with a three-group guidance arrangement.
A-C Training of the world model
The BAGEL-derived world model is trained on robot/human segments with high-quality temporal language labels, mixed with image-editing and video data. Its understanding and generation backbones each have 7B parameters. ViT inputs use 448×336 and VAE inputs/targets 512×384 because their patch sizes differ. Accurate temporal segmentation is important for goal quality; runtime regeneration uses a four-second interval.
A-D Inference speed and optimization
The VLA and high-level policy use one H100; minimal VLA inference is reported at 38 ms, rising to 127 ms with memory and image goals. The separate 14B goal generator uses four H100s with tensor parallelism, eight-bit matrix multiplications and modified SageAttention, producing twenty-five-step guided generations in 1.25 seconds. Robot execution continues asynchronously during goal generation. These hardware and latency details qualify the description of the world model as lightweight.
A-E Comparison of action spaces in cross-embodiment transfer
Across the baseline cross-embodiment tasks, end-effector control provides no clear systematic advantage over joint-space control. The main study therefore keeps joint-space control for a consistent comparison. This does not establish equivalence for every robot or task.
A-F Human subject study for cross-embodiment shirt folding
Ten operators from the most experienced 2% of the fleet, averaging about 375 hours across robots, attempt UR5e shirt folding three times each with no warm-up before the first attempt. They share the policy’s flattened-shirt start, time limit and scoring rubric. The source reports thirty trials and human averages of 90.9% progress/80.6% success, versus 85.6%/80% for the goal-conditioned policy. The reported 80.6% is retained as published rather than reconstructed as a simple integer fraction of thirty; the aggregation details are not fully specified. Similar measured scores do not constitute an equivalence test.
A-G Detailed task descriptions and scoring rubric
The rubrics distinguish complete success from partial progress. Laundry requires a neat fold and stack; espresso requires the specified dispensing, tamping, extraction and serving sequence; box building requires an intact folded box. Sandwich preparation is scored out of 9, shirt inversion out of 7, doorway traversal out of 3, zucchini slicing out of 3, and peeling/cleanup out of 9. Trash replacement has twelve points split among accessing the bin, removing the old bag, installing the new one and returning the bin. Memory tasks score mug swapping out of 4, hidden-object retrieval and drawer closure out of 4, two coffee scoops and lid closure out of 5, and window cleaning/disposal out of 5.
Reverse bussing scores twelve correctly reassigned objects; the reversed appliance task gives six points for food transfer and door operations. Table setting scores seven items, with possible penalties for critically incorrect placements; backpack insertion scores 3, nested containers/lids 6, and bagging two shirts 4. Shirt folding gives three alignment-based fold points plus up to three quality points; all six are required for success. New-task rubrics score a fully depressed French-press plunger out of 1, rice scooping out of 3, loading an air fryer out of 4, unloading all food onto a plate out of 2, and bagel toasting/serving out of 7. These definitions explain why high normalized progress can coexist with lower complete-task success.