At a glance
SPARTA uses visual maps of actionable and already transformed object regions to guide mashing, spreading, and slicing. The same spatial-progress representation supports a learned reinforcement-learning policy and a greedy controller. Real-robot experiments cover three tasks and ten objects, with approximately 1.5–3 hours of online policy training. Evaluation measures transformed area, not binary task success. The demonstration-free claim concerns policy learning; pretrained perception, task-specific motion primitives, and a few greedy bootstrap rollouts remain part of the system.
I Introduction
Many manipulation tasks change an object’s texture, shape, or material state rather than just its pose. A useful controller must identify what remains to transform and choose the next interaction location. SPARTA abstracts RGB appearance into actionable/transformed masks and uses their change both as policy input and as dense reward. The intended transfer is between objects sharing this progress structure, rather than unrestricted transfer across mechanics or arbitrary manipulation skills.
II Related Work
The work connects object-state-change perception with robot control. It adapts SPOC segmentation [29], which was developed for passive video understanding, into an online affordance map. Unlike global visual reward embeddings such as LIV [24], the map localizes incremental progress. Unlike approaches centered on grasp points or a single contact task, it supplies a shared representation for several spatially progressive transformations. Policy training builds on SERL [23] rather than introducing a new reinforcement-learning optimizer.
III Robotic Object State Change
The task is partially observed: RGB and end-effector state provide proxies for evolving object material and geometry. The policy chooses continuous planar displacements on a constrained surface manifold. A predefined task-specific primitive then sweeps, presses, or slices at that location. This reduces learning to where to apply an interaction; it does not learn arbitrary tool trajectories, force strategies, or six-dimensional manipulation from scratch.
IV SPARTA: Robot Policies for OSCs via Visual Spatial Progress
The framework separates visual progress estimation from action selection. Both policy variants receive the same state-change maps, allowing either local greedy selection or learned sequential control.
IV-A Integrating SPOC Visual Affordances for Robotics
Grounded-SAM first segments the object. Farthest-point prompts generate internal regions, which GPT-4o classifies as actionable or transformed. DeAOT propagates masks between expensive model queries, reducing the reported per-frame tracking cost to about 0.2 seconds versus roughly five seconds per GPT query. The robot policy sees the resulting masks and proprioception rather than raw appearance. This abstraction can support object transfer, but depends on perception correctly recognizing transformation and maintaining masks through changing shape and appearance.
IV-B SPARTA-L: Reinforcement Learning with SPOC rewards
SPARTA-L combines a progress reward from changes in transformed-area fraction with a terminal reward above 95% transformation and an entropy term. It uses asynchronous soft actor-critic with SERL/RLPD-style regularization. Dense feedback tells the agent which local actions change the object, while return optimization can trade immediate progress against future access. Reward accuracy still inherits segmentation errors, and transformed-area growth is a visual proxy for the desired physical outcome.
Equations 1–2
| (1) |
| (2) |
IV-C SPARTA-G: Greedy Policy with SPOC Maps
SPARTA-G evaluates eight planar directions and chooses the endpoint neighborhood with the greatest density of actionable pixels. It requires no policy training and can work well with broad, symmetric tools whose footprint tolerates localization error. The learned controller is intended to handle noisy mappings and narrower, directional interactions more effectively. Neither variant removes the task-specific primitive or surface-calibration assumptions.
Equation 3
| (3) |
V Experimental Evaluation
The Franka experiments cover spreading, mashing, and thin slicing over ten objects. Spreading uses fixed-height brush strokes and automatic refilling every two steps; mashing and slicing move laterally and press until a force threshold. The tool lifts after every action before the next image, avoiding occlusion. Episodes start from a fixed workspace corner and last ten steps for spreading or five for mashing and slicing.
Spreading trains for 40 episodes, approximately three hours including refills and resets; mashing and slicing require about 1.5 hours. Resettable clay proxies are used for the latter tasks, with roughly five greedy rollouts to bootstrap exploration. These details qualify the claim of autonomous learning directly from real interactions. The implementation uses a mask encoder, proprioceptive network, and a 1:10 actor-to-critic update ratio.
Comparisons use random actions, a GPT-queried sparse terminal reward, and LIV language-goal rewards under the same training protocol. The metric is transformation coverage from segmentation with human annotation correction. Table I averages three seeds and five rollouts per seed. Its decimal entries are area fractions despite the caption’s percent label; for example, 0.61 represents 61% coverage, not 0.61% success.
Experiments and Results:
Both SPARTA variants substantially exceed the tested learned baselines across the table. SPARTA-L reaches coverage fractions from 0.42 to 0.78 across its object columns; SPARTA-G ranges from 0.39 to 0.75. Learning generally helps fine directional tasks, but it is not uniformly better: the greedy policy is competitive and sometimes stronger, including unseen mashing objects. Weak reward baselines can perform worse than random control, indicating that their learned behavior degrades under this protocol.
On bread spreading, the progress-based cumulative rewards track transformation more smoothly than LIV, and the learned policy reaches over 60% coverage in roughly 90 minutes. These plots support reward usefulness in this setting; they do not independently isolate every architectural or optimization contribution. A separate partially transformed-object test compares greedy state-change maps with whole-object masks. Targeting remaining actionable regions yields the reported threefold coverage efficiency and avoids repeatedly working on already transformed areas.
| Spread | Slice | Mash | ||||||||||
| Seen | Unseen | Seen | Unseen | Seen | Unseen | |||||||
| Model | ||||||||||||
| Random | 0.24 | 0.42 | 0.27 | 0.29 | 0.23 | 0.13 | 0.15 | 0.14 | 0.18 | 0.14 | 0.23 | 0.20 |
| Sparse | 0.14 | 0.10 | 0.07 | 0.11 | 0.13 | 0.09 | 0.08 | 0.09 | 0.13 | 0.08 | 0.09 | 0.18 |
| LIV [10] | 0.17 | 0.14 | 0.12 | 0.16 | 0.12 | 0.10 | 0.09 | 0.11 | 0.13 | 0.09 | 0.10 | 0.08 |
| SPARTA-G | 0.44 | 0.49 | 0.55 | 0.66 | 0.39 | 0.63 | 0.60 | 0.61 | 0.75 | 0.69 | 0.71 | 0.75 |
| SPARTA-L | 0.61 | 0.55 | 0.58 | 0.63 | 0.42 | 0.78 | 0.69 | 0.67 | 0.77 | 0.72 | 0.62 | 0.68 |
VI Lessons and Conclusion
Spatial progress maps provide a common interface between perception and several object-transforming tasks, with either trained or heuristic control. Generalization still degrades for unseen geometry, such as transferring a policy from rectangular food to circular tortillas. Perception is only acquired after the tool lifts, so the current system does not continuously observe contact. Shape-aware experience and occlusion-resilient perception are proposed next steps.
ACKNOWLEDGMENT
The work acknowledges support from the UT Austin IFML NSF AI Institute and DARPA TIAMAT, and thanks Luca Macesanu for tool-attachment design.