Literature
Display
← All papers

Mash, Spread, Slice! Learning to Manipulate Object States via Visual Spatial Progress

Priyanka Mandikal · Jiaheng Hu · Shivin Dass · Sagnik Majumder · Roberto Martín-Martín · Kristen Grauman

· ICRA 2026 ↗

Robotics5 min text · CondensedOriginal paper ↗
Contents

At a glance

SPARTA uses visual maps of actionable and already transformed object regions to guide mashing, spreading, and slicing. The same spatial-progress representation supports a learned reinforcement-learning policy and a greedy controller. Real-robot experiments cover three tasks and ten objects, with approximately 1.5–3 hours of online policy training. Evaluation measures transformed area, not binary task success. The demonstration-free claim concerns policy learning; pretrained perception, task-specific motion primitives, and a few greedy bootstrap rollouts remain part of the system.

I Introduction

Many manipulation tasks change an object’s texture, shape, or material state rather than just its pose. A useful controller must identify what remains to transform and choose the next interaction location. SPARTA abstracts RGB appearance into actionable/transformed masks and uses their change both as policy input and as dense reward. The intended transfer is between objects sharing this progress structure, rather than unrestricted transfer across mechanics or arbitrary manipulation skills.

Refer to caption
Fig. 1: Top: While most robotic manipulation focuses on rigid-body motion, many real-world tasks involve object state changes such as mashing, spreading, or slicing, where objects are progressively transformed. Bottom: SPARTA leverages spatially-progressing affordance maps of actionable vs. transformed regions, successfully demonstrating how to guide real robot manipulation for this family of tasks.

II Related Work

The work connects object-state-change perception with robot control. It adapts SPOC segmentation [29], which was developed for passive video understanding, into an online affordance map. Unlike global visual reward embeddings such as LIV [24], the map localizes incremental progress. Unlike approaches centered on grasp points or a single contact task, it supplies a shared representation for several spatially progressive transformations. Policy training builds on SERL [23] rather than introducing a new reinforcement-learning optimizer.

III Robotic Object State Change

The task is partially observed: RGB and end-effector state provide proxies for evolving object material and geometry. The policy chooses continuous planar displacements on a constrained surface manifold. A predefined task-specific primitive then sweeps, presses, or slices at that location. This reduces learning to where to apply an interaction; it does not learn arbitrary tool trajectories, force strategies, or six-dimensional manipulation from scratch.

Refer to caption
Fig. 2: Overview of SPARTA. At each episode step, our policy takes the current and past SPOC [29] visual-affordance (segmentation) maps as inputs (Sec. IV-A), along with the robot arm’s proprioception data and predicts a displacement action for the arm’s end-effector. SPARTA supports two robot policy variants: (a) SPARTA-L (Learning): a reinforcement learning agent trained using a dense reward that measures the progressive change of object regions from actionable (red) to transformed (green) (Sec. IV-B); (b) SPARTA-G (Greedy): selects among 8 discrete directions based on the local density of actionable pixels, producing a fast, greedy policy guided by visual progress (Sec. IV-C).

IV SPARTA: Robot Policies for OSCs via Visual Spatial Progress

The framework separates visual progress estimation from action selection. Both policy variants receive the same state-change maps, allowing either local greedy selection or learned sequential control.

IV-A Integrating SPOC Visual Affordances for Robotics

Grounded-SAM first segments the object. Farthest-point prompts generate internal regions, which GPT-4o classifies as actionable or transformed. DeAOT propagates masks between expensive model queries, reducing the reported per-frame tracking cost to about 0.2 seconds versus roughly five seconds per GPT query. The robot policy sees the resulting masks and proprioception rather than raw appearance. This abstraction can support object transfer, but depends on perception correctly recognizing transformation and maintaining masks through changing shape and appearance.

Original figure
Fig. 3: Our SPOC affordance map generation pipeline. (a) Grounded-SAM [37] is used to extract an object mask from the initial frame. (b) Farthest-point sampling generates intra-object regions, classified into actionable or transformed by prompting GPT-4o [33] using color-coded overlays. (c) Once classified, transformed regions are tracked across subsequent frames using DeAOT [48] to maintain temporal consistency with minimal computation.

IV-B SPARTA-L: Reinforcement Learning with SPOC rewards

SPARTA-L combines a progress reward from changes in transformed-area fraction with a terminal reward above 95% transformation and an entropy term. It uses asynchronous soft actor-critic with SERL/RLPD-style regularization. Dense feedback tells the agent which local actions change the object, while return optimization can trade immediate progress against future access. Reward accuracy still inherits segmentation errors, and transformed-area growth is a visual proxy for the desired physical outcome.

Equations 1–2
rt=α​Rts​p​o​c+β​Rts​u​c​c+η​Rte​n​t​r​o​p​y.r_{t}=\alpha R^{spoc}_{t}+\beta R^{succ}_{t}+\eta R^{entropy}_{t}. (1)
Rts​p​o​c=At+1t​r​f−Att​r​fAta​c​tR^{spoc}_{t}=\frac{A_{t+1}^{trf}-A_{t}^{trf}}{A_{t}^{act}} (2)

IV-C SPARTA-G: Greedy Policy with SPOC Maps

SPARTA-G evaluates eight planar directions and chooses the endpoint neighborhood with the greatest density of actionable pixels. It requires no policy training and can work well with broad, symmetric tools whose footprint tolerates localization error. The learned controller is intended to handle noisy mappings and narrower, directional interactions more effectively. Neither variant removes the task-specific primitive or surface-calibration assumptions.

Equation 3
at=argmaxat(i)∑x∈𝒩⁡(pt+at(i))𝕀[o′(x)=actionable],a_{t}=\arg\max_{a_{t}^{(i)}}\sum_{x\in\mathcal{N}(p_{t}+a_{t}^{(i)})}\mathbb{I}[o^{\prime}(x)=\texttt{actionable}], (3)

V Experimental Evaluation

The Franka experiments cover spreading, mashing, and thin slicing over ten objects. Spreading uses fixed-height brush strokes and automatic refilling every two steps; mashing and slicing move laterally and press until a force threshold. The tool lifts after every action before the next image, avoiding occlusion. Episodes start from a fixed workspace corner and last ten steps for spreading or five for mashing and slicing.

Spreading trains for 40 episodes, approximately three hours including refills and resets; mashing and slicing require about 1.5 hours. Resettable clay proxies are used for the latter tasks, with roughly five greedy rollouts to bootstrap exploration. These details qualify the claim of autonomous learning directly from real interactions. The implementation uses a mask encoder, proprioceptive network, and a 1:10 actor-to-critic update ratio.

Comparisons use random actions, a GPT-queried sparse terminal reward, and LIV language-goal rewards under the same training protocol. The metric is transformation coverage from segmentation with human annotation correction. Table I averages three seeds and five rollouts per seed. Its decimal entries are area fractions despite the caption’s percent label; for example, 0.61 represents 61% coverage, not 0.61% success.

Experiments and Results:

Both SPARTA variants substantially exceed the tested learned baselines across the table. SPARTA-L reaches coverage fractions from 0.42 to 0.78 across its object columns; SPARTA-G ranges from 0.39 to 0.75. Learning generally helps fine directional tasks, but it is not uniformly better: the greedy policy is competitive and sometimes stronger, including unseen mashing objects. Weak reward baselines can perform worse than random control, indicating that their learned behavior degrades under this protocol.

On bread spreading, the progress-based cumulative rewards track transformation more smoothly than LIV, and the learned policy reaches over 60% coverage in roughly 90 minutes. These plots support reward usefulness in this setting; they do not independently isolate every architectural or optimization contribution. A separate partially transformed-object test compares greedy state-change maps with whole-object masks. Targeting remaining actionable regions yields the reported threefold coverage efficiency and avoids repeatedly working on already transformed areas.

Spread Slice Mash
Seen Unseen Seen Unseen Seen Unseen
Model [Uncaptioned image] [Uncaptioned image] [Uncaptioned image] [Uncaptioned image] [Uncaptioned image] [Uncaptioned image] [Uncaptioned image] [Uncaptioned image] [Uncaptioned image] [Uncaptioned image] [Uncaptioned image] [Uncaptioned image]
[Uncaptioned image] [Uncaptioned image] [Uncaptioned image] [Uncaptioned image] [Uncaptioned image]
Random 0.24 0.42 0.27 0.29 0.23 0.13 0.15 0.14 0.18 0.14 0.23 0.20
Sparse 0.14 0.10 0.07 0.11 0.13 0.09 0.08 0.09 0.13 0.08 0.09 0.18
LIV [10] 0.17 0.14 0.12 0.16 0.12 0.10 0.09 0.11 0.13 0.09 0.10 0.08
SPARTA-G 0.44 0.49 0.55 0.66 0.39 0.63 0.60 0.61 0.75 0.69 0.71 0.75
SPARTA-L 0.61 0.55 0.58 0.63 0.42 0.78 0.69 0.67 0.77 0.72 0.62 0.68
TABLE I: SPARTA shows strong training and generalization results for objects with varying textures, colors and shapes. Metric is transformation coverage (%). Results averaged over 3 seeds, 5 rollouts per seed (15 evaluations total).
Refer to caption
Fig. 4: SPARTA significantly outmatches the baselines vis-a-vis substantially transforming actionable regions across diverse objects with varying colors, shapes, and textures.
Refer to caption
Fig. 5: Reward curves for bread-spreading task. a) Cumulative episode reward curves: SPARTA produces smooth, incremental rewards aligned with visual progress, while LIV rewards remain unstable throughout the episode, offering poor guidance. b) Training curves: stable, dense feedback drives sample-efficient learning, with SPARTA rapidly improving while Sparse and LIV stagnate.
Refer to caption
Fig. 6: Compared to ObjMask, a state-change agnostic policy which naively traverses the entire object area and repeatedly acts on already transformed regions, SPARTA selectively targets only actionable areas, driving efficient state progression.

VI Lessons and Conclusion

Spatial progress maps provide a common interface between perception and several object-transforming tasks, with either trained or heuristic control. Generalization still degrades for unseen geometry, such as transferring a policy from rectangular food to circular tortillas. Perception is only acquired after the tool lifts, so the current system does not continuously observe contact. Shape-aware experience and occlusion-resilient perception are proposed next steps.

ACKNOWLEDGMENT

The work acknowledges support from the UT Austin IFML NSF AI Institute and DARPA TIAMAT, and thanks Luca Macesanu for tool-attachment design.

Concept demonstration

Open image ↗