Overview
The figures connect automatic affordance annotation with task-conditioned visual maps and robot imitation learning. All eight numbered figures, the original system/user prompt and the complete benchmark table remain available.
At a glance
UAD combines fine-grained visual features with a vision-language model’s task knowledge to produce synthetic affordance labels. A small language-conditioned decoder learns from those labels while its visual backbone stays frozen. Used as the visual input to a manipulation policy, the representation supports selected changes in object appearance, category and instruction after ten demonstrations per task. “Unsupervised” describes construction of the affordance training labels; evaluation uses human masks and physical policy training uses human demonstrations.
I Introduction
A robot needs to identify the part of an object relevant to an instruction, such as a handle for grasping or a rim for pouring. VLMs can describe such relationships but struggle to localize them precisely; DINOv2 provides spatial features without task semantics. UAD links the two by assigning language descriptions to clustered object regions and then distilling those associations into a dense predictor.
The contribution is a task-conditioned visual representation and its use in imitation learning. Its visual generalization does not automatically supply new manipulation skills, contact dynamics or motion planning. Foundation-model pretraining remains a substantial source of supervision and prior knowledge.
II Related Works
The paper relates affordance discovery to visual representations for manipulation and to foundation-model knowledge extraction.
II-A Learning and Discovering Visual Affordance for Robotics
Prior affordance methods learn pixelwise action regions, descriptors or continuous action possibilities from human labels or interaction data. UAD instead synthesizes labels from pretrained models. Its scores express task-related visual similarity; they are not calibrated probabilities that a particular robot can successfully execute an action at that pixel.
II-B Pre-trained Visual Representation for Manipulation
Task-conditioned visual representations can change what a policy attends to when the instruction changes. CLIP-like global alignment may miss fine-grained parts, motivating dense affordance supervision. The downstream comparison measures UAD against particular RGB, DINOv2, CLIP and Voltron implementations, rather than every possible adaptation of those encoders.
II-C Foundation Models for Robotics
UAD reformulates localization as a visually prompted question-answering task, then trains a smaller predictor so GPT-4o need not run at every policy step. This builds on earlier visual prompting and robot foundation-model work; the main distinction is the extraction-and-distillation pipeline and its task-conditioned dense outputs.
III Method
The method has three stages: construct synthetic instruction/region labels, train the affordance decoder and use its frozen predictions as part of a robot policy observation.
III-A Extracting Affordance Annotations
The main object database contains 206 assets from 76 BEHAVIOR-1K categories [117] and 667 task instructions. Each object is rendered from 14 views. DINOv2 [14] features are projected into a common point cloud and averaged over visible views. PCA reduces them to three dimensions before clustering, intended to suppress local texture sensitivity. This uses known rendering geometry and foreground masks, rather than reconstructing arbitrary real-world scenes from RGB alone.
CLIP [18] selects a natural-looking view using image/category similarity. GPT-4o receives the original view, a color-coded region overlay and the object category, and proposes fine-grained task descriptions paired with one region each. The source cites reference 10, a GPT-4 technical report, for the GPT-4o used in the implementation; the named model and bibliography are preserved.
For a selected region, the method averages its original DINOv2 features, compares that reference with each point by cosine similarity, clips negative scores to zero and projects the result into each view. This spreads a discrete region choice into a continuous map. It does not measure physical interactions, reachability or success likelihood, and a visually similar region can receive high scores without being functionally equivalent.
III-B Learning Task-conditioned Affordance Model
The predictor freezes DINOv2 and trains three FiLM-conditioned layers [120] with output channels 256, 64 and one. Language embeddings from an OpenAI API control channel-wise transformations shared across spatial locations. Binary cross-entropy fits the synthetic soft masks. The paper calls the final values both logits and numbers in [0,1]; probabilities are bounded, whereas logits are not, so the exact activation/loss convention should be checked in an implementation.
Keeping the visual backbone frozen carries pretrained visual features into real-world scenes. This is a compact supervised distillation stage on automatically produced labels, rather than an absence of a training target.
III-C Policy Learning with Affordance as Observation Space
An RVT-style multi-view policy [121] receives affordance maps together with depth, world-frame XYZ coordinates, pixel positions and proprioception. Thus affordance replaces the visual appearance representation, while geometry and robot state remain available. The action is a six-degree-of-freedom end-effector pose plus a binary gripper command.
The policy assumes detailed subtask instructions, such as grasping a watering can, aligning its spout and watering a plant. Each task gets its own imitation policy. The system does not learn a general autonomous language planner or a universal multi-task motion policy from ten demonstrations.
IV Experiments
Experiments separately test affordance prediction, simulation policies and physical manipulation. Their metrics and supervision differ, so a successful segmentation map should not be equated with successful robot action.
IV-A Task-Conditioned Affordance Prediction
On rendered objects, four sets of 100 image/instruction pairs cover training examples, unseen instances, unseen categories and unseen instructions. AUC is at least 0.92 in each set using human masks. For a selected DROID subset [16], average AUC is 0.840 for UAD, 0.836 for OpenSeeD and 0.500 for the CLIP baseline. The 0.004 margin over OpenSeeD is small and no uncertainty interval is reported. DROID instructions are expanded into detailed substeps, images are center-cropped, and examples with unclear or occluded target objects are removed; this is not unrestricted evaluation on all DROID scenes.
On the AGD20K Unseen split [59], UAD obtains SIM 0.407, higher than the listed baselines. Its standard NSS 1.092 trails LOCATE at 1.157, and KLD 1.878 is worse than LOCATE’s 1.405 and AffordanceLLM’s 1.463. On the authors’ stricter NSS-0.5 metric, UAD scores 2.050 versus LOCATE’s 1.723; other baselines have no reported value. The result is a mixed metric comparison, not uniformly best performance.
The model is not trained on AGD20K images, but five action/object prompts are manually disambiguated to match that benchmark’s annotation conventions. The new threshold favors agreement on central salient regions and changes what is measured. Sparse foreground predictions partly explain weaker KLD against diffuse labels; that explanation does not invalidate the benchmark’s original metrics.
IV-B Policy Learning in Simulation
Three OmniGibson tasks—pouring, opening and insertion—use ten scripted demonstrations per task. Policies are evaluated for object pose, instance, category and instruction changes, with 15 trials per task/condition. Pose randomization at test matches the training ranges; it tests new configurations within those ranges rather than extrapolation to arbitrary locations. Category substitutions preserve useful functional structure, and instruction changes preserve a similar motion pattern.
Figure 6 reports UAD average success of 1.00 for poses, 0.84 for new instances, 0.73 for categories and 0.47 for instructions. DINOv2 also reaches 1.00 on pose changes and 0.71 on categories, so the category advantage is about one success among 45 trials. The instruction setting covers only pouring and insertion: opening has no novel-instruction evaluation, despite the main-text statement that every bar averages three tasks. No repeated training runs or confidence intervals are supplied.
Instruction changes redirect a pouring motion from a bowl to a plant, or insertion from a marker to a carrot. These are useful tests of identifying the intended object/region; they do not show acquisition of a fundamentally new motor skill. Simulation pouring success checks alignment and tilt over the receptacle, not measured liquid transfer.
IV-C Policy Learning in the Real World
A Franka Panda performs drawer opening, plant watering and pen insertion using two RGB-D cameras. Each policy trains on ten kinesthetic demonstrations and is tested ten times with varied configurations. Figure 3 gives 80%, 80% and 60% respectively, totaling 22 successes in 30 trials, or approximately 73%. Success is judged visually by the operator. The small physical experiment has no corresponding real-world baseline comparison or confidence interval, and does not establish the same breadth of transfer as the simulation tests.
V Conclusion & Limitations
The results support automatically distilled task-conditioned maps as a useful visual input for small-data imitation learning. The authors explicitly limit the approach to static-frame affordance and single-object synthetic annotation, and distinguish visual from motion-level generalization. Additional limits include dependence on pretrained-model biases and synthetic geometry, selected evaluation scenes, detailed instruction inputs, narrow task families and small trial counts. Real-world multi-object annotation and temporally extended understanding remain future work.
ACKNOWLEDGMENT
The work reports support from NSF robotics grants and ONR/MURI awards listed in the original acknowledgment.
-A Details on Unsupervised Affordance Annotation Extraction Pipeline (Sec III-A )
The annotation pipeline projects simulator foreground pixels to a downsampled world-frame cloud. DINOv2 ViT-L/14 with registers produces patch features that are bilinearly upsampled; a depth-consistency test decides which views contribute to each point. Mean Shift clusters the three PCA coordinates. If fewer than five clusters result, k-means requests five; articulated objects are clustered separately by simulator-provided link masks. These details are additional structured information available in rendered assets.
The complete prompt below asks for task-relevant parts and one matching color per description, with generic textual examples. It explicitly asks the VLM to make its best guess when uncertain. A nearest-neighbor projection transfers region labels and feature-similarity scores to image pixels. Over-segmentation is preferred because similar features can extend a selected region’s score to neighboring pieces; under-segmentation can merge useful and irrelevant parts. The degree of annotation error is not independently quantified.
## System Prompt
Given (a) the category of an object, (b) a reference image of the original image, (c) an image visualizing clustering of the object into a few regions, each indicated by a distinct color, and (d) a related list of used colors, please:
1. Identify specific regions of the object that serve different purposes in various manipulation tasks.
- Focus on crucial parts and offer detailed and fine-grained descriptions of the regions of interest.
- For each identified region, provide both a Region Description and a Region for action xxx. For example, "handle of plastic bag -- region for agent to hold and lift the bag." Use double quotes only to represent a string element.
- Avoid using multiple single quotes for an element. For example, instead of "adjusting the lamp's position", use "adjusting the lamp's position." Avoid trivial regions like: power cord, power plug, seal, small edges, small corners, different sides of the walls or body, interior base, or exterior edge.
2. Match the colored region in (c) with the proposed task in step 1, considering the functionality and the granularity of the task. The requirements are as follows:
- Compare the original image to the proposals to find the colored region that matches the description, considering the context provided by the explanation. For example, if a given proposal image indicates that the red region covers the handle, and the description mentions a task related to the handle, you should identify the answer as "Red."
- When the described region is clustered into more than one cluster on the image, pick the cluster that is most appropriate according to the explanation provided in the description.
- If the described region is within a cluster but still contains other parts of the object, you should still select the cluster.
- Consider the reason/explanation to make the final decision, and provide your best guess.
If you cannot identify the counterpart on the given image, give your best guess. Do not say you cannot identify something.
## User Prompt
The first image is the original image of the object, and the second image shows the clustering of regions in colors.
This is the color list: {CLUSTER_COLORS}, and this is the object category:{OBJ_CATEGORY}.
I need you to propose the task-guided fine-grained region description and match region description with the one most appropriate color in the images.
Output format: Start with the word "ANSWER: ", followed by a dict, where each key-value pair is in the format of "region description -- region for xxx" : "Color", separated by commas. Specifically, the content after "ANSWER: " should be parseable with Python's ast.literal_eval() and nothing else. All elements in the dict keys or values should be enclosed by double quotes only. The color could only be one color, with the first letter of the color name capitalized and the rest in lowercase.
Appendix A Details on Task-conditioned Affordance Model Training (Sec III-B )
Synthetic mask values below 0.5 are set to zero, followed by a three-pixel Gaussian blur. The decoder uses three FiLM-conditioned convolution layers; linear language-to-channel transformations are reported as initialized with weights one and bias zero. Adam uses learning rate 0.001, batch eight and 30 epochs, approximately 12 hours on one A6000 GPU. The specific language-embedding model and several geometry/clustering thresholds are not identified in the paper, limiting exact reproduction from the text alone.
A-A Details on Task-conditioned Affordance Prediction (Sec IV-A )
Seven MTurk workers label each evaluation image/instruction pair; a pixel is positive when more than three workers select it. Average time is 40 seconds per annotation, compensated at $0.60. Human annotation is therefore present in evaluation even though it is absent from construction of the affordance training set.
For AGD20K, predictions are clipped to [ε, 1−ε] before normalization for KLD. Five prompt changes are explicit: axe “hit” selects the handle used during hitting; bicycle “ride” selects sitting/pushing regions; cup “pour” selects its handle; cup “wash” selects the rim; and cup “hold” selects the handle. These choices align instructions with labeled regions and should be included in any reproduction of the zero-shot evaluation.
Standard NSS evaluates pixels above ground-truth threshold 0.1; NSS-0.5 uses 0.5. Gaussian mixtures around annotated points spread low values beyond object boundaries, so the stricter threshold emphasizes central regions. Figure 8 shows both sets exactly. The CLIP baseline uses upsampled patch/text cosine similarity clipped at zero; OpenSeeD produces a binary mask or all-zero prediction if it finds no mask. Those implementations differ from UAD’s continuous map.
A-B Policy Learning in Simulation Details (Sec IV-B )
The Fetch base is fixed in simulation. A keyframe specifies end-effector pose then gripper action; control interpolates between poses. Pouring and insertion use three cameras, while opening uses two to reduce cabinet occlusion. Pouring uses four keyframes and alignment/tilt success. Opening uses a two-step handle grasp and pull, requiring at least 45° door rotation. Insertion requires the object to end inside the holder.
Training/evaluation pose ranges are ±5/±3 cm for pouring objects; ±5 cm in cabinet XY and ±15° yaw for opening; and ±1.5 cm for the marker/carrot with ±3 cm holder motion for insertion. New instances change models, colors and sizes. New categories use a can for the bottle, a refrigerator for the cabinet, or a coffee cup for the holder. Only pouring and insertion have changed instructions.
RGB is the vanilla baseline. DINOv2 and Voltron features are projected to three channels with learned pointwise convolutions; Voltron uses its frozen language-conditioned V-cond variant [74]. CLIP uses per-pixel text similarity. Policies train for 4,000 epochs at batch three on an A40, about 16 hours for pouring and eight hours for each other task. Visual inputs are normalized, geometry clipped to workspace bounds, out-of-bounds depth zeroed, and random crops applied. Ten demonstrations therefore does not imply a short optimization run.
A-C Policy Learning in Real-World Details (Sec IV-C )
The physical arm runs position commands at 20 Hz using interpolation steps of 5 mm and 1°, workspace clipping, PyBullet inverse kinematics [131] and Deoxys joint impedance control [132]. Two Orbbec Femto Bolt cameras supply RGB and depth at 20 Hz. These are system/control rates, not a reported end-to-end affordance-network inference rate. Ten visually judged trials per task use ten kinesthetic keyframe demonstrations and the simulation training procedure.
A-D Case Study: Scaling to Objaverse-XL Assets
The post-acceptance case study applies annotation to Objaverse-XL assets [118], reaching more than 10,000 object/instruction pairs in aggregate. It samples up to 50 models from abundant LVIS-labeled categories, filters transparent assets that break depth rendering, and removes renders that miss part of the object. This demonstrates processing a broader asset collection; no separate policy-success or affordance-quality results establish a scaling benefit for the enlarged set.