At a glance
KALM uses a vision-language model and segmentation model to propose task-relevant object points, then keeps points whose correspondences can be found consistently across a few demonstrations. A diffusion model predicts end-effector trajectories relative to the selected points.
With ten demonstrations per real task, it improves over selecting points without correspondence verification on handle lifting, drawer opening, and pouring. The simulation study uses manually supplied oracle keypoints, so it evaluates the representation’s data efficiency separately from the automatic discovery pipeline.
I Introduction
Keypoints can remove irrelevant image detail and anchor actions to an object, but manually designing and labeling them is costly. A large model can propose a useful part, yet a plausible proposal may be difficult to localize consistently across views and objects.
KALM combines task relevance with correspondence consistency. One successful video and a task description guide proposals; the other demonstration observations test whether candidate points remain recognizable. The accepted positions and features condition a learned trajectory model. The central contribution is this proposal–verification loop plus an object-relative action representation, rather than using the language model to output motor commands directly.
II Method
Training first selects keypoints, then fits a trajectory model. At inference, the system detects corresponding points in the new scene, samples a relative manipulation trajectory, transforms it into the world frame, and plans an approach to its starting pose.
II-A Problem Formulation
Each skill requires a successful video, a language description, and roughly five to ten demonstrations. Each demonstration includes a calibrated initial RGBD observation and joint trajectory. No additional keypoint labels are required by the real-world discovery procedure.
The method assumes each demonstration has been split into a free-motion approach and a manipulation phase, for example using distance to a keypoint. The learned model predicts only the manipulation segment; a separate planner handles the approach. Thus the input includes calibration and skill segmentation as well as raw demonstrations.
II-B Keypoint Desiderata
A useful point must identify an object part needed for the task and remain recognizable across the training observations. These requirements address different failure modes.
Task relevance.
The language model judges task relevance from the successful video and description. For lifting a coffee-machine handle, the handle is a better reference than the water reservoir. The relevance score is conceptual: the implementation uses prompted proposals, rather than a separately calibrated numerical relevance model.
Cross-instance consistency.
Consistency asks whether the detector can locate the same functional point under different viewpoints, poses, and object shapes. Distinctive corners can be easier to match than textureless surfaces. Successful matching is a proxy for stable identity; it does not independently establish that a consistently matched point is causally sufficient for the task.
II-C Keypoint Proposal and Verification
The loop selects a coarse image region, refines it to an object-part mask, samples 3D points, and tests their matches across demonstrations. If too few candidates pass, it discards the set and asks for another proposal. Algorithm 1 retains the complete procedure.
Coarse-grained region proposal VLM 1 ( ℋ , d ) .
The VLM sees the demonstration frames, task description, and a labeled grid over the initial image. It describes the target object and part before selecting relevant grid cells, using intermediate-reasoning prompting. This produces region proposals, not precise 3D action coordinates.
Mask proposal S ( I 0 ℋ , k ) and VLM 2 ( I 0 ℋ , k , ℳ S ) .
Uniform point prompts within each selected cell generate candidate masks with Segment Anything. Low-confidence and heavily overlapping masks are filtered. A second VLM query, including the earlier conversation and an overlay of the masks, chooses one part to sample.
Fine-grained Point Sampling FPS ( I 0 ℋ , m ) .
Farthest-point sampling spreads candidates through the selected mask’s 3D points. This favors spatial diversity and often includes boundaries. Geometric coverage alone does not establish robust correspondence; the next steps test each candidate.
Keypoint detection function ϕ ( k , I ) .
A reference point stores its position and visual/geometric features. Detection chooses the highest-scoring query-scene correspondence by cosine similarity. A coarse VLM region suppresses unrelated matches, so VLM guidance also participates in detection of a new scene.
For robustness, eight neighbors within two centimeters of the reference point are matched separately. If most query matches do not cluster, detection returns Null. This local consensus test rejects some ambiguous matches but can still accept a wrong, coherent cluster.
Equation 1
| (1) |
Cross-instance consistency verification.
A candidate passes if it is detected in at least 70% of the demonstration observations, using δ = 0.3. The proposal is accepted only if its passing fraction exceeds the set-level threshold γ; otherwise the loop requests a new mask and candidate set.
The paper does not give a numerical γ in this version, and the pseudocode has an unbounded repeat loop. These checks measure detection consistency over available demonstrations, rather than guarantee success on every unseen object.
Equation 2
| (2) |
II-D Learning Keypoint-Centric Trajectory Prediction Models
A trajectory-level Diffuser conditions on the sparse keypoint positions and features and predicts the manipulation motion relative to their center. This reduces dependence on background appearance and absolute translation.
The paper describes an object-relative frame, but does not fully specify its rotational construction here. Generalization to changed viewpoints and object orientation therefore remains an empirical property of the complete representation and detector, rather than a proved invariance to arbitrary rigid transformations.
II-E Implementation
The implementation uses GPT-4o, SAM, DINO visual features upsampled with FeatUp, and Fast Point Feature Histograms for local 3D geometry. Correspondence similarity weights visual and geometric cosine scores by 0.75 and 0.25, respectively.
Each predicted trajectory has 48 poses. A pose contains three position coordinates, a continuous six-dimensional rotation representation, and one gripper value. The conditional diffusion model receives matched point positions plus both feature types and is trained with a standard diffusion objective.
Equation 3
| (3) | ||||
II-F Inference-time Pipeline
At test time, detected keypoints condition denoising of several sampled trajectories. A bidirectional rapidly exploring random tree planner searches for a collision-free approach to each candidate’s initial pose. The system executes the first approachable candidate or reports failure when all candidates are exhausted.
This feasibility check concerns the approach to the learned manipulation segment. The described pipeline predicts that segment from the initial observation; it does not establish collision-free contact execution or a continuously replanned feedback policy throughout the motion.
III Experiment
Simulation isolates the data efficiency of the keypoint representation. Real-robot experiments test whether correspondence verification improves automatically proposed keypoints and downstream execution.
III-A Simulation Experiments in Meta-World
Five Meta-World tasks cover opening/closing drawers, pressing side/top buttons, and pulling a lever. Keypoints are manually marked in simulator XML and located with known object/camera poses. Object planar pose and camera elevation, azimuth, and distance vary; views where the arm initially obscures the object are excluded.
Baselines are RGB Diffuser and RGBD 3D Diffuser Actor, each also tested with added keypoint coordinates. KALM instead conditions a diffusion model on sparse 3D keypoint positions and features. This comparison combines input representation and model-design differences; it is not a test of VLM keypoint discovery in simulation.
With ten demonstrations, KALM’s success rates are 77.00%, 92.33%, 79.67%, 97.33%, and 61.67% across the five tasks. Their average is 81.6%. Adding 2D keypoints helps RGB Diffuser, but simply appending 3D keypoint coordinates does little for the tested 3D Diffuser Actor in this low-data setting. Results use 100 test episodes and three random seeds; the paper calls the reported ± terms variances without a more precise convention.
The learning curve gives KALM 97.9% at 100 demonstrations, 98.8% at 300, and 99.1% at 500. Thus it nearly saturates at 100, rather than literally reaching its maximum there as the prose suggests. 3D Diffuser reaches 91.3% at 500. The oracle-keypoint and controlled-task setting limits how broadly this data-efficiency comparison transfers.
Without Keypoints
With Keypoints
#Demos
Tasks
RGB
RGBD
RGB
RGBD
Diffuser [11]
3D Diffuser Actor [5]
Diffuser
3D Diffuser Actor
KALM (Ours)
10
DrawerOpen
DrawerClose
ButtonSide
ButtonTop
LeverPull
DrawerClose
III-B Real-World Experiment on Franka Arm
A Franka Research 3 arm with a wrist-mounted RealSense D435i performs handle lifting, top-drawer opening, and pouring into a bowl. Each skill uses ten demonstrations. Evaluation changes camera/object poses and separately uses object instances absent from diffusion training, also under pose changes.
The comparison removes consistency verification and instead asks the VLM to select eight sampled points. Both methods run on the same ten setups per task/condition, with the robot reset and moved objects approximately restored.
Under view/pose changes, KALM succeeds in 9/10, 6/10, and 8/10 trials; without verification, the counts are 1/10, 2/10, and 6/10. On unseen objects, KALM gives 6/10, 7/10, and 6/10, versus 0/10, 0/10, and 2/10. The complete table is retained. These are small per-condition trial counts, not precise estimates of deployment reliability.
The authors observe wrong-part localization and degraded trajectories in the no-verification baseline. They attribute the latter partly to inconsistent matches corrupting the training representation. The results support the loop’s usefulness on these three skills; they do not separate every contribution of perception, frame design, and trajectory learning.
| w/o. Verification | KALM (Ours) | |||
| Tasks | View | Cross Obj. | View | Cross Obj. |
| Lifting Handle | 1/10 | 0/10 | 9/10 | 6/10 |
| Opening Drawer | 2/10 | 0/10 | 6/10 | 7/10 |
| Pouring into Bowl | 6/10 | 2/10 | 8/10 | 6/10 |
IV Related Work
Related work connects spatial action abstractions, cross-object correspondence, and pretrained-model guidance.
Abstractions for action representations
Contact points, objects, parts, keypoints, and shape descriptors can provide compact policy inputs or local action frames. Examples include kPAM and part-based manipulation. KALM emphasizes acquiring useful points from existing demonstrations and pretrained models instead of additional keypoint annotation.
Finding keypoint correspondences.
Analytical and learned correspondence methods transfer behaviors between related objects, often with annotated points or additional object data. KALM uses pretrained visual and geometric features plus demonstration consistency as its selection signal.
Language/vision-language robot systems generate plans, constraints, or motions, but a plausible one-shot output can be wrong. KALM verifies a perceptual representation before learning its task motion. It still depends on pretrained models, an existing motion planner, and a task-specific diffusion policy.
V Conclusion
KALM couples semantic proposal with geometric/visual consistency checks to build a compact input for few-shot imitation. Its strongest evidence is the real-world comparison with unverified proposals, while the simulation study demonstrates the value of oracle keypoints.
Remaining practical limits include correspondence ambiguity, calibrated RGBD sensing, phase segmentation, unspecified proposal-loop limits and γ, and a trajectory sampled from the initial observation. Generalization is tested on three real skills with similar functional object shapes, not arbitrary manipulation tasks.
Acknowledgements.
Support is acknowledged from NSF, AFOSR, ONR MURI, ARO, MIT Quest for Intelligence, the MIT–IBM Watson AI Lab, ONR Science of AI, and the Simons Center for the Social Brain.