Literature
Display
← All papers

KALM: Keypoint Abstraction Using Large Models for Object-Relative Imitation Learning

Xiaolin Fang · Bo-Ruei Huang · Jiayuan Mao · Jasmine Shone · Joshua B. Tenenbaum · Tomás Lozano-Pérez · Leslie Pack Kaelbling

· ICRA 2025 ↗

Robotics8 min text · CondensedOriginal paper ↗
Contents

At a glance

KALM uses a vision-language model and segmentation model to propose task-relevant object points, then keeps points whose correspondences can be found consistently across a few demonstrations. A diffusion model predicts end-effector trajectories relative to the selected points.

With ten demonstrations per real task, it improves over selecting points without correspondence verification on handle lifting, drawer opening, and pouring. The simulation study uses manually supplied oracle keypoints, so it evaluates the representation’s data efficiency separately from the automatic discovery pipeline.

I Introduction

Keypoints can remove irrelevant image detail and anchor actions to an object, but manually designing and labeling them is costly. A large model can propose a useful part, yet a plausible proposal may be difficult to localize consistently across views and objects.

KALM combines task relevance with correspondence consistency. One successful video and a task description guide proposals; the other demonstration observations test whether candidate points remain recognizable. The accepted positions and features condition a learned trajectory model. The central contribution is this proposal–verification loop plus an object-relative action representation, rather than using the language model to output motor commands directly.

Refer to caption
Fig. 1: Keypoint Abstraction using Large Models for Object-Relative Imitation Learning (KALM). KALM is a framework that distills keypoint abstraction by prompting and verifying keypoint proposals from large pre-trained models using a small amount of robot demonstration data, which is used to train a keypoint-conditioned policy model. Our method demonstrates strong generalization on multiple real-world manipulation tasks with only 10 demonstrations and no additional labeling effort.
Refer to caption
Fig. 2: KALM overview. (a) Keypoint distillation. Given a demonstration video and a task description, we prompt a VLM to generate a coarse-grained region proposal, which is refined into a fine-grained point set via image segmentation models and VLMs. We use a keypoint detection function ϕ\phi to identify keypoint correspondences across a handful of demonstration trajectories. The final keypoints set is selected based on correspondence consistency verification. These keypoints are used for training a keypoint-conditioned action model. (b) Inference time. Given a new scene, the keypoint detection function ϕ\phi localizes the distilled keypoints. The learned keypoint-conditioned action prediction model generates an object-relative end-effector trajectory based on the keypoint positions and features.

II Method

Training first selects keypoints, then fits a trajectory model. At inference, the system detects corresponding points in the new scene, samples a relative manipulation trajectory, transforms it into the world frame, and plans an approach to its starting pose.

II-A Problem Formulation

Each skill requires a successful video, a language description, and roughly five to ten demonstrations. Each demonstration includes a calibrated initial RGBD observation and joint trajectory. No additional keypoint labels are required by the real-world discovery procedure.

The method assumes each demonstration has been split into a free-motion approach and a manipulation phase, for example using distance to a keypoint. The learned model predicts only the manipulation segment; a separate planner handles the approach. Thus the input includes calibration and skill segmentation as well as raw demonstrations.

II-B Keypoint Desiderata

A useful point must identify an object part needed for the task and remain recognizable across the training observations. These requirements address different failure modes.

Task relevance.

The language model judges task relevance from the successful video and description. For lifting a coffee-machine handle, the handle is a better reference than the water reservoir. The relevance score is conceptual: the implementation uses prompted proposals, rather than a separately calibrated numerical relevance model.

Cross-instance consistency.

Consistency asks whether the detector can locate the same functional point under different viewpoints, poses, and object shapes. Distinctive corners can be easier to match than textureless surfaces. Successful matching is a proxy for stable identity; it does not independently establish that a consistently matched point is causally sufficient for the task.

II-C Keypoint Proposal and Verification

The loop selects a coarse image region, refines it to an object-part mask, samples 3D points, and tests their matches across demonstrations. If too few candidates pass, it discards the set and asks for another proposal. Algorithm 1 retains the complete procedure.

Coarse-grained region proposal VLM 1 ​ ( ℋ , d ) .

The VLM sees the demonstration frames, task description, and a labeled grid over the initial image. It describes the target object and part before selecting relevant grid cells, using intermediate-reasoning prompting. This produces region proposals, not precise 3D action coordinates.

Algorithm 1 Keypoint Proposal and Verification Pipeline
1: Skill description dd, training set 𝒟={⟨Ii,τi⟩}i=0N−1{\mathcal{D}}=\{\langle I_{i},\tau_{i}\rangle\}_{i=0}^{N-1}, demonstration video ℋ={Itℋ}t=0|ℋ|−1{\mathcal{H}}=\{I_{t}^{\mathcal{H}}\}_{t=0}^{|{\mathcal{H}}|-1}
2: Set of proposed keypoints 𝒦{\mathcal{K}} and their matched point in the training set 𝒫:=ϕ⁡(k,Ii),∀k∈𝒦,i∈Ii{\mathcal{P}}:=\phi(k,I_{i}),\forall k\in{\mathcal{K}},i\in I_{i}
3: while 𝐓𝐫𝐮𝐞\mathbf{True} do
4:   𝒦←∅\mathcal{K}\leftarrow\emptyset; 𝒫←∅\mathcal{P}\leftarrow\emptyset
5:   𝒦VLM←VLM1​(H,d)\mathcal{K}_{\textit{VLM}}\leftarrow\textit{VLM}_{1}(H,d) ⊳\triangleright Region proposal
6:   ℳs←∪k∈𝒦VLMS(I0ℋ,k){\mathcal{M}}_{s}\leftarrow\cup_{k\in{\mathcal{K}}_{\textit{VLM}}}S(I_{0}^{\mathcal{H}},k) ⊳\triangleright Image segmentation
7:   m←VLM2​(I0ℋ,k,ℳS)m\leftarrow\textit{VLM}_{2}(I_{0}^{\mathcal{H}},k,{\mathcal{M}}_{\textit{S}}) ⊳\triangleright Mask selection
8:   𝒦FPS←FPS​(I0ℋ,m){\mathcal{K}}_{\textit{FPS}}\leftarrow\textit{FPS}(I_{0}^{\mathcal{H}},m) ⊳\triangleright Fine-grained point sampling
9:   for each k′∈𝒦FPSk^{\prime}\in{\mathcal{K}}_{\textit{FPS}} do
10:    if 1N∑i=0N−1𝟙[ϕ(𝕜′,𝕀𝕚)≠𝐍𝐮𝐥𝐥]≥𝟙−δ\frac{1}{N}\sum_{i=0}^{N-1}\mathbbb{1}\left[\mathbb{\phi}(k^{\prime},I_{i})\neq\mathbf{Null}\right]\geq\bm{1}-\delta then
11:      𝒦=𝒦∪{k′}{\mathcal{K}}={\mathcal{K}}\cup\{k^{\prime}\} ⊳\triangleright Add k′k^{\prime} to the output set
12:      𝒫=𝒫∪{ϕ⁡(k′,Ii)∣∀Ii}{\mathcal{P}}={\mathcal{P}}\cup\{\phi(k^{\prime},I_{i})\mid\forall I_{i}\}      
13:   if |𝒦||𝒦FPS|≥γ\frac{|{\mathcal{K}}|}{|{\mathcal{K}}_{\textit{FPS}}|}\geq\gamma then
14:    𝐛𝐫𝐞𝐚𝐤\mathbf{break}   

Mask proposal S ​ ( I 0 ℋ , k ) and VLM 2 ​ ( I 0 ℋ , k , ℳ S ) .

Uniform point prompts within each selected cell generate candidate masks with Segment Anything. Low-confidence and heavily overlapping masks are filtered. A second VLM query, including the earlier conversation and an overlay of the masks, chooses one part to sample.

Fine-grained Point Sampling FPS ​ ( I 0 ℋ , m ) .

Farthest-point sampling spreads candidates through the selected mask’s 3D points. This favors spatial diversity and often includes boundaries. Geometric coverage alone does not establish robust correspondence; the next steps test each candidate.

Keypoint detection function ϕ ⁡ ( k , I ) .

A reference point stores its position and visual/geometric features. Detection chooses the highest-scoring query-scene correspondence by cosine similarity. A coarse VLM region suppresses unrelated matches, so VLM guidance also participates in detection of a new scene.

For robustness, eight neighbors within two centimeters of the reference point are matched separately. If most query matches do not cluster, detection returns Null. This local consensus test rejects some ambiguous matches but can still accept a wrong, coherent cluster.

Equation 1
ϕ∗​(p,p0k)=⟨Fq​(p),Fref​(p0k)⟩,\phi^{*}(p,p^{k}_{0})=\langle F_{\textit{q}}(p),F_{\textit{ref}}(p^{k}_{0})\rangle, (1)

Cross-instance consistency verification.

A candidate passes if it is detected in at least 70% of the demonstration observations, using δ = 0.3. The proposal is accepted only if its passing fraction exceeds the set-level threshold γ; otherwise the loop requests a new mask and candidate set.

The paper does not give a numerical γ in this version, and the pseudocode has an unbounded repeat loop. These checks measure detection consistency over available demonstrations, rather than guarantee success on every unseen object.

Equation 2
Φ(k):=1N∑i=0N−1𝟙[ϕ(𝕜,𝕀𝕚)≠𝐍𝐮𝐥𝐥]≥𝟙−δ,\Phi(k):=~\frac{1}{N}\sum_{i=0}^{N-1}\mathbbb{1}\left[\mathbb{\phi}(k,I_{i})\neq\mathbf{Null}\right]\geq 1-\delta, (2)

II-D Learning Keypoint-Centric Trajectory Prediction Models

A trajectory-level Diffuser conditions on the sparse keypoint positions and features and predicts the manipulation motion relative to their center. This reduces dependence on background appearance and absolute translation.

The paper describes an object-relative frame, but does not fully specify its rotational construction here. Generalization to changed viewpoints and object orientation therefore remains an empirical property of the complete representation and detector, rather than a proved invariance to arbitrary rigid transformations.

II-E Implementation

The implementation uses GPT-4o, SAM, DINO visual features upsampled with FeatUp, and Fast Point Feature Histograms for local 3D geometry. Correspondence similarity weights visual and geometric cosine scores by 0.75 and 0.25, respectively.

Each predicted trajectory has 48 poses. A pose contains three position coordinates, a continuous six-dimensional rotation representation, and one gripper value. The conditional diffusion model receives matched point positions plus both feature types and is trained with a standard diffusion objective.

Equation 3
⟨Fq​(p),Fref​(p0k)⟩\displaystyle\langle F_{\textit{q}}(p),F_{\textit{ref}}(p^{k}_{0})\rangle =λ1⋅⟨FqDINO​(p),FrefDINO​(p0k)⟩\displaystyle=\lambda_{1}\cdot\langle F^{\textit{DINO}}_{\textit{q}}(p),F^{\textit{DINO}}_{\textit{ref}}(p^{k}_{0})\rangle (3)
+λ2⋅⟨FFPFHq(p),FFPFHref(pk0)⟩,\displaystyle+\lambda_{2}\cdot\langle F^{\textit{FPFH}}_{\textit{q}}(p),F^{\textit{FPFH}}_{\textit{ref}}(p^{k}_{0})\rangle,

II-F Inference-time Pipeline

At test time, detected keypoints condition denoising of several sampled trajectories. A bidirectional rapidly exploring random tree planner searches for a collision-free approach to each candidate’s initial pose. The system executes the first approachable candidate or reports failure when all candidates are exhausted.

This feasibility check concerns the approach to the learned manipulation segment. The described pipeline predicts that segment from the initial observation; it does not establish collision-free contact execution or a continuously replanned feedback policy throughout the motion.

III Experiment

Simulation isolates the data efficiency of the keypoint representation. Real-robot experiments test whether correspondence verification improves automatically proposed keypoints and downstream execution.

III-A Simulation Experiments in Meta-World

Five Meta-World tasks cover opening/closing drawers, pressing side/top buttons, and pulling a lever. Keypoints are manually marked in simulator XML and located with known object/camera poses. Object planar pose and camera elevation, azimuth, and distance vary; views where the arm initially obscures the object are excluded.

Baselines are RGB Diffuser and RGBD 3D Diffuser Actor, each also tested with added keypoint coordinates. KALM instead conditions a diffusion model on sparse 3D keypoint positions and features. This comparison combines input representation and model-design differences; it is not a test of VLM keypoint discovery in simulation.

With ten demonstrations, KALM’s success rates are 77.00%, 92.33%, 79.67%, 97.33%, and 61.67% across the five tasks. Their average is 81.6%. Adding 2D keypoints helps RGB Diffuser, but simply appending 3D keypoint coordinates does little for the tested 3D Diffuser Actor in this low-data setting. Results use 100 test episodes and three random seeds; the paper calls the reported ± terms variances without a more precise convention.

The learning curve gives KALM 97.9% at 100 demonstrations, 98.8% at 300, and 99.1% at 500. Thus it nearly saturates at 100, rather than literally reaching its maximum there as the prose suggests. 3D Diffuser reaches 91.3% at 500. The oracle-keypoint and controlled-task setting limits how broadly this data-efficiency comparison transfers.

Without Keypoints With Keypoints
#Demos Tasks RGB RGBD RGB RGBD
Diffuser [11] 3D Diffuser Actor [5] Diffuser 3D Diffuser Actor KALM (Ours)
10 DrawerOpen 30.00​(±8.29)30.00(\pm 8.29) 30.00​(±6.53)30.00(\pm 6.53) 62.00​(±1.41)62.00(\pm 1.41) 29.33​(±4.64)29.33(\pm 4.64) 77.00​(±2.94)77.00(\pm 2.94)
DrawerClose 50.00​(±1.41)50.00(\pm 1.41) 53.67​(±4.19)53.67(\pm 4.19) 83.33​(±2.62)83.33(\pm 2.62) 50.67​(±6.60)50.67(\pm 6.60) 92.33​(±0.47)92.33(\pm 0.47)
ButtonSide 32.67​(±2.49)32.67(\pm 2.49) 37.67​(±2.05)37.67(\pm 2.05) 49.67​(±11.09)49.67(\pm 11.09) 38.67​(±1.25)38.67(\pm 1.25) 79.67​(±1.25)79.67(\pm 1.25)
ButtonTop 19.00​(±3.56)19.00(\pm 3.56) 21.00​(±4.90)21.00(\pm 4.90) 28.00​(±8.04)28.00(\pm 8.04) 21.00​(±4.32)21.00(\pm 4.32) 97.33​(±0.47)97.33(\pm 0.47)
LeverPull 10.67​(±3.30)10.67(\pm 3.30) 8.67​(±2.36)8.67(\pm 2.36) 21.33​(±1.70)21.33(\pm 1.70) 10.33​(±4.19)10.33(\pm 4.19) 61.67​(±6.13)61.67(\pm 6.13)

TABLE I: Few-shot learning in Meta-World. We evaluate our method, KALM, on five manipulation tasks in the Meta-World [19] simulator, using Diffuser [11] and 3D Diffuser Actor [5] as baselines, along with ablation studies on keypoint usage. Our method consistently outperforms these baselines, demonstrating that keypoints serve as an effective abstraction.
Refer to caption
(a) DrawerOpen
DrawerClose
Refer to caption
(b) ButtonSide
                                       
Refer to caption
(c) ButtonTop
                                       
Refer to caption
(d) LeverPull
                                       
Fig. 3: Testing tasks in Meta-World [19] simulator. We evaluate on 5 tasks in Meta-World with randomized camera and object poses, necessitating the generalization of policies across observational changes. Keypoints are marked in pink for visualization.
Original figure
Fig. 4: Data efficiency. We measure the average success rate across all 5 tasks, with the number of demonstrations increasing from 10 to 500. Our method, KALM, demonstrates superior data efficiency compared to all baselines.

III-B Real-World Experiment on Franka Arm

A Franka Research 3 arm with a wrist-mounted RealSense D435i performs handle lifting, top-drawer opening, and pouring into a bowl. Each skill uses ten demonstrations. Evaluation changes camera/object poses and separately uses object instances absent from diffusion training, also under pose changes.

The comparison removes consistency verification and instead asks the VLM to select eight sampled points. Both methods run on the same ten setups per task/condition, with the robot reset and moved objects approximately restored.

Under view/pose changes, KALM succeeds in 9/10, 6/10, and 8/10 trials; without verification, the counts are 1/10, 2/10, and 6/10. On unseen objects, KALM gives 6/10, 7/10, and 6/10, versus 0/10, 0/10, and 2/10. The complete table is retained. These are small per-condition trial counts, not precise estimates of deployment reliability.

The authors observe wrong-part localization and degraded trajectories in the no-verification baseline. They attribute the latter partly to inconsistent matches corrupting the training representation. The results support the loop’s usefulness on these three skills; they do not separate every contribution of perception, frame design, and trajectory learning.

Refer to caption
(a) Lifting the handle
Refer to caption
(b) Opening the drawer
Refer to caption
(c) Pouring into a bowl
Fig. 5: Testing tasks in the real world. We evaluate different methods on three tasks in the real world with different objects at different poses, and with different camera angles. The testing assets are illustrated in the figure.
w/o. Verification KALM (Ours)
Tasks View Cross Obj. View Cross Obj.
Lifting Handle 1/10 0/10 9/10 6/10
Opening Drawer 2/10 0/10 6/10 7/10
Pouring into Bowl 6/10 2/10 8/10 6/10
TABLE II: Real-world experiments success rate. We evaluate all models in varying camera and object poses (View), as well as on unseen objects (Cross obj.). Our method demonstrates strong performance under view changes and showcases generalization to objects that are not seen during diffusion training time.

IV Related Work

Related work connects spatial action abstractions, cross-object correspondence, and pretrained-model guidance.

Abstractions for action representations

Contact points, objects, parts, keypoints, and shape descriptors can provide compact policy inputs or local action frames. Examples include kPAM and part-based manipulation. KALM emphasizes acquiring useful points from existing demonstrations and pretrained models instead of additional keypoint annotation.

Finding keypoint correspondences.

Analytical and learned correspondence methods transfer behaviors between related objects, often with annotated points or additional object data. KALM uses pretrained visual and geometric features plus demonstration consistency as its selection signal.

Language/vision-language robot systems generate plans, constraints, or motions, but a plausible one-shot output can be wrong. KALM verifies a perceptual representation before learning its task motion. It still depends on pretrained models, an existing motion planner, and a task-specific diffusion policy.

V Conclusion

KALM couples semantic proposal with geometric/visual consistency checks to build a compact input for few-shot imitation. Its strongest evidence is the real-world comparison with unverified proposals, while the simulation study demonstrates the value of oracle keypoints.

Remaining practical limits include correspondence ambiguity, calibrated RGBD sensing, phase segmentation, unspecified proposal-loop limits and γ, and a trajectory sampled from the initial observation. Generalization is tested on three real skills with similar functional object shapes, not arbitrary manipulation tasks.

Acknowledgements.

Support is acknowledged from NSF, AFOSR, ONR MURI, ARO, MIT Quest for Intelligence, the MIT–IBM Watson AI Lab, ONR Science of AI, and the Simons Center for the Social Brain.

Concept demonstration

Open image ↗