Literature
Display
← All papers

CASPER: Inferring Diverse Intents for Assistive Teleoperation with Vision Language Models

Huihan Liu · Rutav Shah · Shuijing Liu · Jack Pittenger · Mingyo Seo · Yuchen Cui · Yonatan Bisk · Roberto Martín-Martín · Yuke Zhu

· CoRL 2025 ↗

Robotics7 min text · CondensedOriginal paper ↗
Contents

Overview

Casper combines background intent inference with user-approved autonomous skills for assistive mobile manipulation.

At a glance

Casper observes short teleoperation histories and scene images, proposes a skill and target object using a vision–language model, and waits for user confirmation before executing. Open-vocabulary perception and eight parameterized skills support navigation, object manipulation, and building interactions. A study with thirteen able-bodied participants reports 88.9% average success versus 76.4% for direct teleoperation, alongside lower reported workload and higher satisfaction. These results cover three structured task families, not unrestricted assistance or clinical use.

1 Introduction

Full teleoperation gives fine control but requires sustained effort; autonomy can reduce that effort while misunderstanding a person’s goal. Casper interprets motion in context—for example, a container and a nearby laptop may both lie near the gripper, but only one is a plausible pouring target. It runs perception and intent inference while the user continues controlling the robot, proposes help only after repeated predictions agree, and requires explicit confirmation. The system broadens object choices while retaining a fixed skill repertoire.

Refer to caption
Figure 1: Casper infers user intents and offers help when confident. Given user teleoperation input, Casper uses VLMs to predict human intent using commonsense reasoning. Upon user confirmation, Casper performs autonomous execution to fulfill the intent using a skill library. Casper’s background reasoning runs in parallel with foreground human control to minimize disruption.

2 Related Work

Classical shared-control methods infer fixed goals from proximity or action history, while learned interfaces often depend on task-specific demonstrations. Foundation models provide contextual priors about objects and affordances. Casper combines these priors with mobile-manipulation skills, following systems such as Bumble [18], and uses repeated model predictions inspired by self-consistency [19]. Agreement is a practical abstention mechanism, not calibrated confidence in human intent.

3 Assistive Teleoperation with Casper

Two asynchronous components infer intentions in the background and execute approved parameterized skills. Human control remains the foreground activity until the user accepts a proposal.

Refer to caption
Figure 2: Casper architecture. VLM-based intent inference runs in parallel with human teleoperation. Casper generates task candidates from observations and infers intent from user inputs among the task candidates, repeating until predictions are self-consistent. Once confirmed by the user, Casper executes the corresponding skill with estimated parameters.

3.1 Problem Formulation

Each subtask’s intent is a pair: a skill from the available library and a detected target object. Inputs include image observations, gripper state, base and end-effector poses, and recent teleoperation. The system alternates a human-controlled trajectory snippet with robot execution of a confirmed intent. This formulation excludes intentions requiring arbitrary new skills, exact motion paths, or expressive styles.

3.2 Inferring Intents in the Background

One model call constructs plausible object–skill candidates from scene affordances and robot state. Another examines subsampled observation history, robot poses, object marks, gripper masks, and motion arrows to select a candidate. This makes interpretation context-dependent rather than simply choosing the nearest object.

Multiple parallel predictions must agree at least as often as a chosen threshold before the system offers help. Otherwise it remains silent while the user continues. The paper gives the general vote rule but does not specify numerical values for the number of calls or threshold. Parallel execution reduces blocking, although inference latency and repeated-call cost remain; agreement can also reinforce a shared model error.

3.3 Fulfilling Intents with Skill Execution

An audio cue announces the proposed action. The user accepts or rejects it by keyboard; acceptance transfers control, while rejection resumes teleoperation until another proposal. A parameter-estimation model identifies the relevant object or pose for a predefined skill. After execution, another cue returns control to the user. Navigation, pick/place/pour, and door or button interactions are composable primitives, not newly learned behaviors at runtime.

Refer to caption
Figure 3: Toy, Shelf, and Door: multi-step mobile manipulation tasks. At each step, the robot disambiguates user intent among multiple plausible goals, selecting the correct one based on user inputs and visual context.

4 Experiments

Evaluation separates end-to-end human task performance from offline intent classification and component ablations. The task-success and intent-classification rates answer different questions.

4.1 User Study: Real-World Mobile Manipulation Tasks

Thirteen participants, mean age 29.4, five women and eight men, practiced then used methods in randomized order on a dual-arm TIAGo robot with a SpaceMouse and live camera view. All were able-bodied. The three task families involve shelf retrieval and pouring, toy transport through a door, and selecting a door-opening method. Baselines are direct teleoperation, proximity-based HAT, and Bayesian RBII. The assistive baselines receive Casper’s skill prediction and share its execution machinery, so they primarily test alternative target selection.

Average success is 88.9% for Casper, 76.4% for direct teleoperation, 40.3% for HAT, and 45.0% for RBII. Reported mean completion times are 218.5, 256.0, 248.0, and 253.1 seconds respectively. Casper’s per-task successes are 83.3%, 91.2%, and 91.2%. The trial denominators and treatment of failures in timing are not fully explained; these percentages should not be treated as thirteen independent Bernoulli trials per task.

Pairwise tests report success and satisfaction improvements at p<0.05. Workload improvements have exceptions: perceived performance is not significantly different from direct teleoperation, and mental/physical demand are not significantly different from HAT. Two satisfaction questions are inapplicable to direct teleoperation despite the prose’s broad claim about all ten measures. Multiple-comparison adjustment and confidence intervals are not reported, limiting the precision of the statistical claims.

Task Success Rate (% ↑\uparrow)
Full Teleop HAT RBII Casper
Shelf 75.075.0 8.38.3 44.444.4 83.3
Toy 79.279.2 37.537.5 33.333.3 91.2
Door 75.075.0 75.075.0 57.157.1 91.2
Average 76.476.4 40.340.3 45.045.0 88.9
Task Completion Time (s ↓\downarrow)
Full Teleop HAT RBII Casper
Shelf 256.5256.5 225.0225.0 252.4252.4 196.1
Toy 391.2391.2 406.3406.3 388.3388.3 362.5
Door 120.4120.4 112.6112.6 118.7118.7 96.8
Average 256.0256.0 248.0248.0 253.1253.1 218.5
Table 1: User study: task success rate and completion time. Casper outperforms baselines in both task success and completion time.
Original figure
Figure 4: User study: user workload and user satisfaction. Casper consistently outperforms the baselines in terms of user workload (left) and user satisfaction (right) with statistical significance (p<0.05p<0.05). Detailed per-task results and full questions of user satisfaction can be found in Appendix C. Note that for user satisfaction scores, “assist helpfully” and “correct intent” are not applicable to Full Teleop.
Original figure
Figure 5: Quantitative results from unit testing and ablation studies. Left: Casper outperforms all baselines in intent inference success rate. Note that no STD is reported for deterministic baselines. The ablation of Casper vs. Capser - No Visual Prompting (VP) highlights the benefit of visual prompting. Middle: Success rates improve with longer teleoperation history. Right: Removing confidence estimation increases false prediction rates across all history lengths.

4.2 Unit Testing: Intent Inference Accuracy

Offline segments are classified correctly only when both the skill and target object match. Casper exceeds proximity-based HAT and two Bayesian variants: RBII-1 uses gripper-to-goal distance, while RBII-2 also models joystick actions with a Boltzmann-rational likelihood. The live study uses RBII-1. Examples show contextual reasoning choosing a pan for pouring despite motion passing closer to an unsuitable box. The paper does not provide a complete segment-count or repeated-sampling protocol for this unit test.

Refer to caption
Figure 6: Unit testing visualization.

4.3 Ablation Studies

Visual prompting increases average intent accuracy by 5.7 percentage points, with a 9.8-point gain on Toy; without it, the VLM variant still exceeds non-VLM baselines by more than ten points. Longer histories improve classification, with diminishing gains beyond 100 timesteps, the chosen study setting. These timesteps are not converted into seconds by a specified sampling rate. Confidence filtering lowers the fraction of emitted predictions that are wrong, but the figure does not fully quantify the corresponding abstention coverage, delay, or user effort.

Refer to caption
Figure 7: Confidence estimation visualization. Casper predicts until the intent is clearer, ensuring more accurate assistance.

5 Conclusion

Contextual intent inference and user-approved skill execution improve the tested shared-control tasks. Continual skill acquisition and stronger uncertainty calibration, including conformal prediction, remain proposed extensions. The present system still relies on a fixed library, model-dependent object understanding, and reliable execution of the chosen skill.

6 Limitations

The study does not establish benefits for users with motor or cognitive disabilities, who were not included. The system does not yet learn new skills from interaction and cannot represent fine-grained path, style, or expressive intent. Generalization beyond the tested objects, rooms, interface, and task families remains to be evaluated.

Acknowledgments

The acknowledgments credit study participants, hardware and research collaborators, and United States and Korean funding programs.

Appendix A Methods

The methods appendix preserves the complete eight-skill interface and the two original prompts used for candidate generation and intent selection.

A.1 Parametrized Skill Library.

The eight skills are pick, place, pour, go to a landmark, navigate near an object, tap a card reader, push a door, and press a button. Arm-only manipulation requires an object already within reach; navigation moves the base before manipulation when needed. Grounded SAM [73] and GroundingDINO [74] provide perception; inverse kinematics controls the arm and occupancy-map planning controls the base. Adding a skill requires an implementation, parameter mapping, and textual description, rather than automatically acquiring motor competence from a description alone.

skill_name: pick_up_object
arguments: object_of_interest
description: pick_up_object skill moves its arms to pick up the object specified in the argument object_of_interest. The pick_up_object skill can only pick up objects within the reach of its arms and does not control the robot base.

skill_name: place_object
arguments: object_of_interest
description: place_object skill moves its arms to place what it is holding to the object specified in the argument object_of_interest. The place_object skill can only place objects within the reach of its arms and does not control the robot base. The robot should place objects onto containers like trash cans, sinks, or items with surfaces like chairs, tables, etc.

skill_name: tap_card_open_door
arguments: None
description: Opens the door by tapping the key card access, if key card access is needed.

skill_name: goto_landmark
arguments: Selected landmark image from the environment from various options.
description: Navigates to the landmark in the environment, example, bedroom, kitchen, tool shop, etc.

skill_name: navigate_to_point_on_ground
arguments: object
description: Moves the robot to a point near the selected object. This skill can be used to move to a point in the room to perform a task, example, navigating near the toaster to make a toast. You should use this skill if there is still some distance between the robot and the object of interest, e.g. there is some area on the floor in between, or there is some hallway. Do not use this if the robot is near and can reach the object directly.

skill_name: push_open_door
arguments: None
description: Opens the door if the robot is in front of the door. The robot moves forward to open the door.

skill_name: pour_object
arguments: object_of_interest
description: pour_object skill moves its arms to pour whatever it is holding to the object (containers) specified in the argument object_of_interest. The pour_object skill can only pour to objects (containers) within the reach of its arms and does not control the robot base.

skill_name: press_button
arguments: button position depending which button you want to press.
description: Equips the robot with the capability of pressing a button. The robot will push the button selected in the argument. The subtask must indicate which button to press, example, 'Press the accessibility button to open the door.'

A.2 Intent Inference Details.

The candidate prompt filters manipulation to objects within 0.7 meters, uses execution history to avoid repeated completed actions, and produces multiple-choice tasks. The selection prompt combines those choices with motion, marked gripper position, and prior success or failure, requesting a skill/subtask response. Its examples encode specific priors about containers, navigation, grasp reach, and rejected actions. The exact original prompts are retained, including the skipped example option and the nominal JSON schema written with non-JSON quote conventions; those are source artifacts, not corrected executable templates.

First, give a list of possible tasks to perform, using the information of the scene, the relevant objects, and relevant skills. Note: - The robot can only manipulate objects that are within 0.7 meters of the robot. If the distance from the robot to the object is greater than 0.7 meters, then you SHOULD NOT include the manipulation skills in the task choices! - The pick up and place skill should be used on smaller objects, and the navigate skill should be used on furniture, like tables, chairs, etc.
You should use the robot history: Eliminate the tasks that the robot has already performed. If the robot has picked up an object, it will not perform the task again! Formulate your results in the format of multiple-choice questions. Example 1: Given that I am farther away and the robot is moving, the possible subtasks to perform are: A) Navigate to the desk with pens on top of it. B) Navigate to the brown colored door. Example 2: Given that I am near the table, the possible subtasks to perform are: A) Place the apple in the pink bowl. B) Pick up the screwdriver with blue handle. Example 3: Given that I am near the table, the possible subtasks to perform are: A) Pick up the blue bowl with pink stripes. B) Pick up the apple. D) Pick up the purple bowl. Example 4: Given that I am in the corridor, the possible subtasks to perform are: A) Go to the kitchen. B) Go to the classroom. INSTRUCTIONS: You are given a sequence of images of the scene. The images are taken from the camera on a mobile robot that is moving its base. Your goal is to determine the robot’s intent based on this sequence of robot observations. You want to make use of the list of skills, the history of the robot’s movement, and the list of task choices to determine the human’s goal. The list of skills that the robot has are below. The tasks are using the skills listed here. {skill description} HISTORY OF PAST EXECUTIONS: You should make use of this information for decision making. {history prompt} Possible task choices: {subtask prompt} Think step by step, keep in mind the following points: 1. Consider the given task choices. 2. Focus on the images, and see if there is a change in robot’s point of view; see how it is moving and changing its position, or if the gripper is getting closer to one of the objects, or turning towards one of the landmarks. If the robot gripper is moving, see where the gripper (as masked in the image) is moving towards based on the green arrow, and use that to determine the task choice option. Then, given the images and the robot’s movement, summarize the previous the robot’s movement. 3. Then, summarize the previous executions made by the robot and feedback received from the human or environment. Finally, answer: What is the robot trying to do? Choose from the list of possible task choices. Example reasoning 1: The robot is moving towards the left, where there is a table with a bowl on it. Since it has already picked up an object, it most likely wants to place the object on the bowl. However, the distance to the table is farther for the robot (greater than 0.7 meters) to place the object. We should first navigate to the table with a bowl on it. Example reasoning 2: The robot arm is moving closer towards the apple. The apple is already within the reach of the robot, that is, less than 0.7 meters. Therefore, it is likely that the robot will pick up the apple. Example reasoning 3: The robot is moving towards the bookshelf with a book in its hand. It is most likely trying to place the book on the bookshelf. However, the robot is far away from the bookshelf. We should first navigate to the bookshelf. Example reasoning 4: The robot is moving towards the table which has a book on it. The robot tried to pick up the book before, but it failed due to IK solver issues. Since the robot is far away from the table, we should first navigate to the table with a book on it using the navigate skill. Example reasoning 5: The robot is near the book shelf which has one thriller and one comedy book. The robot tried to pick up the comedy book but the human stopped it. It is likely that the robot will try to pick up the thriller book. Example reasoning 6: The robot is moving towards the bookshelf with a book in its hand. The robot tried to place the book on the book holder, but it failed due to IK solver issues. Since the robot is far away from the book holder, we should first navigate to the bookshelf using the navigate skill. Example reasoning 7: The robot is holding a bottle in its hand. Given that there are several cups and containers on the table, It is likely that the robot will pour the liquid from the bottle into one of the cups or containers. Provide the skill name in a valid JSON format. Your answer at the end in a valid JSON of this format: {{’subtask’: ”, ’skillname’: ”}} Avoid using the object id in the final JSON response. Describe the object(s) involved in the sub-task instead of using the object id in the JSON response. This is very important. You should only choose from the list of task choices provided! This is very important. If the arm is moving, you should see where the arm GRIPPER TIPS is moving towards, and use that to determine the task choice!! The gripper tip consists of 2 pointly black parts at the end of the gripper, don’t consider the white part. For example, if the gripper is mostly staying on the table level or below the table, then most likely the user is choosing objects on the table (i.e. the bottom row). Else if the gripper moves above the table and obstructs the objects on the table, then most likely the user is choosing something above the table on the top row. You should judge the physical distance between the robot and the object. You can tell that by checking if there is some area (like ground, floor) in between robot and the object. If the robot is far away from the object, it will most likely perform NAVIGATION. It is unlikely that the robot will do tasks that involve touching the object if the robot is far away from the object, e.g. pick up, place, pressing button, tapping card, etc. Pay attention to where the MASKED gripper is moving, and the direction of the arrow of the robot arm’s movement! The arrow means the direction of the gripper’s movement. For example, if the arrow is pointing up and right, it means the gripper is moving to something on the top right. ANSWER: Let’s think step by step.

Appendix B Experiments Details

The experimental appendix describes the robot, operator interface, sensing, and task sequences used in the human study.

B.1 Details on Experiment Setup

Operators use a SpaceMouse and view a live image stream on a laptop. The head-mounted RGB-D camera supplies system observations; audio generated through a text-to-speech service announces assistance and control changes, and keyboard input confirms intent. GPT-4o performs vision–language inference. Alternative models are suggested as compatible but are not evaluated here.

B.2 Tasks

Shelf consists of choosing and picking a pasta jar, navigating to a table, and pouring into a selected container. Toy adds navigation to and opening of a door before navigating to a destination and placing the toy. Door combines navigation with one of several opening mechanisms. The experiments span more than twenty objects, seven furniture types, and five rooms. These choices provide ambiguity within designed tasks rather than an unrestricted open-world task distribution.

Appendix C More Human Study Results

Per-task workload and satisfaction plots are retained alongside all original interview panels. Participants often found Casper easier and more reliable, while direct teleoperation offered precision at greater effort. Errors from the simpler assistive baselines reduced trust, particularly when motion was unexpected. These are selected qualitative accounts rather than a quantified prevalence of each experience.

Casper also had failures despite correct intent recognition, including inaccurate targeting, awkward recovery states, and slow transitions or button-related inference. Thus prediction accuracy alone does not resolve execution quality or interaction timing. The appendix reports eight pilot participants whereas the introduction reports ten; the thirteen-person main study is consistent. The reported eighty-plus interaction hours are not a count of independent evaluation trials.

Original figure
Figure 8: Per-task results for NASA-TLX and user satisfaction scores. Note that for user satisfaction scores, “assist helpfully” and “correct intent” are not applicable to Full Teleop.
“… made it easiest to get the job done and I felt like I had the most success.” “… is very comfortable.” “Never misunderstand what I want to do and never make mistakes.”“The fine-grain control (e.g., press the door open, tap card) is not as smooth.” “I need to do all the control work by myself for [Full Teleop].” “ … takes a lot of human effort and is overall slowest to operate.” “… was annoying to microcontrol the robot but it worked fine.” “By self-controlling, the robot moves very slow.” “… is more precise but demanding.”“… was a bit alarming.” “Every time it will make some mistakes or misunderstand what I want to do.” ”[HAT] was terrible.” “[Casper] is the best, [Full Teleop] is the worst. [HAT] is ok, but I don’t want to try again.” “The inferred intent is always wrong, and the control over the robot actions are always weird and potentially have security issues … Would it accelerate towards me and hit me?” “It was not that bad, it was in the midrange. I was able to complete one task completely and the other one was with difficulties.” “It does try to do the action, but it performs the wrong actions (e.g., press the wrong spot).” “Although the robot knew what I was going to do, it sometimes failed in finishing the task and led to a state where it was harder to control the robot.” “Cannot aim at a specific object well.”“It was a bit slow while I was in control.” “Waiting for the robot to respond and move because it’s slow.” “The inference for pressing the door open is not very prompt.”
Concept demonstration

Open image ↗