Overview
Casper combines background intent inference with user-approved autonomous skills for assistive mobile manipulation.
At a glance
Casper observes short teleoperation histories and scene images, proposes a skill and target object using a vision–language model, and waits for user confirmation before executing. Open-vocabulary perception and eight parameterized skills support navigation, object manipulation, and building interactions. A study with thirteen able-bodied participants reports 88.9% average success versus 76.4% for direct teleoperation, alongside lower reported workload and higher satisfaction. These results cover three structured task families, not unrestricted assistance or clinical use.
1 Introduction
Full teleoperation gives fine control but requires sustained effort; autonomy can reduce that effort while misunderstanding a person’s goal. Casper interprets motion in context—for example, a container and a nearby laptop may both lie near the gripper, but only one is a plausible pouring target. It runs perception and intent inference while the user continues controlling the robot, proposes help only after repeated predictions agree, and requires explicit confirmation. The system broadens object choices while retaining a fixed skill repertoire.
2 Related Work
Classical shared-control methods infer fixed goals from proximity or action history, while learned interfaces often depend on task-specific demonstrations. Foundation models provide contextual priors about objects and affordances. Casper combines these priors with mobile-manipulation skills, following systems such as Bumble [18], and uses repeated model predictions inspired by self-consistency [19]. Agreement is a practical abstention mechanism, not calibrated confidence in human intent.
3 Assistive Teleoperation with Casper
Two asynchronous components infer intentions in the background and execute approved parameterized skills. Human control remains the foreground activity until the user accepts a proposal.
3.1 Problem Formulation
Each subtask’s intent is a pair: a skill from the available library and a detected target object. Inputs include image observations, gripper state, base and end-effector poses, and recent teleoperation. The system alternates a human-controlled trajectory snippet with robot execution of a confirmed intent. This formulation excludes intentions requiring arbitrary new skills, exact motion paths, or expressive styles.
3.2 Inferring Intents in the Background
One model call constructs plausible object–skill candidates from scene affordances and robot state. Another examines subsampled observation history, robot poses, object marks, gripper masks, and motion arrows to select a candidate. This makes interpretation context-dependent rather than simply choosing the nearest object.
Multiple parallel predictions must agree at least as often as a chosen threshold before the system offers help. Otherwise it remains silent while the user continues. The paper gives the general vote rule but does not specify numerical values for the number of calls or threshold. Parallel execution reduces blocking, although inference latency and repeated-call cost remain; agreement can also reinforce a shared model error.
3.3 Fulfilling Intents with Skill Execution
An audio cue announces the proposed action. The user accepts or rejects it by keyboard; acceptance transfers control, while rejection resumes teleoperation until another proposal. A parameter-estimation model identifies the relevant object or pose for a predefined skill. After execution, another cue returns control to the user. Navigation, pick/place/pour, and door or button interactions are composable primitives, not newly learned behaviors at runtime.
4 Experiments
Evaluation separates end-to-end human task performance from offline intent classification and component ablations. The task-success and intent-classification rates answer different questions.
4.1 User Study: Real-World Mobile Manipulation Tasks
Thirteen participants, mean age 29.4, five women and eight men, practiced then used methods in randomized order on a dual-arm TIAGo robot with a SpaceMouse and live camera view. All were able-bodied. The three task families involve shelf retrieval and pouring, toy transport through a door, and selecting a door-opening method. Baselines are direct teleoperation, proximity-based HAT, and Bayesian RBII. The assistive baselines receive Casper’s skill prediction and share its execution machinery, so they primarily test alternative target selection.
Average success is 88.9% for Casper, 76.4% for direct teleoperation, 40.3% for HAT, and 45.0% for RBII. Reported mean completion times are 218.5, 256.0, 248.0, and 253.1 seconds respectively. Casper’s per-task successes are 83.3%, 91.2%, and 91.2%. The trial denominators and treatment of failures in timing are not fully explained; these percentages should not be treated as thirteen independent Bernoulli trials per task.
Pairwise tests report success and satisfaction improvements at p<0.05. Workload improvements have exceptions: perceived performance is not significantly different from direct teleoperation, and mental/physical demand are not significantly different from HAT. Two satisfaction questions are inapplicable to direct teleoperation despite the prose’s broad claim about all ten measures. Multiple-comparison adjustment and confidence intervals are not reported, limiting the precision of the statistical claims.
4.2 Unit Testing: Intent Inference Accuracy
Offline segments are classified correctly only when both the skill and target object match. Casper exceeds proximity-based HAT and two Bayesian variants: RBII-1 uses gripper-to-goal distance, while RBII-2 also models joystick actions with a Boltzmann-rational likelihood. The live study uses RBII-1. Examples show contextual reasoning choosing a pan for pouring despite motion passing closer to an unsuitable box. The paper does not provide a complete segment-count or repeated-sampling protocol for this unit test.
4.3 Ablation Studies
Visual prompting increases average intent accuracy by 5.7 percentage points, with a 9.8-point gain on Toy; without it, the VLM variant still exceeds non-VLM baselines by more than ten points. Longer histories improve classification, with diminishing gains beyond 100 timesteps, the chosen study setting. These timesteps are not converted into seconds by a specified sampling rate. Confidence filtering lowers the fraction of emitted predictions that are wrong, but the figure does not fully quantify the corresponding abstention coverage, delay, or user effort.
5 Conclusion
Contextual intent inference and user-approved skill execution improve the tested shared-control tasks. Continual skill acquisition and stronger uncertainty calibration, including conformal prediction, remain proposed extensions. The present system still relies on a fixed library, model-dependent object understanding, and reliable execution of the chosen skill.
6 Limitations
The study does not establish benefits for users with motor or cognitive disabilities, who were not included. The system does not yet learn new skills from interaction and cannot represent fine-grained path, style, or expressive intent. Generalization beyond the tested objects, rooms, interface, and task families remains to be evaluated.
Acknowledgments
The acknowledgments credit study participants, hardware and research collaborators, and United States and Korean funding programs.
Appendix A Methods
The methods appendix preserves the complete eight-skill interface and the two original prompts used for candidate generation and intent selection.
A.1 Parametrized Skill Library.
The eight skills are pick, place, pour, go to a landmark, navigate near an object, tap a card reader, push a door, and press a button. Arm-only manipulation requires an object already within reach; navigation moves the base before manipulation when needed. Grounded SAM [73] and GroundingDINO [74] provide perception; inverse kinematics controls the arm and occupancy-map planning controls the base. Adding a skill requires an implementation, parameter mapping, and textual description, rather than automatically acquiring motor competence from a description alone.
skill_name: pick_up_object arguments: object_of_interest description: pick_up_object skill moves its arms to pick up the object specified in the argument object_of_interest. The pick_up_object skill can only pick up objects within the reach of its arms and does not control the robot base. skill_name: place_object arguments: object_of_interest description: place_object skill moves its arms to place what it is holding to the object specified in the argument object_of_interest. The place_object skill can only place objects within the reach of its arms and does not control the robot base. The robot should place objects onto containers like trash cans, sinks, or items with surfaces like chairs, tables, etc. skill_name: tap_card_open_door arguments: None description: Opens the door by tapping the key card access, if key card access is needed. skill_name: goto_landmark arguments: Selected landmark image from the environment from various options. description: Navigates to the landmark in the environment, example, bedroom, kitchen, tool shop, etc. skill_name: navigate_to_point_on_ground arguments: object description: Moves the robot to a point near the selected object. This skill can be used to move to a point in the room to perform a task, example, navigating near the toaster to make a toast. You should use this skill if there is still some distance between the robot and the object of interest, e.g. there is some area on the floor in between, or there is some hallway. Do not use this if the robot is near and can reach the object directly. skill_name: push_open_door arguments: None description: Opens the door if the robot is in front of the door. The robot moves forward to open the door. skill_name: pour_object arguments: object_of_interest description: pour_object skill moves its arms to pour whatever it is holding to the object (containers) specified in the argument object_of_interest. The pour_object skill can only pour to objects (containers) within the reach of its arms and does not control the robot base. skill_name: press_button arguments: button position depending which button you want to press. description: Equips the robot with the capability of pressing a button. The robot will push the button selected in the argument. The subtask must indicate which button to press, example, 'Press the accessibility button to open the door.'
A.2 Intent Inference Details.
The candidate prompt filters manipulation to objects within 0.7 meters, uses execution history to avoid repeated completed actions, and produces multiple-choice tasks. The selection prompt combines those choices with motion, marked gripper position, and prior success or failure, requesting a skill/subtask response. Its examples encode specific priors about containers, navigation, grasp reach, and rejected actions. The exact original prompts are retained, including the skipped example option and the nominal JSON schema written with non-JSON quote conventions; those are source artifacts, not corrected executable templates.
Appendix B Experiments Details
The experimental appendix describes the robot, operator interface, sensing, and task sequences used in the human study.
B.1 Details on Experiment Setup
Operators use a SpaceMouse and view a live image stream on a laptop. The head-mounted RGB-D camera supplies system observations; audio generated through a text-to-speech service announces assistance and control changes, and keyboard input confirms intent. GPT-4o performs vision–language inference. Alternative models are suggested as compatible but are not evaluated here.
B.2 Tasks
Shelf consists of choosing and picking a pasta jar, navigating to a table, and pouring into a selected container. Toy adds navigation to and opening of a door before navigating to a destination and placing the toy. Door combines navigation with one of several opening mechanisms. The experiments span more than twenty objects, seven furniture types, and five rooms. These choices provide ambiguity within designed tasks rather than an unrestricted open-world task distribution.
Appendix C More Human Study Results
Per-task workload and satisfaction plots are retained alongside all original interview panels. Participants often found Casper easier and more reliable, while direct teleoperation offered precision at greater effort. Errors from the simpler assistive baselines reduced trust, particularly when motion was unexpected. These are selected qualitative accounts rather than a quantified prevalence of each experience.
Casper also had failures despite correct intent recognition, including inaccurate targeting, awkward recovery states, and slow transitions or button-related inference. Thus prediction accuracy alone does not resolve execution quality or interaction timing. The appendix reports eight pilot participants whereas the introduction reports ten; the thirteen-person main study is consistent. The reported eighty-plus interaction hours are not a count of independent evaluation trials.