Literature
Display
← All papers

EEG-Based Brain-Computer Interface for Robotic Assistance with User Intention Prediction

Ruohan Zhang · Tasha Kim · Yingke Wang · Hanvit Cho · Alex Hodges · Jin Jie Ryan Tan · Chen Wang · Minjune Hwang · Sharon Lee · Ayano Hiranaka · Wensi Ai · Anthony Norcia · Li Fei-Fei · Jiajun Wu

Preprint posted · Research Square (preprint) ↗

Robotics9 min text · CondensedOriginal paper ↗
Contents

At a glance

NOIR-EEG combines noninvasive EEG decoding, parameterized robot skills, and learned intention suggestions. Users select an object, a skill, and its parameters; the robot executes the resulting low-level movement. Predictions from a previous demonstration can bypass much of the slow decoding process, while users retain confirmation and correction.

The preprint reports 16 participants and 15 household tasks, with performance plots evaluated over 30 trials per condition. Mean task time fell from 17.54 to 12.60 minutes with intention prediction. The evidence concerns a controlled tabletop study; it does not establish independent home use, clinical effectiveness, or universal intention decoding.

Introduction

The design avoids asking noisy, low-resolution EEG to specify every motor command. Instead, neural signals communicate a sequence of high-level, parameterized skills, and robot control supplies the detailed motion. A staged decoder combines visually evoked signals with motor imagery, while foundation models learn reusable suggestions from earlier interaction.

Complete original Figure 1
Fig. 1 a. NOIR-EEG is an EEG-based brain-computer interface with a staged human intention decoding pipeline, plus a robotic system with a library of primitive skills. b. NOIR-EEG’s staged decoding pipeline consists of (1) What object to manipulate, determined by SSVEP signals using CCA classifiers; (2) How to interact with the object, decoded from MI signals using FBCSP+SVM algorithms; and (3) Where to interact with the object, also decoded through MI signals using FBCSP+SVM algorithms.

Performing complex tasks

An object detector identifies candidates in the camera image [1]. Objects receive flicker masks at 6, 7.5, 8.57, or 10 Hz; canonical correlation with reference signals identifies which object the user attends to [5]. Four-class motor imagery then chooses a skill and moves a cursor in the image plane; two-class imagery selects height.

The system uses a 128-channel saline EEG net. Motor-imagery signals are filtered over 8–30 Hz and classified with filter-bank common spatial patterns plus a support vector machine [7]. Jaw-clench EMG confirms or rejects choices. Thus the complete interface includes a voluntary muscle signal as well as brain-signal decoding.

Task completion and decoding accuracy

Sixteen participants—11 male and five female—performed tasks spanning seven meal-preparation, four cleaning, two personal-care, and two entertainment activities. They sat separately from the Franka Panda robot. The paper reports that all task trials completed on the first attempt; this is task-level completion with confirmation and correction, not error-free decoding.

Mean completion time was 17.54 minutes. CookPasta averaged about 34 minutes and ten executed skills. Although the task description says 3–9 steps, execution can require additional skills. Decision-making and decoding occupied 63% of total time: object, skill, and parameter selection accounted for 5%, 13%, and 45%. Cursor-based parameter selection was the main bottleneck.

Mean SSVEP accuracy was 90.00% during calibration and 86.20% in testing; four-class motor-imagery accuracy was 50.10% and 49.60%. Paired participant-level tests found no significant calibration/test difference. This does not mean the approximately 50% motor-imagery decoder became highly accurate. Figure 2 retains the distributions, standard deviations, and cursor-path efficiency.

Participants sometimes adapted primitive skills creatively: pushing a wide book to the edge before side-grasping it, or repeatedly moving a knife with Push to slice a banana. These examples illustrate flexible skill composition rather than new learned low-level manipulation skills.

Complete original Figure 2
Fig. 2 NOIR-EEG’s performance. All values are reported at the trial level (N = 30). a. Average task completion time of all 15 tasks and time breakdown. “Other” includes robot execution time and system delays. b. Breakdown of time taken per step. Each data point represents the average value within a single trial. Numbers indicate median values. c. Means and standard deviations for object selection (SSVEP) and skill classification (MI) accuracy during calibration (C) and test time (T). d. Mean cursor path movement efficiency. A value of 100% indicates perfect control.

Intention prediction

After one successful demonstration, the system stores images and chosen objects, skills, and parameters. GPT-4o compares a new image with that recorded sequence and suggests the next object–skill pair. After confirmation, DINOv2 semantic correspondence proposes the parameter location [9]. Suggestions can accommodate changed object placement and appearance within the demonstrated task.

Complete original Figure 3
Fig. 3 (a) Intention learning algorithms can predict participants’ preferences for object, skill, and parameter selection based on their previous data. We use GPT-4o for object and skill selection, and DINOv2 for parameter selection. (b) An example from the task “SetTable”, given a sequence of past images and choices and the current query image, the learning algorithm predicts the object, state, and parameters. (x, y) parameter choices are indicated by the green dots.

Time and effort with prediction

Mean trial time decreased from 17.54 ± 7.07 to 12.60 ± 5.20 minutes, about a 28% reduction; the paired trial-level test reports t(29)=5.59, p<.001. Mean per-step selection times fell from 0.56 to 0.15 minutes for object–skill choice, 0.92 to 0.19 for image-plane coordinates, and 0.46 to 0.19 for height.

Median cursor travel fell from 3,382.50 to zero pixels for image-plane selection and from 847.55 to zero for height. Reported suggestion accuracy exceeded 94%, allowing many suggestions to be accepted directly. Zero median adjustment does not imply every suggested parameter was correct.

Survey medians improved from four to five on five-point scales for experience, perceived robot intelligence, and ease of providing signals. The survey labels the first trial as without prediction and the second as with prediction. The manuscript does not report counterbalancing; practice and condition order therefore remain potential contributors. Thirty trial observations should not be read as thirty independent participants.

Complete original Figure 4
Fig. 4 NOIR-EEG’s performance with intention learning and prediction. All values are reported at the trial level (N = 30). a. Trial completion time with and without intention learning and prediction. Each data point represents a single trial. Numbers indicate median values. b. Breakdown of time per skill step, with and without intention learning. Each data point represents the average value within a single trial. c. Cursor movement distance in pixels with and without learning. Each data point represents the average value within a single trial. d. User survey results on a 5-point Likert scale. Each data point represents a single trial. e. Intention learning algorithm’s performance in terms of object-skill and parameter suggestion accuracy. High suggestion accuracy means that participants can directly accept the suggestion without going through the EEG decoding process, thereby saving time and effort.

Discussion

The central contribution is an integrated interaction system: staged neural decoding makes skill selection possible, while robot intelligence reduces how often that decoder must be used. Earlier EEG robot studies focused on narrower navigation or reaching tasks [8] [10] [13]. The paper’s broader task set is not a controlled head-to-head comparison with every prior system.

The results depend on calibrated EEG hardware, a prepared laboratory, a fixed skill library, visual attention to flicker, and jaw-clench feedback. The manuscript gives no long-term home evaluation or patient-specific efficacy study. It also does not establish generalization to arbitrary unseen tasks or guarantee that confirmation catches every unsafe action.

Author contributions

Ruohan Zhang, Tasha Kim, and Yingke Wang co-led the investigation. Team members developed the decoding pipeline, robot skills, intention learning, and experimental procedures; Kim, Cho, Hodges, and Wang conducted the human experiments. Anthony Norcia, Li Fei-Fei, and Jiajun Wu jointly supervised. The preprint declares no competing interest.

Appendix A: Staged decoding

Appendix A specifies the decoders. SSVEP uses selected visual-cortex channels, motor imagery uses selected motor/visual and peripheral channels, and clench detection uses all channels. Figures 5 and 6 preserve the complete control flow and electrode maps.

Complete original Figure 5
Fig. 5 A staged decoding pipeline for interpreting human intention from EEG signals that include: What object to manipulate, determined by SSVEP signals using CCA classifiers; How to engage with the object; and Where to interact with the object, deciphered through MI signals using FBCSP+SVM algorithms. A safety mechanism monitors muscle tension from the jaw to either confirm or reject the decoding results.

SSVEP object selection

For an EEG matrix with C channels and T time samples, the decoder compares the signal against a set of candidate flicker frequencies. The attended frequency is chosen using canonical correlation with sinusoidal harmonics [4] [5].

Complete original Figure 6
Fig. 6 Map of relevant electrodes (circled in blue) we use for SSVEP (left) and motor imagery (right) decoding.

Reference signals

Each candidate frequency receives sine and cosine components through H harmonics. Samples occur at times 0, 1/f_s, …, (T−1)/f_s; the reference matrix has 2H rows and T columns. Equation 1 is retained exactly.

Equation
Complete original equation 1

Canonical correlation analysis

CCA finds projections of the EEG matrix and reference signals that maximize their correlation. The source’s normalized covariance expression is preserved in Equation 2.

Equation
Complete original equation 2

Frequency detection

The method computes one canonical correlation for each candidate frequency and selects the maximum. Equations 3–4 preserve both the frequency-specific objective and the final decision rule.

Equations
Complete original equation 3
Complete original equation 4

Motor-imagery skill selection

Filter-bank common spatial patterns separate motor-imagery classes using frequency-dependent spatial variance [7]. The appendix presents the pairwise covariance projection used within its multiclass decoding pipeline.

Band-pass filtering

The EEG signal is passed through B band-pass filters, yielding one channel-by-time matrix per frequency band.

Equation
Complete original equation 5

Covariance matrices

For each band, the spatial covariance is normalized by its trace so that overall signal magnitude does not directly determine the covariance scale.

Equation
Complete original equation 6

Spatial projection

A generalized eigenvalue problem between class covariance matrices produces spatial filters. Applying their transpose to the band-filtered EEG yields the projected signal.

Equations
Complete original equation 7
Complete original equation 8

Feature extraction

The source computes each component’s mean squared amplitude and logs its share of the total component variance. Equations 9–10 retain the exact formulas; the displayed “variance” formula does not include an explicit mean-subtraction term.

Equations
Complete original equation 9
Complete original equation 10

Feature concatenation and classification

Features from all selected bands and spatial components are concatenated, then classified by a support vector machine to predict the imagined action.

Equation
Complete original equation 11

Cursor parameter selection

Cursor selection maintains a penalty for each of four image-plane directions. Penalties discourage previously rejected directions, and a temporary boost favors a selected direction. Equation 12 gives the initial penalty state.

Equation
Complete original equation 12

Direction penalties

At each step, the penalty is reduced by a constant and clipped at zero. The prose calls this exponential decay, but Equation 13 specifies subtractive decay; the equation is preserved unchanged. Direction probabilities are divided by one plus the penalty and normalized.

Equations
Complete original equation 13
Complete original equation 14
Complete original equation 15

Direction boost

The boost also decays subtractively to zero. A favored direction’s probability is multiplied by one plus the boost, followed by normalization. The paper switches between penalized and unpenalized probability symbols in these steps; the equations and algorithm retain that notation rather than silently resolving the implementation ambiguity.

Equations
Complete original equation 16
Complete original equation 17
Complete original equation 18

Jaw-clench correction and algorithm

A detected jaw clench increases the penalty on the current direction. Algorithm 1 preserves the full selection loop, including collection, weighting, movement, decay, rejection, and final cursor return.

Equation
Complete original equation 19
Complete original algorithm 1

Clench decoder

The clench decoder distinguishes jaw-muscle tension from background activity and blink artifacts through signal variance, frequency content, spatial distribution, and duration. These are engineered EMG features measured by the EEG equipment.

Signal variance

Median channel variance supplies a global muscle-activity statistic.

Equation
Complete original equation 20

Frequency features

Low-frequency power covers 1–4 Hz, while high-frequency power covers 20 Hz and above. Their ratio includes a small denominator stabilizer.

Equations
Complete original equation 22
Complete original equation 23

Spatial features

Median temporal-channel variance supplies spatial evidence for clenching.

Equation
Complete original equation 24

Duration

The event duration is the number of samples divided by sampling frequency.

Equation
Complete original equation 25

Final classification

The binary decision requires all conditions to hold. The appendix does not give the numeric feature thresholds inside this final rule, so the equations alone do not fully specify a reproducible detector.

Equation
Complete original equation 26

Appendix B: Object–skill retrieval

Appendix B adapts retrieval-based imitation learning [19] [20] [21]. A pretrained vision-language model aligns the current scene with a single recorded task demonstration, allowing the system to propose the next demonstrated action rather than requiring a new EEG choice at every stage.

State understanding and mapping

GPT-4o first interprets gripper and top views, aligns objects with recorded object labels and scene states with skill labels, and understands the order of the demonstration. The original sample prompt is preserved, including its ellipses; the paper does not provide a fully expanded production prompt.

The gripper view image captures what is held in the robot’s gripper during the
current state. ... The top-view image captures the spatial relationships and status
of objects on the table during the current state. ... I will give you a sequence of
all the states in the current task, arranged in order. Please do the following:

1. Fully understand the object types in the gripper view images and pair them
   with the ‘obj’ text annotations. ...
2. Fully understand the objects in the top-view images, the spatial relationships
   between these objects, and the current status of each object, pairing them
   with the ‘skill’ text annotations. ...

After that, I will send you two query images. ...

State inference and retrieval

For a new image pair, the model identifies the closest recorded state and returns the object and skill at the following state, ranked by likelihood. The exact sample query prompt follows.

Identify the state index from the sequence that best matches this new query pair.
Then, return the ‘obj’ and ‘skill’ information of the next state (i.e., index +
1). ... Rank your answer by the order of possibility. ...

One-shot parameter learning

DINOv2 matches semantic features between a 360×240 demonstration image and the test image. A 3×3 neighborhood around the demonstrated point supplies the reference feature; cosine similarity finds a corresponding location.

The top view determines image-plane coordinates. A side-view match determines height by projecting onto the vertical axis associated with the predicted image-plane location, then converting pixels into robot coordinates. The paper describes a 75×100 feature map and “768 patch tokens”; it does not clarify that terminology further.

Appendix C: Experimental procedure

The Stanford institutional review board approved the study under IRB-67685. The manuscript describes hardware setup, participant instructions, calibration, and survey questions, but does not fully tabulate how the 30 trial observations are distributed across all 16 participants.

EEG acquisition and preparation

A 128-channel saline HydroCel Geodesic SensorNet streams EEG through Net Station 5.4 in an electrically and acoustically isolated lab. Preparation includes ten minutes of soaking in conductive solution, fitting the net, reducing each electrode’s impedance to at most 50 kΩ, and covering the net to slow drying. This preparation is additional to the task-completion times reported above.

Participant instructions

Participants attend to the desired flickering object. For motor imagery, they consistently imagine wrist movement or squeezing a ball for left/right classes, depressing a pedal with both feet for the leg class, and fixation for rest. Jaw tension communicates the clench signal.

Interface and calibration

Motor-imagery calibration uses a one-second warning, a 5.5-second imagery cue with the final five seconds recorded, and randomized 0.5–2-second rest. Four blocks yield twenty trials per action. Task-time skill decoding follows a two-second delay after the class-to-skill mapping appears.

Clench calibration collects three 500-millisecond trials for each of rest and clench. At decision time, written instructions and a countdown precede a 500-millisecond opportunity to respond.

Post-experiment survey

The preserved questionnaire separately rates the first and second trials for overall experience, robot intelligence, and difficulty providing signals, then invites comments. These subjective ratings complement measured time and cursor travel; they are not a direct physiological measure of cognitive workload.

Thank you for your participation!

1. Please rate your overall user experience for the two trials
   •     Trial 1: (1-extremely unsatisfied, 5-extremely satisfied)
   •     Trial 2: (1-extremely unsatisfied, 5-extremely satisfied)

2. How intelligent was the robot’s decision during the experiment?
   •     Trial 1: (1-extremely unintelligent, 5-extremely intelligent)
   •     Trial 2: (1-extremely unintelligent, 5-extremely intelligent)

3. How difficult was it for you to provide signals?
   •     Trial 1: (1-extremely difficult, 5-extremely easy)
   •     Trial 2: (1-extremely difficult, 5-extremely easy)

4. Please leave any comments you have here.

Appendix D: Task definitions

Appendix D defines success through initial and goal predicates in the BEHAVIOR Domain Definition Language [24] [25]. All fifteen original task illustrations below retain their scene examples, initial conditions, and complete goal conditions.

Original WipeSpill illustration, including initial and goal predicate definitions
Appendix D: WipeSpill. Original images and complete initial/goal predicates.
Original CollectToy illustration, including initial and goal predicate definitions
Appendix D: CollectToy. Original images and complete initial/goal predicates.
Original SweepTrash illustration, including initial and goal predicate definitions
Appendix D: SweepTrash. Original images and complete initial/goal predicates.
Original CleanBook illustration, including initial and goal predicate definitions
Appendix D: CleanBook. Original images and complete initial/goal predicates.
Original IronCloth illustration, including initial and goal predicate definitions
Appendix D: IronCloth. Original images and complete initial/goal predicates.
Original OpenBasket illustration, including initial and goal predicate definitions
Appendix D: OpenBasket. Original images and complete initial/goal predicates.
Original PourTea illustration, including initial and goal predicate definitions
Appendix D: PourTea. Original images and complete initial/goal predicates.
Original SetTable illustration, including initial and goal predicate definitions
Appendix D: SetTable. Original images and complete initial/goal predicates.
Original GrateCheese illustration, including initial and goal predicate definitions
Appendix D: GrateCheese. Original images and complete initial/goal predicates.
Original CutBanana illustration, including initial and goal predicate definitions
Appendix D: CutBanana. Original images and complete initial/goal predicates.
Original CookPasta illustration, including initial and goal predicate definitions
Appendix D: CookPasta. Original images and complete initial/goal predicates.
Original Sandwich illustration, including initial and goal predicate definitions
Appendix D: Sandwich. Original images and complete initial/goal predicates.
Original Hockey illustration, including initial and goal predicate definitions
Appendix D: Hockey. Original images and complete initial/goal predicates.
Original OpenGift illustration, including initial and goal predicate definitions
Appendix D: OpenGift. Original images and complete initial/goal predicates.
Original TicTacToe illustration, including initial and goal predicate definitions
Appendix D: TicTacToe. Original images and complete initial/goal predicates.

Robot platform and skills

The system uses a Franka arm with three RealSense cameras. Table 1 preserves nine skill families and all parameter choices. An operational-space pose controller executes waypoint trajectories through the Deoxys interface [26] [27]; numerical interpolation supplies reaching trajectories.

Complete original Table 1
Table 1 Parameterized primitive skills for the Franka robots.

Supplementary videos

The original supplement provides two PourTea demonstrations: Video 1, without intention prediction, and Video 2, with intention prediction. These links retain access to the publisher-hosted videos.

Concept demonstration

Open image ↗