Literature
Display
← All papers

An Interactive Agent Foundation Model

Zane Durante · Bidipta Sarkar · Ran Gong · Rohan Taori · Yusuke Noda · Paul Tang · Ehsan Adeli · Shrinidhi Kowshika Lakshmikanth · Kevin Schulman · Arnold Milstein · Demetri Terzopoulos · Ade Famoti · Noboru Kuno · Ashley Llorens · Hoi Vo · Katsu Ikeuchi · Li Fei-Fei · Jianfeng Gao · Naoki Wake · Qiuyuan Huang

· CVPR Workshops 2025 ↗

AI agents8 min text · CondensedOriginal paper ↗
Contents

Overview

This edition condenses the entire paper and all five appendices while retaining twelve figures, nine numbered tables with every example panel, all five displayed equations, and the references. The loose introductory image is reunited with Figure 1. The figure captions preserve the authors’ original claims; the edited text distinguishes those aspirations from the experiments actually reported.

Complete overview of agent AI domains, extracted losslessly from PDF Figure 1.
Figure 1: Overview of an Agent AI system that can perceive and act in different domains and applications. Agent AI is emerging as a promising avenue toward Artificial General Intelligence (AGI). Our model represents an initial step in the development of a model that is highly capable of human-level reasoning across many tasks and levels of granularity.

At a glance

A 277-million-parameter model combines visual encoding, language modeling, and action-token prediction, first training on robotics and gaming and then adapting to domain-specific tasks. The shared training improves several robotic, game-action, and healthcare classification measures. Results support useful transfer across these evaluated tasks, rather than human-level general intelligence or a universal deployment-ready agent.

1 Introduction

Large vision-language models are often trained on images and text without learning the consequences of actions. This work adds robot and game trajectories to connect observations, instructions, and controls. A common architecture is fine-tuned for manipulation, game-action prediction, and healthcare video understanding. Reducing hallucinations through environmental grounding is a motivation; the paper does not report a comprehensive hallucination benchmark.

2 Related Work

2.1 Foundation Models

Foundation models reuse representations learned from broad data. Prior work includes language, vision, video, and embodied models. This paper seeks broader sharing of visual, textual, and action representations while retaining task-specific adaptation.

2.2 Multimodal Understanding

Multimodal models align language with images or video, often by connecting pretrained encoders and language models. The proposed system initializes from CLIP and OPT but jointly trains the visual encoder, projection, and decoder, adding explicit action tokens.

2.3 Agent-Based AI

Agent research spans simulated and physical environments, language-guided planning, and embodied interaction. Here the contribution is a multimodal pretraining recipe and experiments across three domains, rather than a complete implementation or validation of every capability associated with general agent intelligence.

Refer to caption
Figure 2: We propose an Agent AI paradigm for supporting interactive multi-modal generalist agent systems. There are 5 main modules as shown: (1) Agent in Environment and Perception with task-planning and observation, (2) Agent learning, (3) Memory, (4) Action, and (5) Cognition and Consciousness (we use “consciousness” to imply a degree of awareness of an agent’s state and surroundings). A key difference between our approach and some previous interactive strategies is that, after training, the agent’s action will directly impact task planning, as the agent does not need to receive feedback from the environment to plan its next actions.

3 Agent Paradigm

The conceptual paradigm groups perception and task planning, learning, memory, action, and cognition. The paper uses “consciousness” narrowly for awareness of an agent’s state and surroundings; it supplies no evidence of sentience. Perception, planning, and human/environment interaction are proposed requirements for collaborative embodied systems. Figure 2’s conceptual modules should not be mistaken for five independently evaluated components of the implemented network.

Refer to caption
Figure 3: Overview of our Interactive Agent framework. Our foundation model is designed to process multi-modal information that conveys various levels of abstraction. This approach facilitates a comprehensive understanding of the context and environment, thus ensuring that actions are coherent. By training on a variety of task domains and applications, we develop a versatile foundation model that can be fine-tuned for executing optimal actions in a variety of contexts, paving the way towards generally intelligent agents.

4 Agent Foundation Model

The model learns a shared representation of observations, instructions, and low-level controls. Task-specific fine-tuning turns the pretrained model into an action predictor or a video-understanding model. Sharing components can simplify the system, but the paper does not benchmark deployment cost against a complete collection of separate specialist systems.

4.1 Model Architecture

A CLIP ViT-B/16 visual encoder (Radford et al., 2021) maps frames to features; a linear projection passes them to an OPT-125M language/action transformer (Zhang et al., 2022). Text, visual tokens, and previous actions form a causal sequence for predicting the next action. A bounded sliding window limits memory cost. Sinusoidal positions support masked visual prediction, and all components are jointly trained rather than freezing the pretrained encoders. The full model has 277 million parameters.

Equation 1
A^t=Fϕ(W,ℓ(Eθ(V1)),A1,ℓ(Eθ(V2)),A2,…,ℓ⁡(Eθ​(Vt−1)),At−1,ℓ⁡((Eθ​(Vt))).\hat{A}_{t}=F_{\phi}(W,\ell(E_{\theta}(V_{1})),A_{1},\ell(E_{\theta}(V_{2})),A_{2},\\ \dots,\ell(E_{\theta}(V_{t-1})),A_{t-1},\ell((E_{\theta}(V_{t}))). (1)
Refer to caption
Figure 4: Our Unified Tokenization Framework. We propose a general pre-training strategy for predicting input tokens. For text tokens, we use the standard language modeling task with next token prediction. For actions, we expand the vocabulary of the language model to include special “agent” tokens that represent each of the actions available to the language model. Finally, we incorporate visual tokens into our framework by training a visual encoder to predict masked visual tokens.

4.2 Pre-Training Strategy

Three objectives share each sample: next-token prediction for instructions, masked image reconstruction, and autoregressive action-token prediction. The visual objective masks 75% of patches and measures squared pixel error on masked locations. The total loss normalizes the combined terms by the token count. Four-frame windows are used for robotics and nine-frame windows for gaming. The retained equations specify the original loss and conditioning notation.

Equations 2–5
Ll​a​n​g(S)=−∑j=1|W|logpθ(wj|w<j).L_{lang}(S)=-\sum_{j=1}^{|W|}\log p_{\theta}(w_{j}|w_{<j}). (2)
Lm​a​e​(S)=∑t=1T‖u​(Vt)−u​(Dθ​(Eθ​(m​(Vt))))‖22,L_{mae}(S)=\sum_{t=1}^{T}||\textsc{u}(V_{t})-\textsc{u}(D_{\theta}(E_{\theta}(\textsc{m}(V_{t}))))||_{2}^{2}, (3)
La​c​t(S)=−∑t=1T∑i=1|At|logpθ((at)i|W,V≤t,A≤t,(at)<i).L_{act}(S)=-\sum_{t=1}^{T}\sum_{i=1}^{|A_{t}|}\log p_{\theta}((a_{t})_{i}|W,V_{\leq t},A_{\leq t},(a_{t})_{<i}). (4)
L⁡(S)=Ll​a​n​g​(S)+Lm​a​e​(S)+La​c​t​(S)|W|+∑t=0T(|Vt|+|At|).L(S)=\frac{L_{lang}(S)+L_{mae}(S)+L_{act}(S)}{|W|+\sum_{t=0}^{T}(|V_{t}|+|A_{t}|)}. (5)

5 Tasks

The tasks cover simulated manipulation, recorded game play, and hospital video understanding. The healthcare setting predicts descriptions, answers, and agitation/sedation categories; it does not control patient care.

5.1 Robotics Tasks

Language-Table supplies 4.93 million frames of simulated tabletop manipulation. The language-annotated CALVIN subset supplies 1.44 million frames of seven-degree-of-freedom robot manipulation. Both train instruction-conditioned actions, then evaluate task completion or consecutive task chains.

Refer to caption
Figure 5: Our robotics and gaming pre-training pipeline. For simplicity, we use the same notation as in Sections 4.1 and 4.2; we represent our text instruction as WW, input frames as VtV_{t}, our visual encoder and linear projection layer as EθE_{\theta} and ℓ\ell, respectively, our action and language transformer model as FϕF_{\phi}, and the predicted actions at time step tt as A^t\hat{A}_{t}.

5.2 Gaming Tasks

Minecraft contributes 4.7 million frames with recorded controls and inventory metadata; Bleeding Edge contributes 2.3 million frames from seven settings. The original demonstrations lack task instructions, so GPT-4V supplies descriptions. Minecraft inventory changes supplement visual prompting to reduce object and action misidentification. These generated labels are a source of supervision rather than verified human commands.

5.3 Healthcare Tasks

Wall-mounted cameras record ICU activity. Nurses caption short clips and document longer intervals, including RASS assessments and bed position. GPT-4 converts non-identifying captions and documentation into question-answer examples, avoiding direct patient video transfer in the described process. The paper’s privacy claim concerns that stated data-generation procedure; it does not present an independent privacy audit. Healthcare is absent from agent pretraining and used for later adaptation.

Refer to caption
Figure 6: A High-level Overview of our Healthcare Tasks. We leveraged nurse-labeled annotations to train our multimodal agent on healthcare data. To adapt our model for visual question answering, we generated additional training data with GPT-4 using the PHI-safe process shown in Appendix B.

6 Experiments

Evaluation tests pretraining, subsequent adaptation, and comparisons with domain-specific or vision-language baselines. Robotics measures task success, gaming measures held-out action-token agreement, and healthcare reports text perplexity and RASS classification accuracy; these metrics answer different questions.

6.1 Pre-training Experiments

Pretraining uses 13,416,484 sampled frames: 1,233,659 four-frame Language-Table windows, 360,566 four-frame CALVIN windows, 525,309 nine-frame Minecraft windows, and 256,867 nine-frame Bleeding Edge windows. Training runs for 100 epochs on twelve nodes with sixteen V100 GPUs each, taking 175 hours. The initial learning rate is 0.0001 with warmup and cosine decay. Color augmentation changes brightness, saturation, and hue. New tokens encode actions and trajectory boundaries: Minecraft uses 23 buttons and 100 bins per mouse axis; Bleeding Edge uses eleven buttons and discretized joystick rotations and magnitudes. Robot translations, rotations, grippers, and proprioceptive state are also discretized. Figure 7 retains the complete aggregate loss curve.

Refer to caption
Figure 7: Plot of total pre-training loss over 100 epochs.
Table 1: Results for robotics fine-tuning across tasks on CALVIN and Language-Table, along with their corresponding evaluation metrics.
CALVIN Language Table
Model 1 step 2 step 3 step 4 step 5 step Avg Lens Success Rate
MCIL 28.2 2.5 0.3 0.0 0.0 0.31 —
Ours (From Scratch) 20.6 0.8 0.0 0.0 0.0 0.214 40.0
Ours 64.8 29.0 12.3 4.7 1.9 1.127 42.0

6.2 Robotics Experiments

Robot models are fine-tuned separately on Language-Table and CALVIN, keeping the visual, language, and action losses. Image masking falls to 50% during fine-tuning and is removed at evaluation. “From scratch” in the comparisons means without the proposed agent pretraining, while retaining initialization from pretrained CLIP and OPT.

6.2.1 Language-Table

Language-Table training splits 181,020 trajectories into four-frame windows. Evaluation samples fifty trajectories for each of five subtasks and repeats three times. Agent pretraining improves success from 40.0% to 42.0%. The authors acknowledge that stronger previously published systems outperform their model; this experiment chiefly shows a modest benefit over their own baseline.

6.2.2 CALVIN

CALVIN evaluates consecutive language-instructed manipulation tasks under the ABCD-to-D setting, which includes environment D during training. Agent pretraining improves one-task success from 20.6% to 64.8% and average completed chain length from 0.214 to 1.127. The displayed MCIL baseline reaches 28.2% and 0.31, respectively. Only 1.9% of five-task chains succeed, so longer-horizon control remains difficult. The authors use the language-annotated subset, approximately 1% of the original CALVIN data, in this comparison.

Table 2: Performance metrics for gaming data. We report BLEU-4 scores for action prediction in Minecraft (abbreviated as MC), and Bleeding Edge (abbreviated as BE). We choose the last epoch for the pre-trained model and the epochs with the best validation score for the other models.
Model MC (BLEU-4)↑\uparrow BE (BLEU-4)↑\uparrow
Ours (from scratch) 0.174 0.238
Ours (pre-train only) 0.170 0.249
Ours (Pre-train and fine-tuned) 0.272 0.411
Task Text instruction Start frame Predicted Action Ground Truth Action
Minecraft the player is using an iron_sword to attack and kill pigs in a forest… [Uncaptioned image] [STARTACTION] [attack] [ENDOFACTION] [STARTACTION] [attack] [ENDOFACTION]
Bleeding Edge the player is controlling a red robot … fighting other characters [Uncaptioned image] [STARTACTION] [lockon][meleeattack] [lrot162] [lmag4] [ENDOFACTION] [STARTACTION] [lockon][meleeattack] [lrot160] [lmag4] [ENDOFACTION]
Table 3: Examples of actions predicted by our fine-tuned models for Minecraft (above) and Bleeding Edge (below). More examples are presented in Appendix E.

6.3 Gaming Experiments

One hundred held-out videos per game evaluate action prediction with BLEU-4, not online game completion or reward. Pretraining plus fine-tuning raises scores from 0.174 to 0.272 for Minecraft and from 0.238 to 0.411 for Bleeding Edge. Fine-tuning overfits within five epochs, so its reported checkpoint is chosen by validation performance; the pretrained-only model uses its final epoch. These checkpoint and metric choices matter when interpreting the transfer benefit.

6.4 Healthcare Experiments

Healthcare adaptation covers captioning, visual question answering, and RASS classification. The best RASS accuracy comes from agent pretraining, whereas the lowest text perplexity comes from a frozen CLIP/OPT baseline. Thus the results support task-dependent transfer, not uniform improvement across healthcare outputs.

Healthcare Setting

Captioning and question answering train token likelihood; RASS is a ten-class prediction from a nine-frame video. Comparisons include frozen and unfrozen CLIP/OPT with a trained connector, and the proposed temporal architecture without agent pretraining. All are trained for twenty epochs on four V100 GPUs at a 0.00004 learning rate. The agent-pretrained model reaches 95.7% RASS accuracy versus 70.3% for its architecture without agent pretraining, but its perplexity is worse: 106.3 versus 100.0. Frozen CLIP/OPT obtains the best displayed perplexity, 93.3. The source does not establish prospective clinical effectiveness or live patient-care performance.

Table 4: Performance on healthcare text generation and RASS score action recognition, along with the corresponding evaluation metrics. Agent pre-training on robotics and gaming data improves performance for action recognition, but does not improve text generation abilities.
Model Perplexity ↓\downarrow RASS Acc ↑\uparrow
CLIP + OPT (frozen) 93.3 55.4
CLIP + OPT (unfrozen) 102.7 92.6
Ours (from scratch) 100.0 70.3
Ours (Agent pre-trained) 106.3 95.7

7 Ablations and Analysis

Pretraining Loss Curves:

The total pretraining loss and five component/dataset curves decline over 100 epochs. They document optimization behavior, while downstream evaluations provide the evidence for transfer.

Comparisons with GPT-4V:

A selected Minecraft comparison shows GPT-4V producing plausible high-level advice while this model emits specific control tokens. This illustrates an interface difference and is not a comprehensive, controlled comparison of the systems’ planning abilities.

Effects of Agent Pre-Training:

The ablations favor prior agent training for game-action agreement and RASS classification, and joint adaptation over the frozen connector baseline for RASS. They also show that better action recognition does not necessarily yield better language generation. The retained tables expose this tradeoff directly.

8 Conclusion

Joint observation, language, and action training provides a reusable starting point for the evaluated tasks. The most convincing gains are in CALVIN task chains, game-action prediction, and RASS classification, with smaller Language-Table gains and no healthcare text-perplexity improvement. Generality remains bounded by the tested datasets, discretized action spaces, and task-specific fine-tuning.

9 Impact Statement

The authors identify limitations of deployment: real robots would require additional training and safety measures, and the healthcare models have not undergone live clinical validation and are not medical devices. They envision clinician-supervised assistance rather than self-prescription. Their discussion also considers gaming-related social withdrawal, addiction, and anthropomorphism. These are the paper’s stated implications, separate from its experimental claims.

Acknowledgements

The paper acknowledges its institutional support, collaborators, and contributors to the healthcare recordings and demonstrations.

Appendix A Architecture Details

The visual encoder starts from CLIP ViT-B/16 and adds temporal attention after spatial attention, using divided space-time processing. Video masking follows a tubelet pattern and an MAE decoder reconstructs masked pixels. Robotics and gaming expose a causal stream at each frame, whereas healthcare uses a video-level encoder to summarize the complete input clip.

Appendix B GPT-4 Prompting

The healthcare prompt generates questions from supplied non-identifying captions and clinical labels; Figure 8 retains the complete prompt and example output. For gaming, forty-eight numbered frames are arranged in a grid for GPT-4V, as shown in Figure 9. Figure 10 preserves both the GPT-4V response and the model’s action-token output for the same illustrated setting.

Original figure
Figure 8: Our PHI-safe GPT-4 Prompt for Generating Healthcare QA Examples. By ensuring the usage of non-identifying video captions and documentation data, we prevent any identifiable patient data leakage to GPT-4 while simultaneously generating additional visual-language training data. For the particular example shown, we use a RASS score of “0 - Alert and calm”, a caption of “The clinician is helping the patient up from the bed and then helping them walk across the room.”, and a bed angle of “ >45>45°”.
Original figure
Figure 9: Our GPT-4V prompt for games like Bleeding Edge that have 3rd person viewpoints and visually complex scenes. In order to input a large number of frames (48) to GPT-4V, we input the frames as a grid with frame numbers overlaid on each frame (as shown above).

Refer to caption Refer to caption

Figure 10: When using GPT-4V to choose actions given a history of frames, we find that it gives reasonable high-level actions but does not choose precise low-level actions, highlighting the importance of our pre-trained model.

Appendix C Pre-training Loss Curves

All five pretraining loss plots are preserved: CALVIN, Language-Table, Minecraft, Bleeding Edge, and masked image modeling. They share the 100-epoch training horizon; their different scales should not be read as directly comparable downstream task quality.

Refer to caption Refer to caption
Refer to caption Refer to caption
Refer to caption
Figure 11: Plot of all components of the training loss over the 100 epochs of pre-training.

Appendix D Gaming Task Pipeline

The gaming pipeline parallels robotics: each step combines the instruction, projected visual observations, and previous action history to predict low-level controls. Figure 12 retains the full four-stage Minecraft example.

Refer to caption
Figure 12: Our gaming pre-training pipeline. For simplicity, we use the same notation as in Sections 4.1 and 4.2; we represent our text instruction as WW, input frames as VtV_{t}, our visual encoder and linear projection layer as EθE_{\theta} and ℓ\ell, respectively, our action and language transformer model as FϕF_{\phi} and the predicted actions at time step tt as A^t\hat{A}_{t}.

Appendix E Example Outputs

The appendix retains five Language-Table and five CALVIN trajectories, with start, middle, and end frames and predicted controls. Four actor-based healthcare examples cover captioning, question answering, and RASS; these illustrative actor clips mostly receive the calm/alert label, while roughly 60% of the training data has that label. The displayed question-answer example is visibly imperfect: a question about location receives a sedation assessment. Five examples per game compare predicted and recorded controls. Small camera or joystick differences and extra actions are common, and a recorded action is not always the only valid choice. These selected demonstrations illustrate behavior and failure modes rather than establish additional aggregate performance.

Text instruction Start frame Middle frame End frame
Pull the red moon apart from the blue moon. [Uncaptioned image] →\rightarrow [Uncaptioned image] →\rightarrow [Uncaptioned image]
Push the yellow start next to the red moon. [Uncaptioned image] →\rightarrow [Uncaptioned image] →\rightarrow [Uncaptioned image]
Move the red pentagon away from the blue cube. [Uncaptioned image] →\rightarrow [Uncaptioned image] →\rightarrow [Uncaptioned image]
Move the red moon to the bottom of the yellow pentagon. [Uncaptioned image] →\rightarrow [Uncaptioned image] →\rightarrow [Uncaptioned image]
Pull the red moon to the bottom left. [Uncaptioned image] →\rightarrow [Uncaptioned image] →\rightarrow [Uncaptioned image]
Table 5: We show 5 unique demonstrations from Language Table, where our model successfully follows the text instruction. In addition to the high level instruction, we also show the low-level predicted actions of our agent above each frame.
Text instruction Start frame Middle frame End frame
Push the handle to close the drawer. [Uncaptioned image] →\rightarrow [Uncaptioned image] →\rightarrow [Uncaptioned image]
Lift the red block from the sliding cabinet. [Uncaptioned image] →\rightarrow [Uncaptioned image] →\rightarrow [Uncaptioned image]
Pull the handle to open the drawer. [Uncaptioned image] →\rightarrow [Uncaptioned image] →\rightarrow [Uncaptioned image]
Push the sliding door to the left side. [Uncaptioned image] →\rightarrow [Uncaptioned image] →\rightarrow [Uncaptioned image]
Push the sliding door to the right side. [Uncaptioned image] →\rightarrow [Uncaptioned image] →\rightarrow [Uncaptioned image]
Table 6: We show 5 unique demonstrations from CALVIN, where our model successfully follows the text instruction. In addition to the high level instruction, we also show the low-level predicted actions of our agent above each frame.
Task Start frame End frame Model Output
Video Captioning [Uncaptioned image] [Uncaptioned image] The patient is awake and calm. The patient is cooperative. The patient is alert
Video Question Answering [Uncaptioned image] [Uncaptioned image] Q: Where is the patient? A: patient is in deep sedation. The patient likely requires assistance.
Action Recognition (RASS) [Uncaptioned image] [Uncaptioned image] 0 - Alert and calm
Video Captioning [Uncaptioned image] [Uncaptioned image] The patient is awake and calm. They are speaking on the phone.
Table 7: We show 4 demonstrations of our agent model’s outputs on a held-out Healthcare dataset that uses actors instead of actual patients. We demonstrate our model’s outputs across 3 different tasks: video captioning, visual question answering, and RASS score prediction (action recognition). Due to the nature of our actor-collected example videos, the model predicts that the patient is awake and calm (RASS score of 00) for most video clips, despite only 60%~60\% of the training data containing RASS score of 00.
Text instruction Start frame Predicted Action Ground Truth Action
the player is digging and placing dirt blocks to terraform the terrain around their house… [Uncaptioned image] [STARTACTION] [attack] [CAMERAX0] [CAMERAY-1] [ENDOFACTION] [STARTACTION] [attack] [ENDOFACTION]
the player is mining underground using a diamond pickaxe, gathering cobblestone, coal, iron ore… [Uncaptioned image] [STARTACTION] [attack] [CAMERAX-3] [CAMERAY0] [ENDOFACTION] [STARTACTION] [attack] [CAMERAX-3] [CAMERAY0] [ENDOFACTION]
the minecraft player is moving around a village … [Uncaptioned image] [STARTACTION] [forward] [sprint] [ENDOFACTION] [STARTACTION] [forward] [sprint] [ENDOFACTION]
the player is using a brewing stand … [Uncaptioned image] [STARTACTION] [sneak] [use] [ENDOFACTION] [STARTACTION] [sneak] [ENDOFACTION]
the player is … terraforming by digging … [Uncaptioned image] [STARTACTION] [attack] [ENDOFACTION] [STARTACTION] [attack] [ENDOFACTION]
Table 8: We show 5 demonstrations from a held-out Minecraft dataset. In addition to the high level instruction, we show the low-level predicted actions and ground truth actions. We truncate the instructions to show only the parts relevant to the current frames. The most common errors are slight differences in camera movements and occasionally performing unnecessary actions. Note that sometimes the ground truth values are not the only valid actions; for instance, the fourth example predicts that the player will click the bottle, which happens a few frames later in the ground truth trajectory.
Text instruction Start frame Predicted Action Ground Truth Action
the player is using a character with a sword to fight enemies and collect power cells … [Uncaptioned image] [STARTACTION] [lockon][meleeattack] [lrot214] [lmag4] [ENDOFACTION] [STARTACTION] [lockon][meleeattack] [lrot213] [lmag4] [ENDOFACTION]
the player is riding a hoverboard-like vehicle … avoiding or attacking enemy players … [Uncaptioned image] [STARTACTION] [lockon][meleeattack] [lrot204] [lmag4] [ENDOFACTION] [STARTACTION] [lockon][meleeattack] [lrot201] [lmag4] [ENDOFACTION]
the player starts by descending some stairs towards an open area where they engage in combat with an enemy player … [Uncaptioned image] [STARTACTION] [jump] [lockon][specialability1] [lrot199] [lmag4] [ENDOFACTION] [STARTACTION] [jump] [lockon][meleeattack] [lrot201] [lmag4] [ENDOFACTION]
the player … captures an objective point while fighting off multiple opponents … [Uncaptioned image] [STARTACTION] [lockon][meleeattack] [lrot63] [lmag4] [ENDOFACTION] [STARTACTION] [lockon][meleeattack] [lrot63] [lmag4] [ENDOFACTION]
a bleeding edge player is controlling a robot character with a sword … engaging in combat with enemy players … [Uncaptioned image] [STARTACTION] [evade] [lrot236] [lmag4] [ENDOFACTION] [STARTACTION] [evade] [lrot236] [lmag4] [ENDOFACTION]
Table 9: We show 5 unique demonstrations from a held-out Bleeding Edge dataset. In addition to the high level instruction, we show the low-level predicted actions and ground truth actions. We truncate the instructions to show only the parts relevant to the current frames. The most common errors are slight deviations from the precise value of the joysticks, which are naturally noisy. Some other errors include predicting the wrong type of attack, though this typically happens in situations where multiple attacks are still valid.
Concept demonstration

Open image ↗