Literature
Display
← All papers

Behavioral Cloning from Observation

Torabi, Faraz · Warnell, Garrett · Stone, Peter

· IJCAI 2018 ↗

Robotics4 min text · CondensedOriginal paper ↗
Contents

At a glance

Behavioral cloning from observation (BCO) learns a policy from expert state sequences without the expert’s actions. Before seeing demonstrations, the agent explores to learn an inverse dynamics model; afterward it infers missing actions and clones them. Optional further interaction refines the model and policy. Experiments trade off demonstration data, task performance, and interaction before versus after demonstrations.

1 Introduction

Moving environment interaction before a demonstration becomes available can reduce the delay before imitating a new task. BCO learns how the agent’s own actions produce state changes, then reuses that knowledge to interpret demonstrations. “No post-demonstration interaction” does not mean no interaction overall: the initial dynamics-learning phase can be substantial.

Original diagram from PDF page 1, including all mathematical labels.
Figure 1: Behavioral Cloning from Observation (BCO(α\alpha)) framework proposed in this paper. The agent is initialized with a (random) policy which interacts with the environment and collects data to to learn its own agent-specific inverse dynamics model. Then, given state-only demonstration information, the agent uses this learned model to infer the expert’s missing action information. Once these actions have been inferred, the agent performs imitation learning. The updated policy is then used to collect data and this process repeats.

2 Related Work

Ordinary behavioral cloning requires action labels, while inverse reinforcement learning generally needs further environment interaction to optimize an inferred objective. BCO instead predicts actions from consecutive states using a learned inverse model. Its intended reusable component is the agent’s dynamics, rather than a reward tied to one task.

Original diagram from PDF page 3, including all mathematical labels.
Figure 2: Learning timelines for BCO(00), BCO(α\alpha), and the IRL methods we compare against in this paper. The horizontal axis represents time, gray rectangles mark when each technique requires environment interactions. For BCO, the white and gray circles denote the inverse model and policy learning steps, respectively. For BCO, α​|ℐp​r​e|\alpha|\mathcal{I}^{pre}| is the number of post-demonstration environment interactions performed before each model- and policy-improvement step and for IRL methods, |ℐI​R​L||\mathcal{I}^{IRL}| represents the total number of interactions.

3 Problem Formulation

The setting assumes a Markov decision process and comparable demonstrated and learner state transitions. State features are partitioned into agent-specific and task-specific components; inverse dynamics uses the former, while the imitation policy can use the full state. The experiments use supplied state vectors, not raw tutorial videos or arbitrary human-to-robot embodiment transfer.

4 Behavioral Cloning from Observation

The retained algorithm first gathers exploratory transitions and fits inverse dynamics, then labels the observed expert transitions with inferred actions and trains a policy. Repeating these steps with data collected by the current imitation policy produces an optional refinement procedure.

4.1 Inverse Dynamics Model Learning

Random exploration supplies state-transition/action triples. Maximum-likelihood training maps consecutive agent states to an action distribution: categorical for discrete actions and Gaussian for continuous actions. The continuous model predicts action means with a neural network and learns state-independent standard deviations. Accuracy depends on whether exploration covers the transitions needed to interpret demonstrations.

Equation 1
θ∗=arg​maxθ​Πi=0|ℐp​r​e|​pθ​(ai|sia,si+1a),\theta^{*}=\text{arg}\;\text{max}_{\theta}\;\Pi_{i=0}^{|\mathcal{I}^{pre}|}\;p_{\theta}(a_{i}\;|s_{i}^{a},s_{i+1}^{a}), (1)

4.2 Behavioral Cloning

The inverse model assigns the most likely action to each demonstrated state transition. Behavioral cloning then fits those inferred actions from the full demonstrated states. The policy therefore inherits errors from action inference as well as the usual distribution shift of supervised imitation.

Equation 2
ϕ∗=arg​maxϕ​Πi=0N​πϕ​(a~i|si).\phi^{*}=\text{arg}\;\text{max}_{\phi}\;\Pi_{i=0}^{N}\pi_{\phi}(\tilde{a}_{i}\;|\;s_{i})\;. (2)
Algorithm 1 BCO(α\alpha)
1: Initialize the model ℳθ\mathcal{M}_{\theta} as random approximator
2: Set πϕ\pi_{\phi} to be a random policy
3: Set I=|ℐp​r​e|I=|\mathcal{I}^{pre}|
4: while policy improvement do
5:   for time-step t=1 to II do
6:    Generate samples (sta,st+1a)(s^{a}_{t},s^{a}_{t+1}) and ata_{t} using πϕ\pi_{\phi}
7:    Append samples 𝒯πϕa←(sta,st+1a)\mathcal{T}^{a}_{\pi_{\phi}}\leftarrow(s^{a}_{t},s^{a}_{t+1}), 𝒜πϕ←at\mathcal{A}_{\pi_{\phi}}\leftarrow a_{t}
8:   end for
9:   Improve ℳθ\mathcal{M}_{\theta} by modelLearning(𝒯πϕa\mathcal{T}^{a}_{\pi_{\phi}}, 𝒜πϕ\mathcal{A}_{\pi_{\phi}})
10:   Generate set of agent-specific state transitions 𝒯d​e​m​oa\mathcal{T}^{a}_{demo} from the demonstrated state trajectories Dd​e​m​oD_{demo}
11:   Use ℳθ\mathcal{M}_{\theta} with 𝒯d​e​m​oa\mathcal{T}^{a}_{demo} to approximate 𝒜~d​e​m​o\tilde{\mathcal{A}}_{demo}
12:   Improve πϕ\pi_{\phi} by behavioralCloning(Sd​e​m​o,𝒜~d​e​m​oS_{demo},\tilde{\mathcal{A}}_{demo})
13:   Set I=α​|ℐp​r​e|I=\alpha|\mathcal{I}^{pre}|
14: end while
Refer to caption
(a) CartPole
Refer to caption
(b) MountainCar
Refer to caption
(c) Reacher
Refer to caption
(d) Ant
Figure 3: Representative screenshots of the domains considered in this paper.

4.3 Model Improvement

BCO(α) executes its current policy, updates inverse dynamics from the new transitions, and refits the imitation policy. Each round uses α times the initial exploration budget; after T rounds, the extra interaction budget is Tα times that initial budget. BCO(0) skips this refinement. Larger α can improve performance while increasing post-demonstration cost.

5 Implementation and Experimental Results

Tests use CartPole, MountainCar, Reacher, and Ant simulations, with experts trained by trust region policy optimization (TRPO). Comparisons include behavioral cloning, feature expectation matching (FEM), and generative adversarial imitation learning (GAIL). BCO alone lacks expert action labels, but has its own pre-demonstration exploration data; methods therefore have different information and interaction budgets.

Refer to caption
Figure 4: Performance of each technique with respect to the number of post-demonstration interactions. For each domain, ten demonstrated trajectories were considered. BCO(0) is depicted as a horizontal line since all environment interactions happen before the demonstration is provided. Performance values are scaled such that performance of a random policy is zero and the performance of the expert is one. Note that GAIL and FEM have access to demonstration action information whereas BCO does not. *The BCO line is not visible for the CartPole domain because BCO has the same performance as the expert. **FEM is not shown for the Reacher domain because its performance is much worse than the other techniques.

.

Original figure
Figure 5: Performance of imitation agents with respect to the number of available demonstration trajectories. Rectangular bars and error bars represent the mean return and the standard error, respectively, as measured over 5000 trajectories. Returns have been scaled such that the performance of a random policy and the demonstrating agent are zero and one, respectively. *Note that FEM is not shown for the Reacher domain because its performance is much worse than the others. **Note that BC, GAIL, and FEM all have access to demonstration action information whereas BCO(0) does not.

5.1 Training Details and Results

Initial inverse-model data comprise 1,000 interactions for CartPole, 2,000 for MountainCar, 5,000 for Reacher, and 500,000 for Ant. Models range from linear functions to two-layer networks. Reacher separates arm state from target position to reuse dynamics across targets. BCO and behavioral cloning reserve 30% of their data for validation and stop when validation error rises; other methods use all available training data.

5.2 Discussion

Results average twenty runs. In the plotted comparison, GAIL and FEM require at least forty times as many interactions to reach the BCO(0) performance level, while BCO’s exploration happens before demonstrations. This supports reduced post-demonstration delay rather than interaction-free learning. Final normalized returns average 5,000 evaluation episodes, with random and expert performance scaled to zero and one. BCO is weaker on MountainCar, competitive on Reacher and Ant, and improves with additional model-refinement interaction. Coverage, model error, and dependence on state observations limit generalization beyond these simulations.

Refer to caption
Figure 6: The performance of BC and several BCO(α\alpha) techniques (varying α\alpha) with respect to the number of demonstrated trajectories provided. Rectangular bars and error bars represent the mean return and the standard error, respectively, as measured over 5000 trajectories. By increasing α\alpha, more post-demonstration environment interactions are allowed to occur, which increases the performance of the imitation policy. Note that BC has access to demonstration action information whereas BCO does not. Also note that the number of trajectories required for learning a fairly good policy is very small. Each demonstrated trajectory has 5, 50, and 50 transitions for each domain from left to right, respectively. Note that we did not demonstrate the results for CartPole because the results were equally perfect regardless of the value of α\alpha.
MountainCar (pre-demo =𝟐​E​𝟑=2E3) Reacher (pre-demo =𝟓​E​𝟑=5E3) Ant (pre-demo =𝟓​E​𝟓=5E5)
𝐝\bf{d} \\bf{\backslash} 𝜶\bf{\alpha} 𝟐​𝑬−𝟑2E-3 𝟒​𝑬−𝟑4E-3 𝟏​𝑬−𝟐1E-2 𝐝\bf{d} \\bf{\backslash} 𝜶\bf{\alpha} 𝟐​𝑬−𝟑2E-3 𝟒​𝑬−𝟑4E-3 𝟏​𝑬−𝟐1E-2 𝐝\bf{d} \\bf{\backslash} 𝜶\bf{\alpha} 𝟐​𝑬−𝟑2E-3 𝟒​𝑬−𝟑4E-3 𝟏​𝑬−𝟐1E-2
𝟏1 68256825 2347523475 2895028950 𝟏1 210052210052 358736358736 912368912368 𝟓5 602500602500 12700001270000 33625003362500
𝟒4 83878387 1200012000 3120031200 𝟓5 270500270500 486578486578 18375001837500 𝟏𝟎10 940000940000 20750002075000 50000005000000
𝟕7 63006300 2310023100 122200122200 𝟏𝟎10 221421221421 569736569736 10559211055921 𝟏𝟓15 13875001387500 28550002855000 73250007325000
𝟏𝟎10 4546245462 6145061450 8860088600 𝟏𝟓15 509289509289 852210852210 18592101859210 𝟐𝟎20 19250001925000 40550004055000 96875009687500
Table 1: This is an extension to Figure 6 which provides the number of post-demonstration interactions for each case. First vertical column for each domain (dd) shows the number of demonstrated trajectories. As an example, for the MountainCar domain with 11 demonstrated trajectory and α=2​E−3\alpha=2E-3, the average number of post-demonstration interactions is 6825. We can also see at the top of the table that the number of pre-demonstration interactions is 2​E​32E3 so the overall number of interactions would become 88258825. It can be seen that almost always by increasing α\alpha or the number of demonstrated trajectories, the number of post-demonstration interactions increases. Also the overall number of interactions (combined pre- and post-demonstration interactions) in all the cases is far less than the overall number of interactions required by the other methods (FEM and GAIL).

6 Conclusions

A reusable inverse dynamics model can turn state-only demonstrations into supervised policy targets and enable imitation without additional post-demonstration interaction. Optional refinement improves the performance–interaction tradeoff, but prior exploration and compatible state representations remain necessary.

Acknowledgments

The authors disclose Peter Stone’s service on Cogitai’s board and the university’s review of that arrangement.

Concept demonstration

Open image ↗