Literature
Display
← All papers

Learning Shared Safety Constraints from Multi-task Demonstrations

Kim, Konwoo · Swamy, Gokul · Liu, Zuxin · Zhao, Ding · Choudhury, Sanjiban · Wu, Zhiwei Steven

· NeurIPS 2023 ↗

Robotics6 min text · CondensedOriginal paper ↗
Contents

At a glance

Inverse constraint learning (ICL) infers shared safety rules from safe demonstrations and known task rewards. It penalizes rewarding behavior the demonstrator avoided, while using demonstrations from several tasks to reduce overconservative solutions. Theory concerns expected reward and constraint cost under stated assumptions; continuous-control simulations demonstrate the approach.

1 Introduction

A robot may pursue many goals while obeying the same implicit rule, such as not breaking objects. Given each task’s reward, safe demonstrations reveal which tempting shortcuts are unacceptable. Single-task demonstrations leave much of the state space unexplored, making it easy to forbid every unseen behavior. Combining tasks provides broader evidence about what is safe without requiring an explicit hand-written constraint.

2 Related Work

The method extends the game interpretation of inverse reinforcement learning (IRL) [Swamy et al., 2021]: a constraint player distinguishes learner from expert while a constrained reinforcement learning (CRL) player maximizes reward. Unlike exact constraint-recovery methods, the target is expert-level safety and performance; the expert need not be reward-optimal. Known rewards and a suitable constraint class provide information beyond simply replaying demonstrated trajectories.

3 Formalizing Inverse Constraint Learning

The formulation uses a finite-horizon Markov decision process (MDP), bounded rewards and constraint costs, and policies represented through their expected cumulative returns. The theoretical policy and function classes are assumed convex and compact; practical neural implementations need not satisfy the idealized optimization assumptions.

3.1 Prior Work: Inverse RL as Game Solving

IRL can be written as a zero-sum game between a policy and a reward function that distinguishes learner from expert. Matching expectations over a sufficiently rich reward class produces expert-like performance without identifying a unique true reward.

Equation 1
minπ∈Π⁡maxf∈ℱr⁡J⁡(π,f)−J⁡(πE,f),\min_{\pi\in\Pi}\max_{f\in\mathcal{F}_{r}}J(\pi,f)-J(\pi_{E},f), (1)

3.2 Prior Work: Constrained Reinforcement Learning as Game Solving

CRL maximizes reward subject to a bound on expected accumulated constraint cost. Its Lagrangian introduces a multiplier that increases the penalty when the learner exceeds the cost limit. This is an expectation constraint, not a guarantee that every individual trajectory avoids violations.

 Input: Reward rr, constraint cc, learning rates η1:N\eta_{1:N}, tolerance δ\delta
 Output: Trained policy π\pi
 Initialize λ1=0\lambda_{1}=0
 for ii in 1​…​N1\dots N do
  πi←RL​(r=r−λi​c)\pi_{i}\leftarrow\texttt{RL}(r=r-\lambda_{i}c)
  λi←[λi+ηi​(J⁡(πi,c)−δ)]+\lambda_{i}\leftarrow[\lambda_{i}+\eta_{i}(J(\pi_{i},c)-\delta)]^{+}
 end for
 Return Unif(π1:N)\text{Unif}(\pi_{1:N}).
Algorithm 1 CRL (Constrained Reinforcement Learning)
Equations 2–3
minπ∈Π−J⁡(π,r)​ s.t. ​J​(π,c)≤δ,\min_{\pi\in\Pi}-J(\pi,r)\text{ s.t. }J(\pi,c)\leq\delta, (2)
minπ∈Π⁡maxλ>0−J⁡(π,r)+λ⁡(J⁡(π,c)−δ).\min_{\pi\in\Pi}\max_{\lambda>0}-J(\pi,r)+\lambda(J(\pi,c)-\delta). (3)

3.3 Single-Task Inverse Constraint Learning

Single-task ICL alternates constraint updates with a CRL solve whose cost limit equals the expert’s cost under the candidate constraint. The theorem identifies an iterate with at least the expert’s reward and true constraint cost bounded by the expert’s cost plus average regret times the horizon. This assumes the true constraint belongs to the candidate class and the required CRL problems are solved. No-regret updates make the excess-cost bound vanish asymptotically; exact constraint identification is unnecessary.

 Input: Reward rr, constraint class ℱc\mathcal{F}_{c}, trajectories from πE\pi_{E}
 Output: Learned constraint cc
 Initialize c1∈ℱcc_{1}\in\mathcal{F}_{c}
 for ii in 1​…​N1\dots N do
  πi,λi←CRL​(r,ci,δ=J⁡(πE,ci))\pi_{i},\lambda_{i}\leftarrow\texttt{CRL}(r,c_{i},\delta=J(\pi_{E},c_{i}))
  # use any no-regret algo. to pick c, e.g. FTRL:
  ci+1←arg⁡maxc∈ℱc⁡1T​∑ji(J⁡(πj,c)−J⁡(πE,c))−R⁡(c).c_{i+1}\leftarrow\argmax_{c\in\mathcal{F}_{c}}\frac{1}{T}\sum_{j}^{i}(J(\pi_{j},c)-J(\pi_{E},c))-R(c).
 end for
 Return best of c1:Nc_{1:N} on validation data.
Algorithm 2 ICL (Inverse Constraint Learning)
Equations 4–7
minπ∈Π⁡J⁡(πE,r)−J⁡(π,r)\displaystyle\min_{\pi\in\Pi}J(\pi_{E},r)-J(\pi,r) (4)
s.t.​maxc∈ℱc⁡J⁡(π,c)−J⁡(πE,c)≤0.\displaystyle\text{s.t.}\max_{c\in\mathcal{F}_{c}}J(\pi,c)-J(\pi_{E},c)\leq 0. (5)
minπ∈Π⁡maxλ>0⁡J⁡(πE,r)−J⁡(π,r)+λ⁡(maxc∈ℱc⁡J⁡(π,c)−J⁡(πE,c))\displaystyle\min_{\pi\in\Pi}\max_{\lambda>0}J(\pi_{E},r)-J(\pi,r)+\lambda(\max_{c\in\mathcal{F}_{c}}J(\pi,c)-J(\pi_{E},c)) (6)
=maxc∈ℱc⁡maxλ>0​minπ∈Π⁡J⁡(πE,r−λ​c)−J⁡(π,r−λ​c).\displaystyle=\max_{c\in\mathcal{F}_{c}}\max_{\lambda>0}\min_{\pi\in\Pi}J(\pi_{E},r-\lambda c)-J(\pi,r-\lambda c). (7)
Original figure
Figure 1: A visual depiction of the optimization problem we’re trying to solve in ICL. We attempt to pick a constraint that minimizes the value difference over the expert policy a safe policy could have. The star corresponds to the output of CRL.
Equations 8–11
ℓi​(c)=1T​(J⁡(πi,c)−J⁡(πE,c))∈[−1,1]\ell_{i}(c)=\frac{1}{T}(J(\pi_{i},c)-J(\pi_{E},c))\in[-1,1] (8)
c^=arg⁡maxc∈ℱc∑iTℓi(c).\hat{c}=\argmax_{c\in\mathcal{F}_{c}}\sum_{i}^{T}\ell_{i}(c). (9)
Reg​(T)=∑iTℓi​(c^)−∑iTℓi​(ci),\text{Reg}(T)=\sum_{i}^{T}\ell_{i}(\hat{c})-\sum_{i}^{T}\ell_{i}(c_{i}), (10)
J⁡(π,c∗)−J⁡(πE,c∗)≤ϵ¯​T​ and ​J​(π,r)≥J⁡(πE,r).J(\pi,c^{*})-J(\pi_{E},c^{*})\leq\bar{\epsilon}T\text{ and }J(\pi,r)\geq J(\pi_{E},r). (11)

3.4 Multi-task Inverse Constraint Learning

For several rewards sharing dynamics and safety rules, one constraint player aggregates data from all task policies and experts. Each task still has its own CRL problem and cost limit. The extra coverage helps distinguish genuinely unsafe behavior from behavior merely irrelevant to one demonstration task.

 Input: Rewards r1:Kr^{1:K}, constraint class ℱc\mathcal{F}_{c}, trajectories from πE1:K\pi_{E}^{1:K}
 Output: Learned constraint cc
 Set ℱ~c={c∈ℱc|∀k∈[K],J(πEk,c)≤0}\widetilde{\mathcal{F}}_{c}=\{c\in\mathcal{F}_{c}|\forall k\in[K],J(\pi_{E}^{k},c)\leq 0\}
 Initialize c1∈ℱ~cc_{1}\in\widetilde{\mathcal{F}}_{c}
 for ii in 1​…​N1\dots N do
  for kk in 1​…​K1\dots K do
   πik,λik←CRL​(rk,ci,δ=0)\pi_{i}^{k},\lambda_{i}^{k}\leftarrow\texttt{CRL}(r^{k},c_{i},\delta=0)
  end for
  # use any no-regret algo. to pick c, e.g. FTRL:
  ci+1←arg⁡maxc∈ℱ~c⁡1TK​∑ji∑kK(J⁡(πjk,c)−J⁡(πEk,c))−R⁡(c).c_{i+1}\leftarrow\argmax_{c\in\widetilde{\mathcal{F}}_{c}}\frac{1}{TK}\sum_{j}^{i}\sum_{k}^{K}(J(\pi_{j}^{k},c)-J(\pi_{E}^{k},c))-R(c).
 end for
 Return best of c1:Nc_{1:N} on validation data.
Algorithm 3 MT-ICL (Multi-task Inverse Constraint Learning)
Equation 12
maxc∈ℱcminπ1:K∈Πmaxλ1:K>0∑iKJ(πEi,ri−λic)−J(πi,ri−λic).\max_{c\in\mathcal{F}_{c}}\min_{\pi^{1:K}\in\Pi}\max_{\lambda^{1:K}>0}\sum_{i}^{K}J(\pi_{E}^{i},r^{i}-\lambda^{i}c)-J(\pi^{i},r^{i}-\lambda^{i}c). (12)

3.5 A (Strong) Geometric Condition for Identifiability

Exact identification requires unusually strong geometric coverage: sufficiently many distinct optimal expert occupancy measures must span the constraint boundary and satisfy the stated relative-interior conditions. The recovered constraint is unique only up to scale. The paper therefore treats exact recovery as a stronger target than the safety and performance guarantees needed in practice.

Original figure
Figure 2: If we have a sufficient diversity of expert policies, none of which are optimal along the reward vector, we can identify the hyperplane that separates the safe policies from the unsafe policies. The constraint (red, dashed) will be orthogonal to this hyperplane. For this example, because ρπ∈ℝ2\rho_{\pi}\in\mathbb{R}^{2}, we need two expert policies.
Equations 13
maxρπ∈ρΠ⁡⟨ρπ,r⟩​ s.t. ​⟨ρπ,c∗⟩≤0.\displaystyle\max_{\rho_{\pi}\in\rho_{\Pi}}\langle\rho_{\pi},r\rangle\text{ s.t. }\langle\rho_{\pi},c^{*}\rangle\leq 0.
Nul​([ρπE1−ρπE2ρπE|𝒮|​|𝒜|−1−ρπE|𝒮|​|𝒜|]).\text{Nul}\left(\begin{bmatrix}\rho_{\pi_{E}^{1}}-\rho_{\pi_{E}^{2}}\\ \vdots\\ \rho_{\pi_{E}^{|\mathcal{S}||\mathcal{A}|-1}}-\rho_{\pi_{E}^{|\mathcal{S}||\mathcal{A}|}}\\ \end{bmatrix}\right). (13)

3.6 A Statistical Condition for Generalization

If tasks are sampled independently from a common distribution, a finite constraint class and enough task samples allow uniform estimation of constraint-player losses. Additional feasibility and expert-safety assumptions yield approximate expert-level reward and safety in expectation over new tasks. The bounds include task-sampling error and regret; they do not certify arbitrary out-of-distribution tasks or individual trajectories.

Equations 14–15
V⁡(c)=𝔼τ∼P⁡(τ)​[J⁡(πτ,c)−J⁡(πEτ,c)],V(c)=\mathbb{E}_{\tau\sim P(\tau)}[J(\pi^{\tau},c)-J(\pi_{E}^{\tau},c)], (14)
K≥O⁡(log⁡(|ℱc|δ)​(2​T)2ϵ2)K\geq O\left(\log\left(\frac{|\mathcal{F}_{c}|}{\delta}\right)\frac{(2T)^{2}}{\epsilon^{2}}\right) (15)
𝔼τ∼P⁡(τ)​[J⁡(π⁡(rτ),c∗)−J⁡(πEτ,c∗)]≤ϵ¯​T+3​ϵ​T​ and ​𝔼τ∼P⁡(τ)​[J⁡(π⁡(rτ),rτ)−J⁡(πEτ,rτ)]≥−2​ϵ​T.\mathbb{E}_{\tau\sim P(\tau)}[J(\pi(r^{\tau}),c^{*})-J(\pi_{E}^{\tau},c^{*})]\leq\bar{\epsilon}T+3\epsilon T\text{ and }\mathbb{E}_{\tau\sim P(\tau)}[J(\pi(r^{\tau}),r^{\tau})-J(\pi_{E}^{\tau},r^{\tau})]\geq-2\epsilon T.

4 Practical Algorithm

Experiments implement the idealized games with neural networks and approximate policy optimization. The evaluation checks task reward, true constraint cost, and—where the function class permits interpretation—recovery of the underlying constraint.

4.1 Tasks

Ant locomotion tasks impose either a velocity cap or a linear position boundary. A modified D4RL AntMaze makes walls physically permeable but charges a cost for crossing them. Experts are trained with the true constraint. PPO with a proportional–integral–derivative (PID) Lagrangian update supplies the policy optimizer, and behavior cloning initializes each inner solve. Experiments use one NVIDIA 3090 GPU.

4.2 ICL Results

Single-task tests examine simple identifiable constraints; multi-task tests examine whether shared demonstrations recover maze geometry that one task leaves ambiguous.

4.3 Single-Task Continuous Control Results

With restricted state inputs and simple constraint families, learned velocity and position boundaries approach the ground truth while reward and safety approach expert performance. Adding Gaussian noise to expert actions still permits useful constraint and policy learning in these experiments. These results support robustness to the tested noisy experts, not arbitrary demonstration errors.

Original figure
Original figure
Original figure
Original figure
Original figure
Original figure
Figure 3: Over the course of training, the learned ICL constraint recovers the ground-truth constraints for the velocity and position tasks. The learned policy matches expert performance and constraint violation. Standard errors are computed across 3 seeds.
Refer to caption
Figure 4: As ICL training progresses, the learned position constraint (red line) converges to the ground-truth constraint (blue line) and the policy learns to escape unsafe regions (red region).
Original figure
Original figure
Original figure
Original figure
Original figure
Original figure
Figure 5: We re-conduct our position velocity constraint experiments using suboptimal experts to generate demonstrations. While, because of the ill-posedness of the problem, we do not exactly recover the ground truth constraint, we are able to use it to learn a policy that is higher performance than the expert while being just as safe. Standard errors are computed across 3 seeds.

4.4 Multi-Task Continuous Control Results

Ten maze goals with two possible starting locations provide complementary trajectories. Joint ICL approximately reconstructs maze walls and quickly matches expert reward and constraint cost across these tasks. A single-task classifier invents spurious walls, and aggregating separately trained classifiers also performs poorly. Evaluation uses the true maze cost, although the learner’s optimization receives demonstrations and learned costs rather than ground-truth wall labels.

Original figure
Original figure
Original figure
Figure 6: We see that over ICL iterations, we are able to recover the ground-truth walls of the ant-maze, enabling the learner to match expert performance and constraint violations. Results for the second two plots are averaged across all 10 tasks. Standard errors are computed across 3 seeds.
Original figure
(a)
Original figure
(b)
Original figure
(c)
Original figure
(d)
Original figure
(e)
Original figure
(f)
Original figure
(g)
Refer to caption
(h)
Figure 7: We consider the problem of trying to learn the walls of a custom maze (a) based on the AntMaze environment from D4RL (Fu et al., 2020). We consider both a single-task (b) and multi-task (c) setup. We see that the single-task data is insufficient to learn an accurate constraint (d). Averaging or taking the max over the constraints learned from the data for each of the ten goals (e)-(f) also doesn’t work. However, if we use the data from all 10 tasks to learn the constraint (g)-(h), we are able to approximately recover the ground-truth constraint with enough constraint learning iterations.

5 Discussion

Shared demonstrations can reduce unnecessary restrictions while preserving useful safety information. Remaining limits include costly repeated CRL solves, no real-robot evaluation, and unaddressed finite-trajectory-sample errors. The task-generalization theory does not remove these practical gaps.

6 Acknowledgements

The authors acknowledge research discussions, reviewer feedback on suboptimal experts, institutional research support, and an NVIDIA GPU award.

Appendix A Proofs

Proofs establish the single-task regret result and the task-sampling generalization bounds.

A.1 Proof of Theorem 11

The expert is feasible for every inner CRL problem, establishing the reward comparison. The constraint player’s regret bounds excess true cost on average across iterates; therefore at least one iterate satisfies the stated safety bound.

Equations 16–19
J⁡(π,c)−J⁡(πE,c)≤0.J(\pi,c)-J(\pi_{E},c)\leq 0. (16)
1N​∑iN[(J⁡(πi,c^)−J⁡(πE,c^))−(J⁡(πi,ci)−J⁡(πE,ci))]=TN​∑iNℓi​(c^)−ℓi​(ci)≤ϵ¯​T.\frac{1}{N}\sum_{i}^{N}[(J(\pi_{i},\hat{c})-J(\pi_{E},\hat{c}))-(J(\pi_{i},c_{i})-J(\pi_{E},c_{i}))]=\frac{T}{N}\sum_{i}^{N}\ell_{i}(\hat{c})-\ell_{i}(c_{i})\leq\bar{\epsilon}T. (17)
1N​∑iN[(J⁡(πi,c∗)−J⁡(πE,c∗))−(J⁡(πi,ci)−J⁡(πE,ci))]≤ϵ¯​T,\frac{1}{N}\sum_{i}^{N}[(J(\pi_{i},c^{*})-J(\pi_{E},c^{*}))-(J(\pi_{i},c_{i})-J(\pi_{E},c_{i}))]\leq\bar{\epsilon}T, (18)
1N​∑iN(J⁡(πE,c∗)−J⁡(πi,c∗))≤ϵ¯​T.\frac{1}{N}\sum_{i}^{N}(J(\pi_{E},c^{*})-J(\pi_{i},c^{*}))\leq\bar{\epsilon}T. (19)

A.2 Proof of Lemma 3.3

Hoeffding concentration bounds the estimation error for one constraint. A union bound over the finite constraint class gives the required number of task samples for simultaneous accuracy.

Equations 20–22
P⁡(|1K​∑i=0KVk​(c)−𝔼⁡[V⁡(c)]|≥ϵ)≤2​exp⁡(−2​K​ϵ2(2​T)2),P(|\frac{1}{K}\sum_{i=0}^{K}V_{k}(c)-\mathbb{E}[V(c)]|\geq\epsilon)\leq 2\exp\left(\frac{-2K\epsilon^{2}}{(2T)^{2}}\right), (20)
1−2​|ℱc|​exp⁡(−2​K​ϵ2(2​T)2),1-2|\mathcal{F}_{c}|\exp(\frac{-2K\epsilon^{2}}{(2T)^{2}}), (21)
K≥O⁡(log⁡(|ℱc|δ)​(2​T)2ϵ2)K\geq O\left(\log\left(\frac{|\mathcal{F}_{c}|}{\delta}\right)\frac{(2T)^{2}}{\epsilon^{2}}\right) (22)

A.3 Proof of Theorem 3.5

The proof separates reward feasibility from constraint-cost control, first for the population task distribution and then for finitely many tasks. Uniform loss-estimation error and possible expert infeasibility add a total error term proportional to three times the task-estimation tolerance and horizon. The result is probabilistic and concerns expected task performance.

Equations 23–41
u(τ,c)=𝟙{πEτ∈Π(c)}.u(\tau,c)=\mathbbm{1}\{\pi_{E}^{\tau}\in\Pi(c)\}. (23)
𝔼τ∼P⁡(τ)​[J⁡(π⁡(rτ),rτ)−J⁡(πEτ,rτ)]≥−2​ϵ​T,\mathbb{E}_{\tau\sim P(\tau)}[J(\pi(r^{\tau}),r^{\tau})-J(\pi_{E}^{\tau},r^{\tau})]\geq-2\epsilon T, (24)
πEτ∈Π⁡(c^)⇒J⁡(πτ,rτ)=maxπ∈Π⁡(c^)⁡J⁡(π,rτ)≥J⁡(πEτ,rτ).\pi_{E}^{\tau}\in\Pi(\hat{c})\Rightarrow J(\pi^{\tau},r^{\tau})=\max_{\pi\in\Pi(\hat{c})}J(\pi,r^{\tau})\geq J(\pi_{E}^{\tau},r^{\tau}). (25)
𝔼τ​[u⁡(τ,c^)]≥1−ϵ,\mathbb{E}_{\tau}[u(\tau,\hat{c})]\geq 1-\epsilon, (26)
𝔼τ​[J⁡(πτ,rτ)−J⁡(πEτ,rτ)]\displaystyle\mathbb{E}_{\tau}[J(\pi^{\tau},r^{\tau})-J(\pi_{E}^{\tau},r^{\tau})] (27)
≤𝔼τ​[(1−u⁡(τ,c^))​(J⁡(πτ,rτ)−J⁡(πEτ,rτ))]\displaystyle\leq\mathbb{E}_{\tau}[(1-u(\tau,\hat{c}))(J(\pi^{\tau},r^{\tau})-J(\pi_{E}^{\tau},r^{\tau}))] (28)
≤𝔼τ​[(1−u⁡(τ,c^))]​(supτJ⁡(πτ,rτ)−J⁡(πEτ,rτ))\displaystyle\leq\mathbb{E}_{\tau}[(1-u(\tau,\hat{c}))](\sup_{\tau}J(\pi^{\tau},r^{\tau})-J(\pi_{E}^{\tau},r^{\tau})) (29)
≤𝔼τ​[(1−u⁡(τ,c^))]​2​T\displaystyle\leq\mathbb{E}_{\tau}[(1-u(\tau,\hat{c}))]2T (30)
≤−2​ϵ​T\displaystyle\leq-2\epsilon T (31)
ℱ~c={c∈ℱc|∀k∈[K],πEK∈Π(c^)}.\widetilde{\mathcal{F}}_{c}=\{c\in\mathcal{F}_{c}|\forall k\in[K],\,\pi_{E}^{K}\in\Pi(\hat{c})\}. (32)
K≥O⁡(log⁡|ℱc|δ​1ϵ2),K\geq O\left(\log\frac{|\mathcal{F}_{c}|}{\delta}\frac{1}{\epsilon^{2}}\right), (33)
P(|1K∑kK𝟙{πEk∈Π(c)}−𝔼τ[u(τ,c)]|≥ϵ)≤2exp(−2Kϵ2).P(|\frac{1}{K}\sum_{k}^{K}\mathbbm{1}\{\pi_{E}^{k}\in\Pi(c)\}-\mathbb{E}_{\tau}[u(\tau,c)]|\geq\epsilon)\leq 2\exp{(-2K\epsilon^{2})}. (34)
1−2​|ℱc|​exp⁡(−2​K​ϵ2).1-2|\mathcal{F}_{c}|\exp{(-2K\epsilon^{2})}. (35)
K≥O⁡(log⁡|ℱc|δ​1ϵ2).K\geq O\left(\log\frac{|\mathcal{F}_{c}|}{\delta}\frac{1}{\epsilon^{2}}\right). (36)
𝔼τ​[J⁡(πτ,c∗)−J⁡(πEτ,c∗)]≤ϵ¯​T.\mathbb{E}_{\tau}[J(\pi^{\tau},c^{*})-J(\pi_{E}^{\tau},c^{*})]\leq\bar{\epsilon}T. (37)
ℓi​(c)=1T​𝔼τ​[J⁡(πiτ,c)−J⁡(πEτ,c)]∈[−1,1],\ell_{i}(c)=\frac{1}{T}\mathbb{E}_{\tau}[J(\pi^{\tau}_{i},c)-J(\pi^{\tau}_{E},c)]\in[-1,1], (38)
ϵ¯​T\displaystyle\bar{\epsilon}T =1N​∑iN𝔼τ​[J⁡(πiτ,c^)−J⁡(πEτ,c^)]−𝔼τ​[J⁡(πiτ,ci)−J⁡(πEτ,ci)]\displaystyle=\frac{1}{N}\sum_{i}^{N}\mathbb{E}_{\tau}[J(\pi^{\tau}_{i},\hat{c})-J(\pi^{\tau}_{E},\hat{c})]-\mathbb{E}_{\tau}[J(\pi^{\tau}_{i},c_{i})-J(\pi^{\tau}_{E},c_{i})] (39)
≥1N​∑iN𝔼τ​[J⁡(πiτ,c∗)−J⁡(πEτ,c∗)]−𝔼τ​[J⁡(πiτ,ci)−J⁡(πEτ,ci)]\displaystyle\geq\frac{1}{N}\sum_{i}^{N}\mathbb{E}_{\tau}[J(\pi^{\tau}_{i},c^{*})-J(\pi^{\tau}_{E},c^{*})]-\mathbb{E}_{\tau}[J(\pi^{\tau}_{i},c_{i})-J(\pi^{\tau}_{E},c_{i})] (40)
ϵ¯​T≥1N​∑iN𝔼τ​[J⁡(πiτ,c∗)−J⁡(πEτ,c∗)]\bar{\epsilon}T\geq\frac{1}{N}\sum_{i}^{N}\mathbb{E}_{\tau}[J(\pi^{\tau}_{i},c^{*})-J(\pi^{\tau}_{E},c^{*})] (41)

Appendix B Experimental Details

The appendix specifies expert generation, stabilization choices, maze planning, and the complete hyperparameter table.

B.1 CRL

PPO uses a PID update of the Lagrange multiplier to reduce oscillation and cost overshoot. Appending the current constraint value to the policy observation is important for training; during expert CRL this value comes from the true constraint.

B.2 Expert Demonstrations

Each experiment uses 20 expert rollouts. Noisy demonstrations add zero-mean Gaussian action noise with standard deviation 0.7 for velocity tasks and 0.5 for position tasks.

B.3 Single-Task ICL

Each inner policy starts from behavior cloning, with the extra constraint observation zeroed during cloning to prevent information leakage. A temporary cost-limit buffer is annealed to zero. Neural constraint outputs lie in [0,1] and are fitted by squared-error regression using learner/expert targets. The implementation weights data equally rather than by theoretical multipliers and applies a log transformation for stable training.

B.4 Multi-Task ICL

Maze navigation combines value iteration on a 10×10 grid with low-level directional CRL controllers. Ten goals each have two starting locations. Visualized walls average predictions over 100,000 sampled points per cell and threshold at 0.5; this discretized visualization is distinct from the continuous learned constraint.

B.5 Hyperparameters

The retained table gives PPO settings, PID learning rates, constraint fitting batches and steps, demonstration counts, cost buffers, annealing, and inner/outer training epochs.

Hyperparameter Value
PPO Learning Rate 0.0003
PPO Value Loss Weight 0.25
PPO Epsilon Clip 0.2
PPO GAE Lambda 0.97
PPO Discount Factor 0.99
PPO Batch Size 512
PPO Hidden Sizes [128, 128]
P-update Learning Rate 0.05
I-update Learning Rate 0.0005
D-update Learning Rate 0.1
Constraint Batch Size 4096
Constraint Learning Rate 0.05
Constraint Update Steps 250
Steps per Epoch 20000
CRL Velocity Epochs 50
CRL Position Epochs 100
ICL Expert Demonstrations 20
ICL Velocity Cost Limit 20
ICL Position Cost Limit 100
ICL Anneal Rate 10
ICL Velocity Outer Epochs 20
ICL Position Outer Epochs 10
ICL Velocity Epochs 10
ICL Position Epochs 50
Table 1: Experiment hyperparameters.

Appendix C Comparison to Chou et al. [2020]

The appendix compares ICL with an adaptation of the constraint-learning method of Chou et al. [2020].

C.1 Experimental Setup

Direct trajectory-space sampling is impractical for these long, high-dimensional tasks. The baseline instead searches with a behavior-cloned policy and maximum-entropy reinforcement learning, then fits a constraint to higher-reward sampled trajectories and expert data. Mixed-integer feasibility is replaced by a regression relaxation. These substantial implementation adaptations limit conclusions about the original algorithm itself.

C.2 Baseline Comparison Results

Across three seeds, the adapted baseline produces less accurate constraints and greater violations than the final ICL iterate in the reported position and velocity tests. The retained tables give constraint error, reward gap, and safety-cost gap with their preferred directions; improvement is empirical and not uniformly monotonic across iterations.

Algo. |c∗−c||c^{*}-c| (↓) J⁡(πE,r)−J⁡(π,r)J(\pi_{E},r)-J(\pi,r) (↓) J⁡(πE,c∗)−J⁡(π,c∗)J(\pi_{E},c^{*})-J(\pi,c^{*}) (↑)
ICL (iter 1) 0.052 569.277 270.111
ICL (iter 10) 0.000 -72.149 133.59
Chou et al. 0.311 384.772 -1958.372
Table 2: Baseline Comparison (Position Constraint)
Algo. |c∗−c||c^{*}-c| (↓) J⁡(πE,r)−J⁡(π,r)J(\pi_{E},r)-J(\pi,r) (↓) J⁡(πE,c∗)−J⁡(π,c∗)J(\pi_{E},c^{*})-J(\pi,c^{*}) (↑)
ICL (iter 1) 0.249 12.267 -244.25
ICL (iter 10) 0.006 -46.771 -21.602
Chou et al. 0.246 -28.315 -134.58
Table 3: Baseline Comparison (Velocity Constraint)
Concept demonstration

Open image ↗