Inverse constraint learning (ICL) infers shared safety rules from safe demonstrations and known task rewards. It penalizes rewarding behavior the demonstrator avoided, while using demonstrations from several tasks to reduce overconservative solutions. Theory concerns expected reward and constraint cost under stated assumptions; continuous-control simulations demonstrate the approach.
1 Introduction
A robot may pursue many goals while obeying the same implicit rule, such as not breaking objects. Given each task’s reward, safe demonstrations reveal which tempting shortcuts are unacceptable. Single-task demonstrations leave much of the state space unexplored, making it easy to forbid every unseen behavior. Combining tasks provides broader evidence about what is safe without requiring an explicit hand-written constraint.
2 Related Work
The method extends the game interpretation of inverse reinforcement learning (IRL) [Swamy et al., 2021]: a constraint player distinguishes learner from expert while a constrained reinforcement learning (CRL) player maximizes reward. Unlike exact constraint-recovery methods, the target is expert-level safety and performance; the expert need not be reward-optimal. Known rewards and a suitable constraint class provide information beyond simply replaying demonstrated trajectories.
3 Formalizing Inverse Constraint Learning
The formulation uses a finite-horizon Markov decision process (MDP), bounded rewards and constraint costs, and policies represented through their expected cumulative returns. The theoretical policy and function classes are assumed convex and compact; practical neural implementations need not satisfy the idealized optimization assumptions.
IRL can be written as a zero-sum game between a policy and a reward function that distinguishes learner from expert. Matching expectations over a sufficiently rich reward class produces expert-like performance without identifying a unique true reward.
Equation 1
(1)
3.2 Prior Work: Constrained Reinforcement Learning as Game Solving
CRL maximizes reward subject to a bound on expected accumulated constraint cost. Its Lagrangian introduces a multiplier that increases the penalty when the learner exceeds the cost limit. This is an expectation constraint, not a guarantee that every individual trajectory avoids violations.
Single-task ICL alternates constraint updates with a CRL solve whose cost limit equals the expert’s cost under the candidate constraint. The theorem identifies an iterate with at least the expert’s reward and true constraint cost bounded by the expert’s cost plus average regret times the horizon. This assumes the true constraint belongs to the candidate class and the required CRL problems are solved. No-regret updates make the excess-cost bound vanish asymptotically; exact constraint identification is unnecessary.
Input: Reward , constraint class , trajectories from
Figure 1: A visual depiction of the optimization problem we’re trying to solve in ICL. We attempt to pick a constraint that minimizes the value difference over the expert policy a safe policy could have. The star corresponds to the output of CRL.Equations 8–11
(8)
(9)
(10)
(11)
3.4 Multi-task Inverse Constraint Learning
For several rewards sharing dynamics and safety rules, one constraint player aggregates data from all task policies and experts. Each task still has its own CRL problem and cost limit. The extra coverage helps distinguish genuinely unsafe behavior from behavior merely irrelevant to one demonstration task.
Input: Rewards , constraint class , trajectories from
3.5 A (Strong) Geometric Condition for Identifiability
Exact identification requires unusually strong geometric coverage: sufficiently many distinct optimal expert occupancy measures must span the constraint boundary and satisfy the stated relative-interior conditions. The recovered constraint is unique only up to scale. The paper therefore treats exact recovery as a stronger target than the safety and performance guarantees needed in practice.
Figure 2: If we have a sufficient diversity of expert policies, none of which are optimal along the reward vector, we can identify the hyperplane that separates the safe policies from the unsafe policies. The constraint (red, dashed) will be orthogonal to this hyperplane. For this example, because , we need two expert policies.Equations 13
(13)
3.6 A Statistical Condition for Generalization
If tasks are sampled independently from a common distribution, a finite constraint class and enough task samples allow uniform estimation of constraint-player losses. Additional feasibility and expert-safety assumptions yield approximate expert-level reward and safety in expectation over new tasks. The bounds include task-sampling error and regret; they do not certify arbitrary out-of-distribution tasks or individual trajectories.
Equations 14–15
(14)
(15)
4 Practical Algorithm
Experiments implement the idealized games with neural networks and approximate policy optimization. The evaluation checks task reward, true constraint cost, and—where the function class permits interpretation—recovery of the underlying constraint.
4.1 Tasks
Ant locomotion tasks impose either a velocity cap or a linear position boundary. A modified D4RL AntMaze makes walls physically permeable but charges a cost for crossing them. Experts are trained with the true constraint. PPO with a proportional–integral–derivative (PID) Lagrangian update supplies the policy optimizer, and behavior cloning initializes each inner solve. Experiments use one NVIDIA 3090 GPU.
Single-task tests examine simple identifiable constraints; multi-task tests examine whether shared demonstrations recover maze geometry that one task leaves ambiguous.
4.3 Single-Task Continuous Control Results
With restricted state inputs and simple constraint families, learned velocity and position boundaries approach the ground truth while reward and safety approach expert performance. Adding Gaussian noise to expert actions still permits useful constraint and policy learning in these experiments. These results support robustness to the tested noisy experts, not arbitrary demonstration errors.
Figure 3: Over the course of training, the learned ICL constraint recovers the ground-truth constraints for the velocity and position tasks. The learned policy matches expert performance and constraint violation. Standard errors are computed across 3 seeds.Figure 4: As ICL training progresses, the learned position constraint (red line) converges to the ground-truth constraint (blue line) and the policy learns to
escape unsafe regions (red region).
Figure 5: We re-conduct our position velocity constraint experiments using suboptimal experts to generate demonstrations. While, because of the ill-posedness of the problem, we do not exactly recover the ground truth constraint, we are able to use it to learn a policy that is higher performance than the expert while being just as safe. Standard errors are computed across 3 seeds.
4.4 Multi-Task Continuous Control Results
Ten maze goals with two possible starting locations provide complementary trajectories. Joint ICL approximately reconstructs maze walls and quickly matches expert reward and constraint cost across these tasks. A single-task classifier invents spurious walls, and aggregating separately trained classifiers also performs poorly. Evaluation uses the true maze cost, although the learner’s optimization receives demonstrations and learned costs rather than ground-truth wall labels.
Figure 6: We see that over ICL iterations, we are able to recover the ground-truth walls of the ant-maze, enabling the learner to match expert performance and constraint violations. Results for the second two plots are averaged across all 10 tasks. Standard errors are computed across 3 seeds.
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
Figure 7: We consider the problem of trying to learn the walls of a custom maze (a) based on the AntMaze environment from D4RL (Fu et al., 2020). We consider both a single-task (b) and multi-task (c) setup. We see that the single-task data is insufficient to learn an accurate constraint (d). Averaging or taking the max over the constraints learned from the data for each of the ten goals (e)-(f) also doesn’t work. However, if we use the data from all 10 tasks to learn the constraint (g)-(h), we are able to approximately recover the ground-truth constraint with enough constraint learning iterations.
5 Discussion
Shared demonstrations can reduce unnecessary restrictions while preserving useful safety information. Remaining limits include costly repeated CRL solves, no real-robot evaluation, and unaddressed finite-trajectory-sample errors. The task-generalization theory does not remove these practical gaps.
6 Acknowledgements
The authors acknowledge research discussions, reviewer feedback on suboptimal experts, institutional research support, and an NVIDIAGPU award.
Appendix A Proofs
Proofs establish the single-task regret result and the task-sampling generalization bounds.
A.1 Proof of Theorem 11
The expert is feasible for every inner CRL problem, establishing the reward comparison. The constraint player’s regret bounds excess true cost on average across iterates; therefore at least one iterate satisfies the stated safety bound.
Equations 16–19
(16)
(17)
(18)
(19)
A.2 Proof of Lemma 3.3
Hoeffding concentration bounds the estimation error for one constraint. A union bound over the finite constraint class gives the required number of task samples for simultaneous accuracy.
Equations 20–22
(20)
(21)
(22)
A.3 Proof of Theorem 3.5
The proof separates reward feasibility from constraint-cost control, first for the population task distribution and then for finitely many tasks. Uniform loss-estimation error and possible expert infeasibility add a total error term proportional to three times the task-estimation tolerance and horizon. The result is probabilistic and concerns expected task performance.
Equations 23–41
(23)
(24)
(25)
(26)
(27)
(28)
(29)
(30)
(31)
(32)
(33)
(34)
(35)
(36)
(37)
(38)
(39)
(40)
(41)
Appendix B Experimental Details
The appendix specifies expert generation, stabilization choices, maze planning, and the complete hyperparameter table.
PPO uses a PID update of the Lagrange multiplier to reduce oscillation and cost overshoot. Appending the current constraint value to the policy observation is important for training; during expert CRL this value comes from the true constraint.
B.2 Expert Demonstrations
Each experiment uses 20 expert rollouts. Noisy demonstrations add zero-mean Gaussian action noise with standard deviation 0.7 for velocity tasks and 0.5 for position tasks.
Each inner policy starts from behavior cloning, with the extra constraint observation zeroed during cloning to prevent information leakage. A temporary cost-limit buffer is annealed to zero. Neural constraint outputs lie in [0,1] and are fitted by squared-error regression using learner/expert targets. The implementation weights data equally rather than by theoretical multipliers and applies a log transformation for stable training.
Maze navigation combines value iteration on a 10×10 grid with low-level directional CRL controllers. Ten goals each have two starting locations. Visualized walls average predictions over 100,000 sampled points per cell and threshold at 0.5; this discretized visualization is distinct from the continuous learned constraint.
B.5 Hyperparameters
The retained table gives PPO settings, PID learning rates, constraint fitting batches and steps, demonstration counts, cost buffers, annealing, and inner/outer training epochs.
The appendix compares ICL with an adaptation of the constraint-learning method of Chou et al. [2020].
C.1 Experimental Setup
Direct trajectory-space sampling is impractical for these long, high-dimensional tasks. The baseline instead searches with a behavior-cloned policy and maximum-entropy reinforcement learning, then fits a constraint to higher-reward sampled trajectories and expert data. Mixed-integer feasibility is replaced by a regression relaxation. These substantial implementation adaptations limit conclusions about the original algorithm itself.
C.2 Baseline Comparison Results
Across three seeds, the adapted baseline produces less accurate constraints and greater violations than the final ICL iterate in the reported position and velocity tests. The retained tables give constraint error, reward gap, and safety-cost gap with their preferred directions; improvement is empirical and not uniformly monotonic across iterations.
Abbeel and Ng [2004]
Pieter Abbeel and Andrew Y Ng.
Apprenticeship learning via inverse reinforcement learning.
In Proceedings of the twenty-first international conference on
Machine learning, page 1, 2004.
Achiam et al. [2017]
Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel.
Constrained policy optimization.
In International conference on machine learning, pages 22–31.
PMLR, 2017.
Amin et al. [2017]
Kareem Amin, Nan Jiang, and Satinder Singh.
Repeated inverse reinforcement learning.
Advances in neural information processing systems, 30, 2017.
Chou et al. [2020]
Glen Chou, Dmitry Berenson, and Necmiye Ozay.
Learning constraints from demonstrations.
In Algorithmic Foundations of Robotics XIII: Proceedings of the
13th Workshop on the Algorithmic Foundations of Robotics 13, pages 228–245.
Springer, 2020.
Fu et al. [2020]
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine.
D4rl: Datasets for deep data-driven reinforcement learning.
arXiv preprint arXiv:2004.07219, 2020.
Garcıa and Fernández [2015]
Javier Garcıa and Fernando Fernández.
A comprehensive survey on safe reinforcement learning.
Journal of Machine Learning Research, 16(1):1437–1480, 2015.
Gleave and Habryka [2018]
Adam Gleave and Oliver Habryka.
Multi-task maximum entropy inverse reinforcement learning.
arXiv preprint arXiv:1805.08882, 2018.
Gu et al. [2022]
Shangding Gu, Long Yang, Yali Du, Guang Chen, Florian Walter, Jun Wang, Yaodong
Yang, and Alois Knoll.
A review of safe reinforcement learning: Methods, theory and
applications.
arXiv preprint arXiv:2205.10330, 2022.
Haarnoja et al. [2018]
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha,
Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al.
Soft actor-critic algorithms and applications.
arXiv preprint arXiv:1812.05905, 2018.
Hadfield-Menell et al. [2017]
Dylan Hadfield-Menell, Smitha Milli, Pieter Abbeel, Stuart J Russell, and Anca
Dragan.
Inverse reward design.
Advances in neural information processing systems, 30, 2017.
Kim and Oh [2022]
Dohyeong Kim and Songhwai Oh.
Efficient off-policy safe reinforcement learning using trust region
conditional value at risk.
IEEE Robotics and Automation Letters, 7(3):7644–7651, 2022.
Kitani et al. [2012]
Kris M Kitani, Brian D Ziebart, James Andrew Bagnell, and Martial Hebert.
Activity forecasting.
In European conference on computer vision, pages 201–214.
Springer, 2012.
Kolter et al. [2008]
J Zico Kolter, Mike P Rodgers, and Andrew Y Ng.
A control architecture for quadruped locomotion over rough terrain.
In 2008 IEEE International Conference on Robotics and
Automation, pages 811–818. IEEE, 2008.
Lindner et al. [2023]
David Lindner, Xin Chen, Sebastian Tschiatschek, Katja Hofmann, and Andreas
Krause.
Learning safety constraints from demonstrations with unknown rewards.
arXiv preprint arXiv:2305.16147, 2023.
Liu et al. [2022]
Zuxin Liu, Zhepeng Cen, Vladislav Isenbaev, Wei Liu, Steven Wu, Bo Li, and Ding
Zhao.
Constrained variational policy optimization for safe reinforcement
learning.
In International Conference on Machine Learning, pages
13644–13668. PMLR, 2022.
Liu et al. [2023]
Zuxin Liu, Zijian Guo, Zhepeng Cen, Huan Zhang, Jie Tan, Bo Li, and Ding Zhao.
On the robustness of safe reinforcement learning under observational
perturbations.
In International Conference on Learning Representations, 2023.
McMahan [2011]
Brendan McMahan.
Follow-the-regularized-leader and mirror descent: Equivalence
theorems and l1 regularization.
In Proceedings of the Fourteenth International Conference on
Artificial Intelligence and Statistics, pages 525–533. JMLR Workshop and
Conference Proceedings, 2011.
McPherson et al. [2021]
David L McPherson, Kaylene C Stocking, and S Shankar Sastry.
Maximum likelihood constraint inference from stochastic
demonstrations.
In 2021 IEEE Conference on Control Technology and Applications
(CCTA), pages 1208–1213. IEEE, 2021.
Moskovitz et al. [2023]
Ted Moskovitz, Brendan O’Donoghue, Vivek Veeriah, Sebastian Flennerhag,
Satinder Singh, and Tom Zahavy.
Reload: Reinforcement learning with optimistic ascent-descent for
last-iterate convergence in constrained mdps.
arXiv preprint arXiv:2302.01275, 2023.
Ng et al. [2006]
Andrew Y Ng, Adam Coates, Mark Diel, Varun Ganapathi, Jamie Schulte, Ben Tse,
Eric Berger, and Eric Liang.
Autonomous inverted helicopter flight via reinforcement learning.
In Experimental robotics IX, pages 363–372. Springer, 2006.
Ratliff et al. [2009]
Nathan D Ratliff, David Silver, and J Andrew Bagnell.
Learning to search: Functional gradient techniques for imitation
learning.
Autonomous Robots, 27(1):25–53, 2009.
Ray et al. [2019]
Alex Ray, Joshua Achiam, and Dario Amodei.
Benchmarking safe exploration in deep reinforcement learning.
arXiv preprint arXiv:1910.01708, 7, 2019.
Scobee and Sastry [2019]
Dexter RR Scobee and S Shankar Sastry.
Maximum likelihood constraint inference for inverse reinforcement
learning.
arXiv preprint arXiv:1909.05477, 2019.
Silver et al. [2010]
David Silver, J Andrew Bagnell, and Anthony Stentz.
Learning from demonstration for autonomous navigation in complex
unstructured terrain.
The International Journal of Robotics Research, 29(12):1565–1592, 2010.
Sriperumbudur et al. [2009]
Bharath K Sriperumbudur, Kenji Fukumizu, Arthur Gretton, Bernhard
Schölkopf, and Gert RG Lanckriet.
On integral probability metrics,phi-divergences and
binary classification.
arXiv preprint arXiv:0901.2698, 2009.
Stooke et al. [2020]
Adam Stooke, Joshua Achiam, and Pieter Abbeel.
Responsive safety in reinforcement learning by pid lagrangian
methods.
In International Conference on Machine Learning, pages
9133–9143. PMLR, 2020.
Swamy et al. [2021]
Gokul Swamy, Sanjiban Choudhury, J. Andrew Bagnell, and Zhiwei Steven Wu.
Of moments and matching: A game-theoretic framework for closing the
imitation gap, 2021.
URLhttps://arxiv.org/abs/2103.03236.
Swamy et al. [2023]
Gokul Swamy, Sanjiban Choudhury, J Andrew Bagnell, and Zhiwei Steven Wu.
Inverse reinforcement learning without reinforcement learning.
arXiv preprint arXiv:2303.14623, 2023.
Todorov et al. [2012]
Emanuel Todorov, Tom Erez, and Yuval Tassa.
Mujoco: A physics engine for model-based control.
In 2012 IEEE/RSJ international conference on intelligent robots
and systems, pages 5026–5033. IEEE, 2012.
Vazquez-Chanlatte et al. [2018]
Marcell Vazquez-Chanlatte, Susmit Jha, Ashish Tiwari, Mark K Ho, and Sanjit
Seshia.
Learning task specifications from demonstrations.
Advances in neural information processing systems, 31, 2018.
Weng et al. [2022]
Jiayi Weng, Huayu Chen, Dong Yan, Kaichao You, Alexis Duburcq, Minghao Zhang,
Yi Su, Hang Su, and Jun Zhu.
Tianshou: A highly modularized deep reinforcement learning library.
Journal of Machine Learning Research, 23(267):1–6, 2022.
URLhttp://jmlr.org/papers/v23/21-1127.html.
Xu et al. [2019]
Kelvin Xu, Ellis Ratner, Anca Dragan, Sergey Levine, and Chelsea Finn.
Learning a prior over intent via meta-inverse reinforcement learning.
In International conference on machine learning, pages
6952–6962. PMLR, 2019.
Yu et al. [2019]
Lantao Yu, Tianhe Yu, Chelsea Finn, and Stefano Ermon.
Meta-inverse reinforcement learning with probabilistic context
variables.
Advances in neural information processing systems, 32, 2019.
Ziebart et al. [2012]
Brian Ziebart, Anind Dey, and J Andrew Bagnell.
Probabilistic pointing target prediction via inverse optimal control.
In Proceedings of the 2012 ACM international conference on
Intelligent User Interfaces, pages 1–10, 2012.
Ziebart et al. [2008a]
Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al.
Maximum entropy inverse reinforcement learning.
In Aaai, volume 8, pages 1433–1438. Chicago, IL, USA,
2008a.
Ziebart et al. [2008b]
Brian D Ziebart, Andrew L Maas, Anind K Dey, and J Andrew Bagnell.
Navigate like a cabbie: Probabilistic reasoning from observed
context-aware behavior.
In Proceedings of the 10th international conference on
Ubiquitous computing, pages 322–331, 2008b.
Zucker et al. [2011]
Matt Zucker, Nathan Ratliff, Martin Stolle, Joel Chestnutt, J Andrew Bagnell,
Christopher G Atkeson, and James Kuffner.
Optimization and learning for rough terrain legged locomotion.
The International Journal of Robotics Research, 30(2):175–191, 2011.