Overview
At a glance
CERES learns state-dependent constraints on a policy’s actions from positive and negative experience, then projects proposed actions into the predicted admissible region. A separate recovery policy helps label uncertain experience by testing whether the agent can remain alive. Simulated navigation experiments show better learning and rewards than the tested PPO baseline, including when the main policy receives fewer observations. The learned labels and constraints are approximate, so the method does not certify safety during exploration.
1 Introduction
Reward penalties can make exploration either reckless or excessively conservative. Instead of learning only a reward or an imitation policy, this paper uses demonstrations to learn which actions should be disallowed. Its supervised component separates positive from negative state–action pairs; CERES then generates these examples while learning the task and a recovery behavior. The authors’ implementation accompanies the work.








2 Background and Motivation
2.1 Reinforcement Learning
The setting is a discounted Markov decision process with continuous actions. A neural Gaussian policy is optimized with PPO using environment trajectories. Safety constraints are learned separately from the reward-driven task policy, reducing reliance on a carefully balanced collision penalty. [26]
Equation 1
| (1) |
2.2 Safe Reinforcement Learning
A quadratic program changes the proposed action as little as possible while satisfying linear inequalities. Earlier approaches supply these inequalities manually; here a network predicts them from the current state. Positive examples preserve the possibility of recovery, negative examples lead to failure, and uncertain examples require further investigation. Finite recovery trials are a heuristic for these labels, not proof that recovery is impossible. [22]
Equation 2
| (2) |
3 Learning Action-Space Constraints from Positive and Negative Demonstrations
3.1 Definitions
State-dependent action-space constraints
Each inequality defines a half-space in action coordinates. The constraint network predicts its normal and offset from the observed state. Satisfaction and violation margins measure which side of each boundary an action occupies; their geometric interpretation becomes useful after normalizing each normal vector.
Equations 3–8
| (3) |
| (4) |
| (5) |
| (6) |
| (7) | |||
| (8) |
3.2 Constraint Training Loss
A positive action should satisfy every constraint, so its loss uses the largest violation. A negative action should violate at least one, so its loss uses the smallest satisfaction margin. The combined objective is differentiable through the constraint network and can be optimized directly from labeled examples. Its raw form admits degenerate solutions, motivating the following parameterization.
Equations 9–14
| (9) |
| (10) |
| (11) |
| (12) |
| (13) |
| (14) |
3.3 Optimizing the Constraint Loss in Practice
Constraint Normalization
Zero normals and offsets trivially minimize the unnormalized loss without separating useful actions. Dividing by the normal’s predicted magnitude can create unstable scales or near-zero denominators. The paper instead constrains the normals by construction.
Unit constraint matrices
The network predicts generalized spherical angles and converts them to a Cartesian normal with radius fixed at one. This gives unit-length normals through differentiable trigonometric operations, avoiding a separately normalized, arbitrarily scaled prediction.
Avoiding constraint incompatibility
All predicted inequalities share an interior action, ensuring that their intersection is nonempty. Nonnegative offsets provide clearance around that action, with minimum and maximum bounds controlling exploration volume. A feasible interior point is not necessarily a physically safe action; feasibility of the projection and accuracy of the learned safety model are distinct.
3.4 Application
In a fixed two-dimensional maze, 500 mouse-controlled trajectories provide positive examples. Immediate failures and a task-specific reversed-path heuristic supply negatives, producing 7,228 positive and 15,994 negative pairs. A network learns two action inequalities from balanced minibatches. Classification accuracy improves with the loss, and using the constraints improves subsequent PPO rewards. These labeling shortcuts depend on the simple, effectively instantaneous position-control dynamics.


4 Constrained Exploration and Recovery from Experience Shaping
4.1 Overview
For tasks with delayed consequences, the direct policy collects trajectories toward the goal. Actions followed by sufficiently long survival can be labeled positive, and terminal failure supplies a negative example. Actions near an unresolved ending remain uncertain. A chosen survival horizon sets the practical meaning of recoverability.
4.2 Demonstration Sorting by Learning Recovery
The recovery policy starts from uncertain successor states and is rewarded for surviving rather than completing the original task. Survival for the chosen horizon marks the preceding action as recoverable; repeated failed attempts mark it negative. Labels propagate along the tested trajectory, and checking near its midpoint reduces repeated trials. The method requires identifiable outcomes and the ability to restore selected states, or an adequate replay-based substitute.
4.3 Detailed Algorithm
The retained procedures alternate direct-policy sampling, recovery trials, replay-buffer updates, and constraint learning. The policy-gradient update stores the raw policy action, treating correction as part of environment interaction; the constraint learner stores the corrected action actually executed. Confusing these two actions would change the intended learning problem.
5 Experiments
5.1 Practical Implementation
Early constraint estimates can be inaccurate, so constraints are activated probabilistically according to their current classification accuracy. Some experiments constrain only the direct policy, allowing broader recovery exploration. This explicitly permits unconstrained and potentially unsafe actions during training.
5.2 Obstacle Avoidance with Dynamics
Random-obstacle experiments compare position increments with force control, where velocity and braking make safety depend on future steps. Four variants combine these action models with full or reduced observations for the main policy. In the reduced-observation condition, the constraint network still receives the full state, including eight obstacle-distance beams. CERES improves the reported reward curves relative to matched PPO runs; it delegates obstacle information to a separate network rather than eliminating the need to sense obstacles.
6 Discussion and future work
Positive and negative experience can train useful action constraints, and recovery experiments extend the approach to delayed failures without initial expert demonstrations. The evidence is limited to simulated navigation, learned finite-horizon labels, and state-reset access. Transferring to real robots or tasks with ambiguous failure definitions remains future work; an inaccurate recovery policy can reject recoverable actions or fail to discover genuinely safe ones.