Literature
Display
← All papers

SCoCaT: Success Conditioned Constrained Reinforcement Learning for Spacecraft Docking

Aman Arora · Ricard Marsal I Castan · Matteo El-Hariry · Miguel Olivares-Mendez

Preprint posted · CoRL 2026 (accepted) ↗

Robotics13 min text · CondensedOriginal paper ↗
Contents

Overview

Success-conditioned constrained reinforcement learning for docking: preventing policies from hovering near a goal to avoid constraint penalties, with complete theory, pseudocode, figures and ablations.

At a glance

Termination-based constraint learning can reward safe-looking inactivity when approaching a goal activates constraints and shortens future return. SCoCaT adds a success-value critic whose targets ignore those artificial terminations, then combines its advantage with the original reward advantage. Simulated planar and six-degree-of-freedom docking improve over CaT, with a measurable compliance tradeoff; four physical runs per method demonstrate limited zero-shot transfer. The method learns constraint-aware behavior but does not supply a deployment-time safety certificate.

1 Introduction

Constraints as Terminations (CaT) reduces future reward through violation-dependent survival probabilities, leaving a policy-network forward pass at deployment. Near a terminal goal, this can favor hovering just outside the active-constraint corridor. SCoCaT gives in-tolerance states a separate success signal, inspired by Direct Behavior Specification, and trains that value stream using only environment endings. This preserves CaT’s reward stream while adding an incentive to enter and remain at the goal. The spacecraft motivation is constrained onboard computation; the experiments do not establish that all model-predictive or barrier controllers are infeasible on all flight hardware.

Contributions.

The contributions are a stylized analysis of hovering, a two-critic PPO implementation and docking experiments on simulated planar/6-DoF vehicles plus an air-bearing testbed. The theoretical claims depend on reward, survival and horizon assumptions; empirical compliance is separate from formal safety.

Original figure
Original figure
Figure 1: Left: Ablation Pareto scatter Each point is one reward×\timessuccess indicator configuration; For floating platform, results show that SCoCaT occupies the ideal region (compliance ≥0.97\geq 0.97, GCO >0.5>0.5); CaT is compliant but collapses to near-zero task performance. Stars are the corresponding 6U CubeSat runs under the dense reward. Right: Constraint compliance vs. task success on two platforms. Each radar axis shows docking-corridor entry rate or per-constraint compliance (1−violation rate1{-}\text{violation rate}); larger area is better. For the Floating Platform, PPO enters the corridor but violates safety constraints. CaT satisfies constraints but fails to enter the corridor (Section 3.1). SCoCaT achieves high corridor-entry and high constraint compliance. The same pattern holds on the 6U CubeSat: CaT enters the corridor in 3.7%3.7\% of episodes and never completes the task, while SCoCaT reaches 79.4%79.4\% task completion rate (TCR; Section 3.4) at 0.9450.945 compliance against CaT’s 0.9710.971, a gap carried entirely by line-of-sight, the one constraint our tiering classifies as soft. PPO is unconstrained and buys its higher success at the lowest compliance of any method (0.8390.839), with 12×12{\times} the docking-velocity and 3×3{\times} the angular-rate violation rate of SCoCaT.

2 Related Work

Related work contrasts primal-dual constraint penalties, early-termination reformulations, safety-budget state augmentation and gradient projection. CaT’s prior locomotion evaluations emphasize continuing dense-reward tasks. SCoCaT adapts success bootstrapping from an off-policy Lagrangian approach to on-policy termination-based learning. Separate critics change the success target’s continuation rule; they do not automatically guarantee the combined policy satisfies every constraint.

Reinforcement learning for spacecraft proximity operations.

Prior spacecraft RL demonstrates docking/proximity control, while runtime assurance and barrier methods supply external constraint handling at added online computation. Air-bearing platforms and UAVs are established terrestrial analogues. This paper evaluates internalized constraints through simulated spacecraft and a planar air-bearing deployment; orbital flight and broad cross-embodiment validation remain outside its evidence.

3 Methodology

The formulation bounds discounted occupancy of individual constraint violations. CaT converts these constraints into a modified survival-weighted objective. Such a surrogate and observed violation rates are distinct from proving that the trained neural policy satisfies the original chance constraints.

Equation 1
ℙ(s,a)∼ργπ[ci(s,a)>0]≤ε~i∀i∈ℐ,\mathbb{P}_{(s,a)\sim\rho^{\pi}_{\gamma}}\!\left[c_{i}(s,a)>0\right]\leq\tilde{\varepsilon}_{i}\qquad\forall\,i\in\mathcal{I}, (1)

3.1 Problem Formulaton

Feasibility collapse means concentrating near the goal boundary while rarely entering the goal. Proposition 1’s zero-reward absorbing goal makes hovering preferable under any discounted objective; its proof does not require CaT’s goal-entry penalty. The specifically CaT-dependent case is Proposition 1b: a rewarding, non-absorbing goal becomes less valuable when repeated survival discount suppresses its return. The three-state model isolates a possible mechanism, not a complete proof that every docking failure has this cause.

3.2 Constraints as Terminations

Each constraint produces a clipped violation magnitude scaled by its severity; the largest sets the step’s termination probability. CaT weights the future reward by survival. The actual pseudocode additionally scales the current reward by one minus that probability. Hard constraints reset immediately; softer violations reduce effective return without adding an online safety optimizer.

Equation 2
JCaT​(π)=𝔼τ∼π​[∑t=0∞(∏t′=0t−1γ⁡(1−δt′))​r​(st,at)],J_{\mathrm{CaT}}(\pi)={\mathbb{E}_{\tau\sim\pi}\!\left[\sum_{t=0}^{\infty}\!\left(\prod_{t^{\prime}=0}^{t-1}\gamma(1-\delta_{t^{\prime}})\right)r(s_{t},a_{t})\right]}, (2)

3.3 Success-Conditioned Constraints as Terminations (SCoCaT)

A per-step binary success indicator requires position, orientation, linear-velocity and angular-velocity tolerances simultaneously. It supplies credit before an entire 100-step hold is completed, but remains sparse outside the narrow corridor. A continuous alternative gives smoothly increasing pose credit and a near-goal velocity penalty. It helps sparse-reward exploration but changes the practical tradeoff. The original indicator formulas are retained.

Equations 3–4
Isucc,t=[∥pt−p𝒢∥<εp]⋅[eorient,t<εo]⋅[∥vt∥<εv]⋅[∥ωt∥<εω],I_{\mathrm{succ},t}=\mathbf{1}\!\left[\|p_{t}-p_{\mathcal{G}}\|<\varepsilon_{p}\right]\cdot\mathbf{1}\!\left[e_{\mathrm{orient},t}<\varepsilon_{o}\right]\cdot\mathbf{1}\!\left[\|v_{t}\|<\varepsilon_{v}\right]\cdot\mathbf{1}\!\left[\|\omega_{t}\|<\varepsilon_{\omega}\right], (3)
Isucc,tcont=exp(−‖pt−p𝒢‖σp)⋅exp(−eorient,tσo)⋅[1−wt(1−e−∥vt∥/σve−∥ωt∥/σω)],I_{\mathrm{succ},t}^{\mathrm{cont}}=\exp\!\left(-\frac{\|p_{t}-p_{\mathcal{G}}\|}{\sigma_{p}}\right)\cdot\exp\!\left(-\frac{e_{\mathrm{orient},t}}{\sigma_{o}}\right)\cdot\left[1-w_{t}\left(1-e^{-\|v_{t}\|/\sigma_{v}}\,e^{-\|\omega_{t}\|/\sigma_{\omega}}\right)\right], (4)

Dual advantage streams.

The reward critic uses CaT-modified continuation, while the success critic bootstraps through artificial CaT terminations and stops only at environment endings. Two GAE streams therefore estimate different returns. Feeding CaT termination into both would damp success values precisely where goal approach activates constraints. This separation is the central mechanism, rather than simply appending a success bonus to the original reward.

Equations 5–6
Ar,t\displaystyle A_{r,t} =GAE⁡(rt⋅(1−δt),Vr,δt),\displaystyle=\mathrm{GAE}\!\left(r_{t}\cdot(1-\delta_{t}),\;V_{r},\;\delta_{t}\right), (5)
As,t\displaystyle A_{s,t} =GAE⁡(Isucc,t,Vs,dtenv).\displaystyle=\mathrm{GAE}\!\left(I_{\mathrm{succ},t},\;V_{s},\;d_{t}^{\mathrm{env}}\right). (6)

Advantage scalarization.

The implementation sums reward and success advantages, then standardizes their sum per batch. It does not independently normalize each stream, so their relative scale still matters: reward is multiplied by 0.1 while success is not. The stylized result removes the hover optimum when success value exceeds the lost hover return. Equal scalarization introduces no new weight in this implementation, but it is not invariant to arbitrary reward scaling or a guarantee of neural PPO convergence. The complete colored algorithm is preserved in native PDF form.

Equations 7–8
Attot=standardize⁡(Ar,t+As,t),A_{t}^{\mathrm{tot}}=\mathrm{standardize}\!\left(A_{r,t}+A_{s,t}\right), (7)
Atot​(s∂→s𝒢)=−R​γ1−γ⏟Ar,< 0+γ​Vs​(s𝒢)⏟As,> 0​ once trained.A^{\mathrm{tot}}(s_{\partial}\to s_{\mathcal{G}})\;=\;\underbrace{-\frac{R\gamma}{1-\gamma}}_{A_{r},\;<\,0}\;+\;\underbrace{\gamma\,V_{s}(s_{\mathcal{G}})}_{A_{s},\;>\,0\text{ once trained}}. (8)
Algorithm 1 SCoCaT-PPO. Red: CaT modifications [5]. Blue: SCoCaT additions.
1: policy π θ \pi_{\theta} , critics V r V_{r} , V s V_{s} ,
constraints { c i , p i max } i ∈ ℐ \{c_{i},\,p_{i}^{\max}\}_{i\in\mathcal{I}} 2: for epoch = 1 =1 to N N do 3: τ ← CollectTrajectories ​ ( π θ ) \tau\leftarrow\textsc{CollectTrajectories}(\pi_{\theta}) 4: δ t ← max i ⁡ p i max ⋅ clip ⁡ ( c i + ​ ( s t , a t ) / c i max , 0 , 1 ) \delta_{t}\leftarrow\max_{i}\;p_{i}^{\max}\cdot\mathrm{clip}(c_{i}^{+}(s_{t},a_{t})/c_{i}^{\max},\,0,\,1) 5: τ . rewards ← τ . rewards × ( 1 − δ t ) \tau.\texttt{rewards}\leftarrow\tau.\texttt{rewards}\times(1-\delta_{t}) 6: τ . dones cat ← δ t \tau.\texttt{dones}_{\mathrm{cat}}\leftarrow\delta_{t} 7: A r ← GAE ( τ . rewards , V r , τ . dones cat ) A_{r}\leftarrow\mathrm{GAE}(\tau.\texttt{rewards},\;V_{r},\;\tau.\texttt{dones}_{\mathrm{cat}}) ( 5 ) 8: A s ← GAE ( τ . I succ , V s , τ . dones env ) A_{s}\leftarrow\mathrm{GAE}(\tau.I_{\mathrm{succ}},\;V_{s},\;\tau.\texttt{dones}_{\mathrm{env}}) ( 6 ) 9: A tot ← standardize ⁡ ( A r + A s ) A^{\mathrm{tot}}\leftarrow\mathrm{standardize}(A_{r}+A_{s}) ( 7 ) 10: UpdatePolicy ​ ( π θ , A tot ) \textsc{UpdatePolicy}(\pi_{\theta},\;A^{\mathrm{tot}}) 11: UpdateCritic ( V r , τ . rewards , τ . dones cat ) \textsc{UpdateCritic}(V_{r},\;\tau.\texttt{rewards},\;\tau.\texttt{dones}_{\mathrm{cat}}) 12: UpdateCritic ( V s , τ . I succ , τ . dones env ) \textsc{UpdateCritic}(V_{s},\;\tau.I_{\mathrm{succ}},\;\tau.\texttt{dones}_{\mathrm{env}}) 13: end for

3.4 Implementation Details

Implementation separates training success signals, evaluation metrics and constraint severity. These definitions are essential to interpreting the reported gains.

Task definition.

Training requires 100 consecutive in-tolerance steps, with 10 mm position tolerance and orientation tolerance 2° on the planar platform or 10° on CubeSat. Evaluation relaxes planar orientation to 0.1 rad and tightens linear velocity from 0.05 to 0.01 m/s; angular velocity remains below 5°/s. Entry means ever reaching the pose corridor; TCR means occupying it for at least 10% of an episode, without requiring consecutive steps; GCO is time occupancy. Hold requires 100 consecutive pose-corridor steps, while declared success additionally applies the terminal velocity gate. TCR is therefore a dwell diagnostic, not the declared success criterion. Final errors marked with a star condition on corridor entry and omit failed-to-enter episodes. Constraint severity distinguishes hard bounds/collision, hardware-related limits and softer preferences.

4 Experiments

Both platforms use eight binary thrusters; CubeSat’s oblique corner thrusters require coordinated actuation for translations and rotations. Five training seeds share Isaac Lab/PPO infrastructure and a 50,000-step tolerance curriculum. Comparisons include unconstrained PPO, CaT with/without curriculum, success-bonus variants and a gradient-projection sibling. Unconstrained PPO is a reference tradeoff, not a mathematical upper bound on achievable success. Only the planar platform is tested physically.

4.1 Simulation Results

On the planar simulation, dense-reward SCoCaT reaches 90.6±6.6% declared success, versus zero for PPO and CaT, with 98.8% aggregate compliance versus CaT’s 99.4%. The headline 250-fold increase compares TCR 1.000 to 0.004, not declared docking success. On CubeSat, SCoCaT’s 61.3% declared success exceeds CaT’s zero but falls below PPO’s 93.1%; compliance is 94.5%, versus 97.1% and 83.9%. Seeds are highly variable. Dense/terminal-proximity planar rewards perform similarly, but binary success fails with sparse terminal reward; continuous success partly recovers it. Adding success directly as a reward bonus raises dwell while worsening constraint violations. High CaT compliance can reflect avoiding the corridor rather than successful controlled docking.

Sim-to-Real Transfer on Floating Platform Testbed.

Policies trained with terminal-proximity reward transfer without hardware fine-tuning to an OptiTrack-observed air-bearing platform with 5 Hz binary thrusters. Training randomizes mass by ±0.15 kg and applies constant disturbance force/torque magnitudes up to 0.2 N/0.2 Nm. Hardware scoring is pose-only at 10 mm/0.1 rad: PPO succeeds 4/4, CaT 0/4 and SCoCaT 3/4. SCoCaT reduces several violations versus PPO, but angular velocity is not a pass/fail gate. The CaT-versus-SCoCaT comparison has reported Fisher p=0.14; four runs per method support feasibility, not a powered superiority claim or the full simulation success criterion. Figure 2’s right panel is simulation occupancy, not a hardware learning curve.

Refer to caption
(a) Air-bearing platform testbed showing a successful SCoCaT approach trajectory into the goal corridor.
Original figure
(b) Corridor occupancy per timestep (mean ±1​σ\pm 1\sigma, 5 seeds) through the 100-step hold window (shaded).
Figure 2: Sim-to-real docking evaluation (Section 4.1). Left: Hardware setup and target interface. Right: SCoCaT Simulation result for Floating Platform showing average corridor occupancy.
Table 1: Sim-to-real results on the floating platform testbed (4 runs per method, zero-shot). Constraint violation rates are the fraction of timesteps in violation (↓\downarrow). v𝒢v_{\mathcal{G}} and ω𝒢\omega_{\mathcal{G}}: mean linear and angular body velocity measured while inside the 10 mm goal corridor. PPO uses the same terminal-proximity reward as CaT and SCoCaT (fair comparison). Bold: best per column.
Task Constraint violation rate (↓\downarrow) In-corridor velocity (↓\downarrow)
Method dfd_{f} (cm)↓\downarrow Hold (ts)↑\uparrow SR LoS AngVel Actions v𝒢v_{\mathcal{G}} (m/s) ω𝒢\omega_{\mathcal{G}} (rad/s)
PPO 2.4±2.12.4\pm 2.1 41.8±12.3\mathbf{41.8\pm 12.3} 𝟒/𝟒\mathbf{4/4} 0.855±0.1740.855\pm 0.174 0.592±0.045\mathbf{0.592\pm 0.045} 0.012±0.0190.012\pm 0.019 0.022±0.0020.022\pm 0.002 0.111±0.0060.111\pm 0.006
CaT 4.0±2.24.0\pm 2.2 11.5±3.711.5\pm 3.7 0/40/4 0.121±0.136\mathbf{0.121\pm 0.136} 0.628±0.1600.628\pm 0.160 0.000±0.000\mathbf{0.000\pm 0.000} 0.012†\mathbf{0.012}^{\dagger} 0.039†\mathbf{0.039}^{\dagger}
SCoCaT 1.0±0.9\mathbf{1.0\pm 0.9} 23.0±9.423.0\pm 9.4 3/43/4 0.153±0.1510.153\pm 0.151 0.682±0.0920.682\pm 0.092 0.000±0.000\mathbf{0.000\pm 0.000} 0.024±0.0010.024\pm 0.001 0.142±0.0200.142\pm 0.020
† n=1n{=}1 run entered the corridor; single-trial value.

5 Conclusion

Decoupling success continuation from artificial constraint termination substantially improves docking engagement over CaT in these experiments. The planar simulation meets the declared criterion reliably, while CubeSat retains a safety/success and seed-stability tradeoff. Physical transfer demonstrates feasibility under pose-only scoring; it does not establish orbital readiness or certified constraint satisfaction.

6 Limitations

Limitations include mission-dependent scalarization/severity choices, PPO-only implementation, a systematic angular-velocity sim-to-real gap, CubeSat compliance loss and bimodal seeds, and no observation-noise randomization. Hardware testing has four runs per method and omits velocity from its success criterion. Generalization beyond the two docking embodiments remains untested. The CubeSat aggregate compliance gap is mainly line-of-sight, but the original table also shows a small angular-rate difference, so “entirely” line-of-sight is stronger than the tabulated values.

Acknowledgments

The authors acknowledge European Union grant 101161690 for the Ice2Thrust (S4I2T) project and reviewer/area-chair feedback.

Appendix A Proofs

The proofs compare stylized hover and goal-reaching returns. They clarify which assumptions yield a CaT-specific pathology and how success-value scaling changes the preference. Original formulas remain available, including a survival-indexing inconsistency noted below.

Proof of Proposition 1

Proposition 1 compares a positive-reward hover state with an absorbing goal that yields no future reward. Hover dominance in that model follows from giving up the remaining reward horizon.

Setup.

The three states are far, just outside the goal and inside the goal. The outside state has reward R and zero artificial termination; the absorbing goal has zero reward. These assumptions deliberately simplify docking dynamics and exploration.

Value of the hover policy.

Hovering forever collects the geometric discounted sum R/(1−gamma), because no constraint termination applies there. This establishes the reference value used in both propositions.

Equation 9
VCaT⋆​(s∂)=∑t=0∞γt​R=R1−γ.V^{\star}_{\mathrm{CaT}}(s_{\partial})=\sum_{t=0}^{\infty}\gamma^{t}R=\frac{R}{1-\gamma}. (9)

Value of any goal-entering policy.

Entering the absorbing goal at any finite step truncates the positive hover-reward series, producing strictly smaller return. This proves the stated toy-model preference, but the entry constraint itself is unnecessary for this particular argument.

Equation 10
VCaTπ′​(s∂)=R​1−γk1−γ≤R1−γ−R​γk1−γ<R1−γ=VCaT⋆​(s∂).V^{\pi^{\prime}}_{\mathrm{CaT}}(s_{\partial})=R\,\frac{1-\gamma^{k}}{1-\gamma}\;\leq\;\frac{R}{1-\gamma}-\frac{R\gamma^{k}}{1-\gamma}\;<\;\frac{R}{1-\gamma}=V^{\star}_{\mathrm{CaT}}(s_{\partial}). (10)

Scope of the argument, and the role of condition (ii).

Proposition 1b instead gives a non-absorbing goal positive reward. Under the appendix’s survival convention, hovering wins when R exceeds kappa times goal reward, with kappa about 0.023 at gamma=0.99 and termination probability 0.3. The source’s Equation 12 writes a series beginning at t=0 with factor (1−delta)^t, but its closed form includes an extra leading (1−delta). Equation 2 likewise begins survival before the current step, whereas Algorithm 1 scales current reward. The original equations are preserved; their off-by-one convention is not silently repaired. The qualitative conclusion that repeated survival discount suppresses goal value remains distinct from this exact threshold convention.

Equations 11–12
R>κ​r𝒢,κ=(1−δ¯)​(1−γ)(1−γ)+γ​δ¯.R\;>\;\kappa\,r_{\mathcal{G}},\qquad\kappa\;=\;\frac{(1-\bar{\delta})(1-\gamma)}{(1-\gamma)+\gamma\bar{\delta}}. (11)
VCaTδ​(s𝒢)=∑t=0∞γt​(1−δ¯)t​r𝒢=r𝒢​(1−δ¯)1−γ⁡(1−δ¯).V^{\delta}_{\mathrm{CaT}}(s_{\mathcal{G}})=\sum_{t=0}^{\infty}\gamma^{t}(1-\bar{\delta})^{t}\,r_{\mathcal{G}}=\frac{r_{\mathcal{G}}(1-\bar{\delta})}{1-\gamma(1-\bar{\delta})}. (12)

Proof of Proposition 2

The second proof adds a success return that continues independently of CaT’s artificial termination. It compares goal-reaching-and-staying behavior with hovering, under its stated value and horizon assumptions.

Setup.

Define the combined objective as CaT reward return plus an environment-terminated success return. In the idealized infinite-horizon comparison, the successful policy remains in the goal and receives one success unit per step. Actual reset-after-hold behavior is handled separately by the finite-horizon refinement.

Objective comparison.

Hovering receives no success return; reaching the goal at step k adds a discounted success stream. Under the binary infinite-horizon model, the difference is positive when the scaled hover reward R is below one. This is a conditional comparison of policies, not a statement that an untrained critic immediately supplies that value.

Equations 13–15
JCaT​(π′)\displaystyle J_{\mathrm{CaT}}(\pi^{\prime}) =R​1−γk1−γ,\displaystyle=R\,\frac{1-\gamma^{k}}{1-\gamma}, (13)
Js​(π′)\displaystyle J_{s}(\pi^{\prime}) =∑t=k∞γt=γk1−γ.\displaystyle=\sum_{t=k}^{\infty}\gamma^{t}=\frac{\gamma^{k}}{1-\gamma}. (14)
JSCoCaT​(π′)−JSCoCaT​(π⋆)=γk​(1−R)1−γ>0,J_{\mathrm{SCoCaT}}(\pi^{\prime})-J_{\mathrm{SCoCaT}}(\pi^{\star})=\frac{\gamma^{k}(1-R)}{1-\gamma}>0, (15)

The threshold on R , in three forms.

The stated thresholds are R<1 for binary infinite-horizon success, R<0.67 using the claimed in-tolerance continuous-signal floor, and about R<0.64 for a 598-step horizon with a 100-step hold. Reward scaling gives R≤0.099 or 0.198 for the two dense configurations, while success stays unscaled. The appendix checks these sufficient bounds; the continuous floor depends on the specified smoothing/tolerance assumptions, and the separate refinements should not be read as an exhaustive theorem for arbitrary rewards or critic errors.

Equation 16
JSCoCaT(π′)−JSCoCaT(π⋆)=γk​[(1−γT)−R⁡(1−γH−k)]1−γ> 0⟺R<1−γT1−γH−k,J_{\mathrm{SCoCaT}}(\pi^{\prime})-J_{\mathrm{SCoCaT}}(\pi^{\star})=\frac{\gamma^{k}\big[(1-\gamma^{T})-R\,(1-\gamma^{H-k})\big]}{1-\gamma}\;>\;0\quad\Longleftrightarrow\quad R\;<\;\frac{1-\gamma^{T}}{1-\gamma^{H-k}}, (16)

Local optimality.

The proof mixes an arbitrarily small amount of a better goal-reaching policy into the hover policy to argue hover is not a local optimum of the idealized objective. This policy-space reasoning does not establish convergence of a parameterized, finite-sample PPO implementation.

Advantage interpretation.

The advantage interpretation separates lost hover reward from discounted success value. Goal entry becomes favorable once the success estimate exceeds the hover-value threshold. Using CaT endings in the success critic could reduce that estimate and restore the failure. The claimed value convergence is part of the stylized argument, not a general monotonicity guarantee for nonlinear critic training.

Equations 17–18
Ar​(s∂→s𝒢)\displaystyle A_{r}(s_{\partial}\to s_{\mathcal{G}}) =R+γ⋅0−R1−γ=−R​γ1−γ,\displaystyle=R+\gamma\cdot 0-\frac{R}{1-\gamma}=-\frac{R\gamma}{1-\gamma}, (17)
As​(s∂→s𝒢)\displaystyle A_{s}(s_{\partial}\to s_{\mathcal{G}}) =0+γ​Vs​(s𝒢)−Vs​(s∂)=γ​Vs​(s𝒢).\displaystyle=0+\gamma\,V_{s}(s_{\mathcal{G}})-V_{s}(s_{\partial})=\gamma\,V_{s}(s_{\mathcal{G}}). (18)

Appendix B Reward Bonus vs. Two-Critic: Why They Differ

A success bonus and a separate success critic have different continuation semantics. Dense planar bonus variants attain high occupancy but worsen docking-velocity violations from 0.001 to 0.037 and line-of-sight violations from 0.066 to 0.826 compared with SCoCaT. These measured differences motivate the separate critic, without proving all scalarizations preserve constraints.

Bellman decomposition.

An additive bonus inserted after current-reward survival scaling is still bootstrapped using CaT termination. Its value decomposes into the CaT reward value and a survival-damped success value. SCoCaT’s success critic instead uses environment-only continuation, so the fixed-point targets differ.

Equations 19–20
rtaug=rt​(1−δt)+λb​Isucc,t,r_{t}^{\mathrm{aug}}=r_{t}(1-\delta_{t})+\lambda_{b}\,I_{\mathrm{succ},t}, (19)
Vaug=Vrδ+λb​V~sδ,V^{\mathrm{aug}}=V_{r}^{\delta}+\lambda_{b}\,\tilde{V}_{s}^{\delta}, (20)

Why constraint compliance degrades.

The bonus competes with constraint pressure inside a single reward/value stream, and the ablation shows substantially worse compliance. The prose says the bonus can fire despite excessive velocity, but the binary indicator defined in Equation 3 explicitly gates on velocity; that explanation is internally inconsistent if the same indicator is used. The empirical result is retained without treating that sentence as a verified implementation detail.

Equation 21
Aaug​(s∂→s𝒢)=−R​γ1−γ⏟Ar+λb​A~sδ​(s∂→s𝒢)⏟damped by ​δA^{\mathrm{aug}}(s_{\partial}\to s_{\mathcal{G}})=\underbrace{-\frac{R\gamma}{1-\gamma}}_{A_{r}}+\lambda_{b}\underbrace{\tilde{A}^{\delta}_{s}(s_{\partial}\to s_{\mathcal{G}})}_{\text{damped by }\delta} (21)

Why the two-critic formulation preserves constraint compliance.

The two-critic formulation leaves the reward advantage’s CaT penalty intact and adds a success advantage with different continuation. Binary success additionally requires velocity tolerance. This structural separation explains the intended mechanism, but the combined policy gradient can still trade objectives; the reported nonzero violation rates are evidence against reading “preserves compliance” as an unconditional guarantee.

Appendix C Training Details

Episodes start 0.01–4 m from a randomly positioned goal, at zero linear/angular velocity. Planar heading spans 0–180°; CubeSat yaw spans 0–180° and pitch/roll 0–90° independently. The goal lies within 2 m of the origin. These training distributions define the reported generalization envelope.

C.1 Constraints

Table 3 retains all constraint thresholds and severity values. Hard bounds and planar collision reset immediately. Docking-speed limits apply within 0.5 m: 0.05 m/s planar and 0.1 m/s CubeSat, distinct from the stricter success gates. Angular thresholds/severities also differ by platform, despite the main text’s broad “same values” description. Fuel and line-of-sight remain softer terms. The curriculum raises medium/soft severities from 0.05 to 0.125 over an episode; that is a separate baseline configuration.

Sensitivity Analysis for p i max .

The three-seed severity sweep reduces fuel violations from roughly 63% to 3% and keeps docking-speed violations near/below 1.1%, while line-of-sight remains above 84%. However, line-of-sight severity is not swept, and this earlier constraint implementation predates the paper’s later speed definition; its absolute rates are not comparable to the main table’s 0.066 line-of-sight value. The sweep demonstrates nonuniform response, not that simply increasing line-of-sight severity was tested and failed.

Table 2: Per-constraint violation rates (%, ↓\downarrow) under varying pmaxp^{\max} severity on the Floating Platform (SCoCaT, 3 seeds). The sweep varies pmaxp^{\max} for docking velocity, angular rate and fuel only; effective values are shown in that order. Line-of-sight is not swept, so the LoS column reports the rate under each swept combination. These runs use 3 seeds and a constraint implementation that predates the one used for all other tables; only the trend across severity levels is comparable to Table 9, where the default configuration gives 0.0660.066.
Regime Eff. pmaxp^{\max} (dock / ang / fuel) Dock.Vel.↓\downarrow Fuel↓\downarrow LoS↓\downarrow
Soft 0.075 / 0.05 / 0.025 1.1±0.61.1\pm 0.6 63.3±3.063.3\pm 3.0 98.7±0.398.7\pm 0.3
Mid 0.30 / 0.20 / 0.10 0.5±0.30.5\pm 0.3 39.7±5.639.7\pm 5.6 98.5±0.398.5\pm 0.3
Hard 1.0 / 1.0 / 1.0 0.1±0.20.1\pm 0.2 3.2±0.83.2\pm 0.8 84.0±0.784.0\pm 0.7
Table 3: Active CaT constraints for both platforms. pimaxp_{i}^{\max} is the maximum per-step termination probability at full violation; δt\delta_{t} scales linearly with violation magnitude [5]. Hard constraints trigger immediate resets. FP = Floating Platform; CS = CubeSat. “–”: constraint inactive on that platform.
Threshold pimaxp_{i}^{\max}
ID Constraint FP CS FP CS Definition
𝒞bounds\mathcal{C}_{\text{bounds}} Out-of-bounds ‖p‖2>8​m\|p\|_{2}>8\,\text{m} ‖p‖2>8​m\|p\|_{2}>8\,\text{m} 1.01.0 1.01.0 Exits workspace boundary.
𝒞coll\mathcal{C}_{\text{coll}} Collision Fcontact>0.1​NF_{\text{contact}}>0.1\,\text{N} – 1.01.0 – Contact force with target.
𝒞vd\mathcal{C}_{v_{d}} Docking velocity ‖v‖2>0.05​m/s\|v\|_{2}>0.05\,\text{m/s} ‖v‖2>0.1​m/s\|v\|_{2}>0.1\,\text{m/s} 0.30.3 0.30.3 Linear speed within 0.5​m0.5\,\text{m} of port.
𝒞ω\mathcal{C}_{\omega} Angular velocity |ω|>0.1​rad/s|\omega|>0.1\,\text{rad/s} |ω|>0.3​rad/s|\omega|>0.3\,\text{rad/s} 0.20.2 0.10.1 Body angular rate during approach.
𝒞fuel\mathcal{C}_{\text{fuel}} Fuel consumption ‖u‖1>5.0\|u\|_{1}>5.0 ‖u‖1>6.0\|u\|_{1}>6.0 0.10.1 0.10.1 ℓ1\ell_{1}-norm of thrust commands per step.
𝒞LoS\mathcal{C}_{\text{LoS}} Line of sight ∠⁡(h^,p^𝒢)>30​°\angle(\hat{h},\,\hat{p}_{\mathcal{G}})>30\degree 0.0150.015 0.0150.015 Heading deviation from port direction.
  Hard (pmax=1.0p^{\max}=1.0): immediate reset.   Medium: hardware-critical.   Soft: operational preference.

C.2 Observations

Observations include body-frame velocities, relative goal information and the previous action. Planar angles use sine/cosine pairs; CubeSat orientation uses the first two columns of a rotation matrix. Table 4’s base state dimensions are eight and fifteen before the separately concatenated previous action. State estimation comes from simulation or motion capture; degraded-observation robustness is untested.

Table 4: Observation Space for the Docking Tasks.
Task Dimension Variables Components
(a) Floating Platform (3-DoF)
GoToPose 88
[vx,vy,ωz][v_{x},\;v_{y},\;\omega_{z}]
[d,cos⁡(α),sin⁡(α),cos⁡(ψ),sin⁡(ψ)][d,\;\cos(\alpha),\;\sin(\alpha),\;\cos(\psi),\;\sin(\psi)]
Base velocities
Target information
(b) CubeSat (6 DoF)
GoToPose3D 1515
[vx,vy,vz,ωx,ωy,ωz][v_{x},\;v_{y},\;v_{z},\;\omega_{x},\;\omega_{y},\;\omega_{z}]
[Δpx,Δpy,Δpz,𝐑][\Delta_{p_{x}},\;\Delta_{p_{y}},\;\Delta_{p_{z}},\;\mathbf{R}]
Base velocities
Target information

C.3 Hyperparameters

Policy, reward critic and success critic have independent weights, each with two 64-unit ELU hidden layers. The policy is multi-categorical for binary thrusters; critics are deterministic regressors. Training uses 4,096 environments, 32-step rollouts, eight learning epochs, gamma 0.99, GAE lambda 0.95 and reward scale 0.1. The complete table retains PPO clipping, adaptive learning-rate and normalization choices, including no time-limit bootstrap.

Table 5: Hyperparameter Configuration
Hyperparameter Value
Rollouts per update 32
Number of environments 4,096
Learning epochs per update 8
Mini-batches per epoch 8
Discount factor γ=0.99\gamma=0.99
GAE λ\lambda 0.95
Optimiser Adam
Learning rate α=5×10−4\alpha=5\times 10^{-4}
LR scheduler KLAdaptiveLR
   KL target threshold 0.008
   αmin\alpha_{\text{min}} 10−610^{-6}
   αmax\alpha_{\text{max}} 10−210^{-2}
   KL factor (halve/double) 2.0
   LR factor 1.5
Gradient norm clip 1.0
ℒπ\mathcal{L}_{\pi}: ratio clip ε\varepsilon 0.2
ℒV\mathcal{L}_{V}: value clip 0.2
Clip predicted values True
ℒH\mathcal{L}_{H}: entropy scale 0.0
ℒV\mathcal{L}_{V} scale 2.0
KL early-stop threshold 0 (disabled)
State pre-processor RunningStandardScaler
   εRSS\varepsilon_{\text{RSS}} 10−810^{-8}
   clip threshold 5.0
Value pre-processor RunningStandardScaler
Reward shaper scale 0.1
Random exploration steps 0
Time-limit bootstrap False

C.4 Reward Formulations

Terminal-proximity reward is smooth in final pose but emitted only at episode end; sparse reward is a terminal binary pose test. Dense reward includes pose attraction, progress, boundary and velocity terms and can be negative. Although the prose says safety enters exclusively through terminations, these dense boundary/velocity penalties overlap safety-related quantities. All exact formulas remain in Table 6; the practical objective should be read from those formulas.

Table 6: Reward formulations. dp,td_{p,t}: distance to docking port; eθ,te_{\theta,t}: heading (3-DoF) or orientation (6-DoF) error; Δ​dt=dp,t−1−dp,t\Delta d_{t}\!=\!d_{p,t-1}-d_{p,t}: one-step closing distance; dmaxd_{\max}: workspace boundary distance; c(x)=clamp(x−x0, 0,⋅)c(x)\!=\!\mathrm{clamp}(x-x_{0},\,0,\,\cdot): velocity dead-zone clamp; 𝟏​[term]\mathbf{1}[\mathrm{term}]: episode-termination indicator; (εp,εθ)(\varepsilon_{p},\varepsilon_{\theta}): success tolerances.
Reward rtr_{t} Notes
Terminal-proximity 𝟏​[term]⋅exp⁡(−dp,tσp)⋅exp⁡(−eθ,tσo)\mathbf{1}[\mathrm{term}]\cdot\exp\!\left(-\frac{d_{p,t}}{\sigma_{p}}\right)\cdot\exp\!\left(-\frac{e_{\theta,t}}{\sigma_{o}}\right) Dense signal
at episode-end
Dense wpose​exp⁡(−dp,tσp)​exp⁡(−eθ,tσo)w_{\mathrm{pose}}\,\exp\!\left(-\tfrac{d_{p,t}}{\sigma_{p}}\right)\exp\!\left(-\tfrac{e_{\theta,t}}{\sigma_{o}}\right)
+wprog​Δ​dt​(dmax−dp,t)+\;w_{\mathrm{prog}}\,\Delta d_{t}\,(d_{\max}-d_{p,t})
−wbnd​exp⁡(−(dmax−dp,t))-\;w_{\mathrm{bnd}}\,\exp\!\bigl(-(d_{\max}-d_{p,t})\bigr)
−wv​c​(‖vt‖)−wω​c​(‖ωt‖)-\;w_{v}\,c(\|v_{t}\|)\;-\;w_{\omega}\,c(\|\omega_{t}\|)
Every-step
Sparse 𝟏​[term]⋅[dp,t<εp,eθ,t<εθ]\mathbf{1}[\mathrm{term}]\cdot\mathbf{1}\!\left[d_{p,t}<\varepsilon_{p},\;e_{\theta,t}<\varepsilon_{\theta}\right] Terminal; binary

Appendix D Full Ablation Study: Floating Platform (3-DoF)

The full planar tables show all reward/success-signal/bonus/projection combinations. Dense SCoCaT achieves near-unit entry/TCR and GCO 0.739, but declared success is 0.906 in Table 7. Sparse binary success fails to bootstrap; continuous success reaches TCR 0.492 with standard deviation 0.473. Bonus variants often improve occupancy or conditional final error while degrading compliance. Conditional error metrics exclude non-entering runs, and high-variance sparse results should not be read as reliable docking.

Table 7: Floating Platform success under the declared criterion, 5 seeds, dense reward. TCR and Compl. reproduce Tables 8 and 9; Hold and Decl. are the consecutive-dwell and full criteria of Section 3.4. TCR credits PPO where neither baseline meets the criterion.
Method TCR ↑\uparrow Hold ↑\uparrow Decl. ↑\uparrow Compl. ↑\uparrow
PPO 0.4980.498 ±\pm0.488 0.0000.000 0.0000.000 0.8030.803 ±\pm0.011
CaT 0.0040.004 ±\pm0.008 0.0000.000 0.0000.000 0.994\mathbf{0.994}
SCoCaT 1.000\mathbf{1.000} ±\pm0.001 0.906\mathbf{0.906} ±\pm0.066 0.906\mathbf{0.906} ±\pm0.066 0.9880.988 ±\pm0.002
Table 8: Task-performance ablation on the Floating Platform (3-DoF, simulation). Each condition uses 5 independent seeds. Ent.: fraction of episodes that ever entered the goal corridor (↑\uparrow). TCR: fraction of episodes in which the agent spent ≥\geq10% of timesteps inside the goal corridor (↑\uparrow). GCO: mean fraction of episode timesteps inside the goal corridor (↑\uparrow). Pos.∗/Hdg.∗: mean final position (mm) and heading (∘) error for corridor-entering episodes (↓\downarrow). Dwell∗: mean position error (mm) from first corridor entry to episode end, for corridor-entering episodes (↓\downarrow). App.: mean planar approach speed (m/s) at the step before first corridor entry (↓\downarrow). Bold: best per column among completed runs. Reward: dense (default), TP = terminal-proximity, sparse = end-of-episode binary. Signal: binary (default), cont. = continuous.
Method Ent. ↑\uparrow TCR ↑\uparrow GCO ↑\uparrow Pos.∗ (mm) ↓\downarrow Hdg.∗ (∘) ↓\downarrow Dwell∗ (mm) ↓\downarrow App. (m/s) ↓\downarrow
PPO 0.9080.908 ±\pm0.154 0.4980.498 ±\pm0.488 0.1290.129 ±\pm0.123 29.829.8 ±\pm19.7 0.7350.735 ±\pm0.074 29.229.2 ±\pm18.5 0.0390.039 ±\pm0.019
PPO (TP) 1.000\mathbf{1.000} 0.9970.997 ±\pm0.003 0.3040.304 ±\pm0.053 11.611.6 ±\pm1.5 0.7090.709 ±\pm0.044 11.411.4 ±\pm1.3 0.0260.026 ±\pm0.004
PPO (sparse) 0.0000.000 0.0000.000 0.0000.000 – – – –
CaT 0.5380.538 ±\pm0.374 0.0040.004 ±\pm0.008 0.0140.014 ±\pm0.015 37.837.8 ±\pm12.2 0.8430.843 ±\pm0.107 33.733.7 ±\pm9.2 0.016\mathbf{0.016} ±\pm0.003
CaT (TP) 0.7550.755 ±\pm0.155 0.0060.006 ±\pm0.006 0.0180.018 ±\pm0.009 26.526.5 ±\pm4.3 0.7440.744 ±\pm0.069 24.124.1 ±\pm4.3 0.0180.018 ±\pm0.003
CaT+Curr 0.1320.132 ±\pm0.214 0.0000.000 0.0020.002 ±\pm0.003 42.242.2 ±\pm19.4 0.9650.965 ±\pm0.293 28.128.1 ±\pm3.9 0.0200.020 ±\pm0.006
CaT+Curr (TP) 0.3040.304 ±\pm0.297 0.0000.000 0.0040.004 ±\pm0.005 31.231.2 ±\pm4.3 0.7990.799 ±\pm0.089 26.626.6 ±\pm1.5 0.0180.018 ±\pm0.005
CaT+Curr (sparse) 0.0000.000 0.0000.000 0.0000.000 – – – –
SCoCaT 1.0001.000 ±\pm0.001 1.0001.000 ±\pm0.001 0.7390.739 ±\pm0.014 5.55.5 ±\pm0.5 0.3000.300 ±\pm0.025 5.55.5 ±\pm0.5 0.0320.032 ±\pm0.004
SCoCaT (cont.) 0.9990.999 ±\pm0.001 0.9990.999 ±\pm0.001 0.5510.551 ±\pm0.052 7.67.6 ±\pm0.9 0.2970.297 ±\pm0.029 7.67.6 ±\pm0.8 0.0180.018 ±\pm0.003
SCoCaT (TP) 0.9990.999 ±\pm0.001 0.9990.999 ±\pm0.001 0.7510.751 ±\pm0.006 5.35.3 ±\pm0.4 0.3190.319 ±\pm0.039 5.35.3 ±\pm0.4 0.0310.031 ±\pm0.004
SCoCaT (TP, cont.) 0.9990.999 ±\pm0.001 0.9990.999 ±\pm0.001 0.5700.570 ±\pm0.041 7.37.3 ±\pm0.5 0.2680.268 ±\pm0.026 7.67.6 ±\pm0.5 0.0210.021 ±\pm0.003
SCoCaT (sparse) 0.0000.000 0.0000.000 0.0000.000 – – – –
SCoCaT (sparse, cont.) 0.4980.498 ±\pm0.474 0.4920.492 ±\pm0.473 0.2450.245 ±\pm0.262 9.79.7 ±\pm2.8 0.5670.567 ±\pm0.346 8.98.9 ±\pm2.2 0.0180.018 ±\pm0.010
SCoCaT-bonus 1.0001.000 1.0001.000 0.8160.816 ±\pm0.005 3.63.6 ±\pm0.3 0.3400.340 ±\pm0.015 3.73.7 ±\pm0.3 0.0320.032 ±\pm0.004
SCoCaT-bonus (cont.) 1.0001.000 1.000\mathbf{1.000} 0.7470.747 ±\pm0.008 3.53.5 ±\pm0.2 0.4240.424 ±\pm0.020 3.73.7 ±\pm0.2 0.0330.033 ±\pm0.004
SCoCaT-bonus (TP) 0.9990.999 ±\pm0.001 0.9990.999 ±\pm0.001 0.8150.815 ±\pm0.009 3.43.4 ±\pm0.2 0.3490.349 ±\pm0.007 3.53.5 ±\pm0.1 0.0340.034 ±\pm0.004
SCoCaT-bonus (TP, cont.) 1.0001.000 1.0001.000 0.7530.753 ±\pm0.021 3.63.6 ±\pm0.3 0.4060.406 ±\pm0.032 3.73.7 ±\pm0.4 0.0340.034 ±\pm0.005
SCoCaT-bonus (sparse) 0.0000.000 0.0000.000 0.0000.000 – – – –
SCoCaT-bonus (sparse, cont.) 0.9990.999 ±\pm0.001 0.9990.999 ±\pm0.001 0.835\mathbf{0.835} ±\pm0.010 1.5\mathbf{1.5} ±\pm0.3 0.2180.218 ±\pm0.024 1.7\mathbf{1.7} ±\pm0.3 0.0330.033 ±\pm0.009
SCoCaT (Both) 0.9990.999 ±\pm0.001 0.9990.999 ±\pm0.001 0.7670.767 ±\pm0.011 5.25.2 ±\pm0.2 0.2970.297 ±\pm0.014 5.35.3 ±\pm0.2 0.0320.032 ±\pm0.004
SCoCaT (Both, cont.) 0.9960.996 ±\pm0.007 0.9960.996 ±\pm0.007 0.6880.688 ±\pm0.028 5.45.4 ±\pm0.5 0.2280.228 ±\pm0.021 5.65.6 ±\pm0.6 0.0200.020 ±\pm0.006
SCoCaT (Both, TP) 1.0001.000 ±\pm0.001 0.9990.999 ±\pm0.001 0.7740.774 ±\pm0.013 4.64.6 ±\pm0.3 0.3630.363 ±\pm0.050 4.74.7 ±\pm0.3 0.0350.035 ±\pm0.005
SCoCaT (Both, TP, cont.) 1.0001.000 ±\pm0.001 0.9990.999 ±\pm0.002 0.6280.628 ±\pm0.090 6.36.3 ±\pm1.3 0.2290.229 ±\pm0.025 6.66.6 ±\pm1.3 0.0210.021 ±\pm0.006
SCoCaT (Both, sparse) 0.0000.000 0.0000.000 0.0000.000 – – – –
SCoCaT (Both, sparse, cont.) 1.0001.000 ±\pm0.001 1.0001.000 ±\pm0.001 0.7880.788 ±\pm0.014 4.04.0 ±\pm0.3 0.213\mathbf{0.213} ±\pm0.011 4.44.4 ±\pm0.3 0.0270.027 ±\pm0.004
CaT-PEGrad 0.4070.407 ±\pm0.381 0.3380.338 ±\pm0.328 0.0820.082 ±\pm0.081 7.27.2 ±\pm1.0 0.4890.489 ±\pm0.081 7.87.8 ±\pm1.2 0.0110.011 ±\pm0.002
CaT-PEGrad (cont.) 0.5240.524 ±\pm0.264 0.2040.204 ±\pm0.178 0.0460.046 ±\pm0.033 18.818.8 ±\pm3.6 0.4570.457 ±\pm0.063 15.015.0 ±\pm2.8 0.0120.012 ±\pm0.001
CaT-PEGrad (TP) 0.1460.146 ±\pm0.327 0.1190.119 ±\pm0.266 0.0280.028 ±\pm0.062 6.76.7 0.4200.420 7.17.1 0.0130.013
CaT-PEGrad (TP, cont.) 0.5290.529 ±\pm0.191 0.1930.193 ±\pm0.156 0.0460.046 ±\pm0.030 17.217.2 ±\pm2.2 0.5440.544 ±\pm0.085 15.615.6 ±\pm1.8 0.0140.014 ±\pm0.003
Table 9: Constraint-compliance ablation on the Floating Platform (3-DoF, simulation). Each condition uses 5 independent seeds. Compl.: 1−v¯1-\bar{v} where v¯\bar{v} is the mean per-timestep violation rate averaged over the active CaT constraints (↑\uparrow). Remaining columns: individual per-timestep constraint violation rates (↓\downarrow). The average runs over all six active constraints; out-of-bounds and collision are 0.0000.000 throughout and are omitted from the columns, see Table 13 for collision results under the obstacle-avoidance setting. Bold: best per column among completed runs. Reward: dense (default), TP = terminal-proximity, sparse = end-of-episode binary. Signal: binary (default), cont. = continuous.
Method Compl. ↑\uparrow Dock.Vel. ↓\downarrow Ang.Vel. ↓\downarrow LoS ↓\downarrow Fuel ↓\downarrow
PPO 0.8030.803 ±\pm0.011 0.0700.070 ±\pm0.037 0.0040.004 ±\pm0.001 0.8320.832 ±\pm0.025 0.2760.276 ±\pm0.048
PPO (TP) 0.7980.798 ±\pm0.014 0.0440.044 ±\pm0.003 0.0070.007 ±\pm0.001 0.8220.822 ±\pm0.094 0.3400.340 ±\pm0.052
PPO (sparse) 0.6460.646 ±\pm0.136 0.0060.006 ±\pm0.005 0.6570.657 ±\pm0.400 0.8290.829 ±\pm0.007 0.4920.492 ±\pm0.490
CaT 0.9940.994 0.0010.001 0.0000.000 0.0360.036 ±\pm0.003 0.0010.001
CaT (TP) 0.994\mathbf{0.994} ±\pm0.001 0.0010.001 0.0000.000 0.035\mathbf{0.035} ±\pm0.003 0.001\mathbf{0.001}
CaT+Curr 0.9860.986 ±\pm0.010 0.0020.002 0.0060.006 ±\pm0.001 0.0550.055 ±\pm0.031 0.0230.023 ±\pm0.036
CaT+Curr (TP) 0.9900.990 ±\pm0.005 0.0020.002 0.0070.007 ±\pm0.001 0.0420.042 ±\pm0.020 0.0090.009 ±\pm0.012
CaT+Curr (sparse) 0.7940.794 ±\pm0.099 0.0100.010 ±\pm0.007 0.0990.099 ±\pm0.124 0.6820.682 ±\pm0.377 0.4450.445 ±\pm0.356
SCoCaT 0.9880.988 ±\pm0.002 0.0010.001 ±\pm0.001 0.0000.000 0.0660.066 ±\pm0.009 0.0020.002
SCoCaT (cont.) 0.9890.989 ±\pm0.001 0.0020.002 0.0000.000 0.0650.065 ±\pm0.005 0.0020.002
SCoCaT (TP) 0.9900.990 ±\pm0.002 0.0010.001 ±\pm0.001 0.0000.000 0.0580.058 ±\pm0.011 0.0020.002
SCoCaT (TP, cont.) 0.9900.990 ±\pm0.001 0.0010.001 ±\pm0.001 0.0000.000 0.0560.056 ±\pm0.007 0.0020.002
SCoCaT (sparse) 0.8240.824 ±\pm0.133 0.0050.005 ±\pm0.006 0.1430.143 ±\pm0.317 0.5960.596 ±\pm0.322 0.3100.310 ±\pm0.423
SCoCaT (sparse, cont.) 0.9570.957 ±\pm0.056 0.001\mathbf{0.001} ±\pm0.001 0.000\mathbf{0.000} 0.2570.257 ±\pm0.334 0.0020.002 ±\pm0.004
SCoCaT-bonus 0.8260.826 ±\pm0.008 0.0370.037 ±\pm0.001 0.0040.004 ±\pm0.001 0.8260.826 ±\pm0.043 0.1780.178 ±\pm0.011
SCoCaT-bonus (cont.) 0.8320.832 ±\pm0.008 0.0390.039 ±\pm0.001 0.0040.004 ±\pm0.001 0.7970.797 ±\pm0.030 0.1650.165 ±\pm0.024
SCoCaT-bonus (TP) 0.8290.829 ±\pm0.006 0.0380.038 ±\pm0.001 0.0050.005 ±\pm0.002 0.8010.801 ±\pm0.047 0.1840.184 ±\pm0.023
SCoCaT-bonus (TP, cont.) 0.8340.834 ±\pm0.011 0.0380.038 ±\pm0.001 0.0050.005 ±\pm0.001 0.7970.797 ±\pm0.045 0.1580.158 ±\pm0.031
SCoCaT-bonus (sparse) 0.7970.797 ±\pm0.100 0.0040.004 ±\pm0.004 0.2190.219 ±\pm0.389 0.8440.844 ±\pm0.072 0.1480.148 ±\pm0.220
SCoCaT-bonus (sparse, cont.) 0.7720.772 ±\pm0.039 0.0370.037 ±\pm0.001 0.0240.024 ±\pm0.036 0.7790.779 ±\pm0.213 0.5280.528 ±\pm0.150
SCoCaT (Both) 0.9870.987 ±\pm0.007 0.0010.001 ±\pm0.001 0.0000.000 0.0740.074 ±\pm0.040 0.0020.002
SCoCaT (Both, cont.) 0.9850.985 ±\pm0.005 0.0030.003 ±\pm0.002 0.0000.000 0.0850.085 ±\pm0.028 0.0020.002
SCoCaT (Both, TP) 0.9820.982 ±\pm0.015 0.0060.006 ±\pm0.010 0.0130.013 ±\pm0.028 0.0850.085 ±\pm0.044 0.0050.005 ±\pm0.009
SCoCaT (Both, TP, cont.) 0.9870.987 ±\pm0.002 0.0020.002 ±\pm0.001 0.0000.000 0.0710.071 ±\pm0.012 0.0020.002 ±\pm0.001
SCoCaT (Both, sparse) 0.7720.772 ±\pm0.175 0.0100.010 ±\pm0.010 0.3030.303 ±\pm0.426 0.5900.590 ±\pm0.335 0.4640.464 ±\pm0.431
SCoCaT (Both, sparse, cont.) 0.9770.977 ±\pm0.002 0.0240.024 ±\pm0.002 0.0030.003 ±\pm0.001 0.1120.112 ±\pm0.010 0.0020.002 ±\pm0.001
CaT-PEGrad 0.8820.882 ±\pm0.112 0.0030.003 ±\pm0.005 0.2010.201 ±\pm0.434 0.5020.502 ±\pm0.313 0.0010.001 ±\pm0.001
CaT-PEGrad (cont.) 0.9560.956 ±\pm0.003 0.0010.001 0.0060.006 0.2550.255 ±\pm0.021 0.0010.001
CaT-PEGrad (TP) 0.7880.788 ±\pm0.114 0.0140.014 ±\pm0.010 0.4980.498 ±\pm0.450 0.7050.705 ±\pm0.243 0.0520.052 ±\pm0.054
CaT-PEGrad (TP, cont.) 0.9510.951 ±\pm0.019 0.0010.001 0.0100.010 ±\pm0.010 0.2750.275 ±\pm0.089 0.0070.007 ±\pm0.014

D.1 CaT-PEGrad.

The appendix’s PEGrad adaptation computes separate reward/success PPO gradients, projects the reward gradient orthogonal to success, rescales it and combines it with success as the primary direction. This reverses the main text’s description of projecting success against reward; the retained appendix formulas specify the reported derivation. Orthogonality is a local first-order property, not a finite-step guarantee of improved success or safety. Extra backward passes and critic-only clipping preserve that construction. It improves over CaT on some engagement measures but trails SCoCaT in the reported tradeoff.

Equations 22–25
ℒS​(θ)\displaystyle\mathcal{L}_{S}(\theta) =−𝔼⁡[min⁡(rt​(θ)​A^tS,clip⁡(rt​(θ),±ϵ)​A^tS)],\displaystyle=-\mathbb{E}\!\left[\min\!\left(r_{t}(\theta)\,\hat{A}^{S}_{t},\;\operatorname{clip}(r_{t}(\theta),1\!\pm\!\epsilon)\,\hat{A}^{S}_{t}\right)\right], (22)
ℒR​(θ)\displaystyle\mathcal{L}_{R}(\theta) =−𝔼⁡[min⁡(rt​(θ)​A^tR,clip⁡(rt​(θ),±ϵ)​A^tR)],\displaystyle=-\mathbb{E}\!\left[\min\!\left(r_{t}(\theta)\,\hat{A}^{R}_{t},\;\operatorname{clip}(r_{t}(\theta),1\!\pm\!\epsilon)\,\hat{A}^{R}_{t}\right)\right], (23)
gR⟂S=gR−gS⊤​gR‖gS‖2​gS,g_{R\perp S}=g_{R}-\frac{g_{S}^{\top}g_{R}}{\|g_{S}\|^{2}}\,g_{S}, (24)
gR⟂S←gR⟂S⋅min⁡(1,‖gS‖‖gR⟂S‖).g_{R\perp S}\leftarrow g_{R\perp S}\cdot\min\!\left(1,\;\frac{\|g_{S}\|}{\|g_{R\perp S}\|}\right). (25)

Appendix E Full Ablation Study: 6U CubeSat (6-DoF)

CubeSat binary SCoCaT has entry/TCR 0.794, hold 0.694 and declared success 0.613±0.359, compared with PPO’s 0.931±0.068 and CaT’s zero declared success. Continuous SCoCaT enters every episode but declares success only 0.450±0.489. Thus entry is not successful holding, and seed variability is substantial. Binary compliance is 0.945 versus CaT 0.971: line-of-sight violations rise 0.135→0.286, while angular violations also rise slightly 0.035→0.039 and docking-speed violations remain 0.004. The aggregate is an average across constraint types, not a probability of a violation-free episode.

Table 10: Task-performance results on the 6U CubeSat (6-DoF, simulation). 5 independent seeds; dense reward throughout. Scored at the goal corridor (position ≤\leq 10 mm, orientation ≤\leq 10∘) over the [0.01,4][0.01,4] m spawn range. Ent.: fraction of episodes that ever entered the goal corridor (↑\uparrow). TCR: fraction of episodes in which the agent spent ≥\geq10% of timesteps inside the goal corridor (↑\uparrow). Hold: fraction of episodes with ≥\geq100 consecutive in-corridor timesteps, pose only (↑\uparrow). Decl.: Hold and the terminal-velocity gate (‖v‖<0.01\|v\|<0.01 m/s, ‖ω‖<5∘\|\omega\|<5^{\circ}/s), the success criterion declared in Section 3.4 (↑\uparrow). GCO: mean fraction of episode timesteps inside the goal corridor (↑\uparrow). Pos.∗/Ori.∗: mean final position (mm) and orientation (∘) error for corridor-entering episodes (↓\downarrow). Dwell∗: mean position error (mm) from first corridor entry to episode end (↓\downarrow). App.: mean 3-D approach speed (m/s) at the step before first corridor entry (↓\downarrow). PPO is unconstrained: it upper-bounds success and lower-bounds compliance (Table 11). Bold: best per column.
Method Ent. ↑\uparrow TCR ↑\uparrow Hold ↑\uparrow Decl. ↑\uparrow GCO ↑\uparrow Pos.∗ (mm) ↓\downarrow Ori.∗ (∘) ↓\downarrow Dwell∗ (mm) ↓\downarrow App. (m/s) ↓\downarrow
PPO 1.000\mathbf{1.000} 1.000\mathbf{1.000} 1.000\mathbf{1.000} 0.931\mathbf{0.931} ±\pm0.068 0.715\mathbf{0.715} ±\pm0.011 3.6\mathbf{3.6} ±\pm0.7 0.503\mathbf{0.503} ±\pm0.058 3.8\mathbf{3.8} ±\pm0.5 0.010\mathbf{0.010} ±\pm0.001
CaT 0.0370.037 ±\pm0.084 0.0000.000 0.0000.000 0.0000.000 0.0010.001 ±\pm0.001 48.948.9 2.2452.245 49.049.0 0.0130.013
SCoCaT (bin.) 0.7940.794 ±\pm0.444 0.7940.794 ±\pm0.444 0.6940.694 ±\pm0.408 0.6130.613 ±\pm0.359 0.3970.397 ±\pm0.256 7.17.1 ±\pm0.8 4.5774.577 ±\pm1.374 7.37.3 ±\pm0.6 0.0120.012 ±\pm0.003
SCoCaT (cont.) 1.000\mathbf{1.000} 0.8500.850 ±\pm0.192 0.4690.469 ±\pm0.485 0.4500.450 ±\pm0.489 0.4560.456 ±\pm0.334 8.38.3 ±\pm3.7 1.3951.395 ±\pm0.389 8.38.3 ±\pm3.6 0.0160.016 ±\pm0.009
Table 11: Constraint-compliance results on the 6U CubeSat (6-DoF, simulation). 5 independent seeds; dense reward throughout. Compl.: 1−v¯1-\bar{v} where v¯\bar{v} is the mean per-timestep violation rate over the six active CaT constraints (↑\uparrow). Remaining columns: individual per-timestep constraint violation rates (↓\downarrow). PPO is unconstrained and lower-bounds compliance; CaT’s high compliance is not a safety result, as a policy that never enters the corridor cannot violate in-corridor constraints (Table 10). Bold: best per column.
Method Compl. ↑\uparrow Dock.Vel. ↓\downarrow Ang.Vel. ↓\downarrow LoS ↓\downarrow Fuel ↓\downarrow OoB ↓\downarrow
PPO 0.8390.839 ±\pm0.007 0.0470.047 ±\pm0.003 0.1180.118 ±\pm0.014 0.7920.792 ±\pm0.039 0.0080.008 ±\pm0.011 0.000\mathbf{0.000}
CaT 0.971\mathbf{0.971} ±\pm0.007 0.004\mathbf{0.004} ±\pm0.002 0.035\mathbf{0.035} ±\pm0.023 0.135\mathbf{0.135} ±\pm0.043 0.000\mathbf{0.000} ±\pm0.001 0.000\mathbf{0.000}
SCoCaT (bin.) 0.9450.945 ±\pm0.018 0.004\mathbf{0.004} ±\pm0.005 0.0390.039 ±\pm0.041 0.2860.286 ±\pm0.082 0.000\mathbf{0.000} 0.000\mathbf{0.000}
SCoCaT (cont.) 0.9410.941 ±\pm0.019 0.0170.017 ±\pm0.019 0.0720.072 ±\pm0.074 0.2640.264 ±\pm0.020 0.0010.001 ±\pm0.001 0.000\mathbf{0.000}

Appendix F Obstacle Avoidance Results

Adding an obstacle activates hard collision termination. Binary SCoCaT reaches the pose-corridor SR diagnostic in 0.770 of episodes, versus PPO 0.237 and CaT 0.027, with reported per-timestep collision violations rounded to zero. Aggregate compliance is 0.965 versus CaT 0.982. The table’s SR is episode success at the real-experiment pose criterion, not the fully velocity-gated declared metric. Rare collision risk cannot be inferred to be zero from rounded timestep rates. The prose calls the obstacle cylindrical and the caption spherical; both original descriptions are retained.

Table 12: Task-performance results on the Floating Platform with obstacle avoidance (GoToPoseObs, 3-DoF). A spherical obstacle is placed uniformly at random in the approach path each episode. Ent.: fraction of episodes that ever entered the goal corridor (↑\uparrow). SR: episode success rate at the real-experiment criterion (≤\leq 0.01 m, ≤\leq 0.1 rad ≈\approx 5.7∘) (↑\uparrow). Bold: best per column.
Method Ent. ↑\uparrow SR ↑\uparrow GCO ↑\uparrow Pos.∗ (mm) ↓\downarrow Hdg.∗ (∘) ↓\downarrow Dwell∗ (mm) ↓\downarrow App. (m/s) ↓\downarrow
PPO 0.913\mathbf{0.913} ±\pm0.147 0.2370.237 ±\pm0.203 0.1290.129 ±\pm0.122 29.229.2 ±\pm18.7 0.7330.733 ±\pm0.074 25.725.7 ±\pm14.4 0.0360.036 ±\pm0.010
CaT 0.4580.458 ±\pm0.335 0.0270.027 ±\pm0.028 0.0120.012 ±\pm0.014 37.237.2 ±\pm11.7 0.8810.881 ±\pm0.095 29.629.6 ±\pm8.9 0.017\mathbf{0.017} ±\pm0.003
SCoCaT 0.7790.779 ±\pm0.025 0.770\mathbf{0.770} ±\pm0.026 0.538\mathbf{0.538} ±\pm0.017 5.5\mathbf{5.5} ±\pm0.6 0.3030.303 ±\pm0.019 5.6\mathbf{5.6} ±\pm0.5 0.0390.039 ±\pm0.002
SCoCaT (cont.) 0.7690.769 ±\pm0.041 0.5900.590 ±\pm0.066 0.4150.415 ±\pm0.037 7.57.5 ±\pm0.9 0.294\mathbf{0.294} ±\pm0.031 7.77.7 ±\pm0.8 0.0240.024 ±\pm0.003
Table 13: Constraint-compliance results on the Floating Platform with obstacle avoidance (GoToPoseObs, 3-DoF). The collision constraint 𝒞coll\mathcal{C}_{\text{coll}} (pmax=1.0p^{\max}=1.0) is active in addition to all constraints in Table 3. Out-of-bounds violations are omitted (never violated). Bold: best per column.
Method Compl. ↑\uparrow Collision ↓\downarrow Dock.Vel. ↓\downarrow Ang.Vel. ↓\downarrow LoS ↓\downarrow Fuel ↓\downarrow
PPO 0.8040.804 ±\pm0.011 0.0010.001 0.0660.066 ±\pm0.034 0.0040.004 ±\pm0.001 0.8310.831 ±\pm0.025 0.2740.274 ±\pm0.047
CaT 0.982\mathbf{0.982} ±\pm0.003 0.000\mathbf{0.000} 0.001\mathbf{0.001} 0.000\mathbf{0.000} 0.107\mathbf{0.107} ±\pm0.015 0.001\mathbf{0.001}
SCoCaT 0.9650.965 ±\pm0.004 0.0000.000 0.0010.001 0.0010.001 0.2080.208 ±\pm0.022 0.0020.002
SCoCaT (cont.) 0.9660.966 ±\pm0.006 0.0000.000 0.0010.001 0.0010.001 ±\pm0.001 0.2030.203 ±\pm0.033 0.0020.002
Concept demonstration

Open image ↗