Literature
Display
← All papers

A Robust Approach to Belief Mismatch in Non-cooperative Dynamic Games

Ryan Park · Li-huan Shen · Jesse Milzman · Negar Mehr · David Fridovich-Keil

Preprint posted · arXiv (preprint) ↗

Robotics7 min text · CondensedOriginal paper ↗
Contents

At a glance

An ego planner guards against other agents having different state estimates by adding a hypothetical adversary that perturbs predicted belief updates. The resulting augmented game is solved locally using an iterative linear-quadratic-Gaussian method. Two simulated case studies show reduced realized costs when the opponent behaves more antagonistically than expected, with a corresponding conservatism cost in friendlier conditions. Figure 4’s 48% result concerns reduction in extra cost caused by one drift mismatch, not a 48% reduction in total cost or a general safety guarantee. This edition retains all six original figures, seven numbered equations and the original experiment dynamics/cost definitions.

I Introduction

A pedestrian may estimate a different vehicle distance and choose to cross when a car’s shared-belief model predicts yielding. The figure illustrates the motivation rather than an evaluated traffic scenario. The method keeps a separate joint-state distribution for each modeled agent, then asks how adverse errors in those estimates could change their actions. Nature acts only inside the ego’s planning problem. This reduces reliance on fictitious knowledge of other agents’ beliefs while preserving a tractable local approximation; it does not reconstruct their private measurements or solve the full incomplete-information game.

Refer to caption
Fig. 1: Simplified depiction of an interaction between an autonomous car and a pedestrian. The car plans under the assumption that the pedestrian estimates a 10 ft gap to the vehicle and will yield. However, if the pedestrian believes the distance is greater (e.g. 20 ft) and attempts to cross the street, then the car is unable to slow down in time and the two collide.

II Related Works

The work combines approximate belief-space planning with adversarial robustness. Its distinction is to perturb predicted belief dynamics, allowing opponents’ modeled responses to change indirectly rather than granting the adversary direct control of their actions.

II-A Belief Space Planning

Belief-space planning reasons over state distributions, but exact continuous or discrete formulations can be intractable. Gaussian extended-Kalman approximations, reduced beliefs, particles and search methods offer different compromises. The proposed method retains Gaussian beliefs and maximum-likelihood future observations, building on stochastic dynamic games in belief space. Its additional concern is uncertainty about the belief-update model itself, beyond uncertainty in the physical state.

II-B Robustness in Multi-Agent Interactions

Classical minimax control treats uncertain behavior adversarially, while distributionally robust methods account for uncertainty in probability models. Here the disturbance enters non-ego belief updates; modeled opponents still choose actions according to their own objectives. The construction is inspired by H-infinity control, but the multi-agent game is generally not a simple two-player zero-sum problem. Related safety filters and robust linear-quadratic formulations have different scopes and guarantees.

III Preliminaries

Superscripts index agents and subscripts index time. Omitting an index stacks its entries, a colon denotes an interval, and a negated agent index denotes everyone except that agent. The belief vector contains one state distribution per modeled agent, not one common estimate forced to be identical for all agents.

III-A Belief Space Games

Agents act under noisy dynamics and sensing, with sensor models assumed common knowledge. Each belief is approximated as Gaussian and propagated by an extended Kalman filter. During planning, unrealized future measurements are replaced by nominal maximum-likelihood observations. Agents may have different estimated means/covariances, but the augmented vector of those estimates is still treated as known within the planning game. This remaining information assumption motivates the robustness modification. Every agent’s stage cost uses its own belief over the joint state and the joint action, allowing conflicting objectives.

Equations 1–3
xt+1\displaystyle x_{t+1} =f⁡(xt,ut,mt),\displaystyle=f(x_{t},u_{t},m_{t}), mt∼𝒩⁡(0,I)\displaystyle m_{t}\sim\mathcal{N}(0,I) (1)
zti\displaystyle z^{i}_{t} =hi​(xt,nti)∈ℝηzi,\displaystyle=h^{i}(x_{t},n^{i}_{t})\in\mathbb{R}^{\eta_{z^{i}}}, nti∼𝒩⁡(0,I)\displaystyle n^{i}_{t}\sim\mathcal{N}(0,I) (2)
Ji​(bi,u):=∑t=1TCti​(bti,ut)J^{i}(b^{i},u):=\sum\limits_{t=1}^{T}C_{t}^{i}(b^{i}_{t},u_{t}) (3)

III-B Feedback Nash Equilibrium

A feedback Nash equilibrium makes each agent’s continuation strategy optimal against the others’ fixed equilibrium strategies at every modeled belief. The numerical method seeks local conditions from approximate feedback-equilibrium methods and local Nash analysis, not a globally optimal equilibrium. Strategies depend on the augmented common planning variable, whose components can differ across agents; that distinction does not make private beliefs fully observable in the real interaction.

Equations 4a–4b
γti​(bt)=arg⁡minuti\displaystyle\gamma^{i}_{t}(b_{t})=\underset{u^{i}_{t}}{\arg\min} Cti​(bti,ut)+Vt+1i​(bt+1)\displaystyle\;C^{i}_{t}(b^{i}_{t},u_{t})+V^{i}_{t+1}(b_{t+1}) (4a)
s.t.bt+1=β⁡(bt,ut)\displaystyle\text{s.t.}\quad b_{t+1}=\beta(b_{t},u_{t})\; (4b)

IV Technical Approach

The robust game is an internal decision model for the ego. The formulation adds a penalized adversarial belief disturbance, then modifies the forward belief rollout and backward local game approximation.

IV-A Problem Formulation

Label the ego N and nature N+1. Nature shifts the predicted means of non-ego Gaussian belief updates while retaining their nominal update covariances; the ego’s own belief update remains nominal. Nature minimizes its squared disturbance cost minus the ego’s stage cost plus its continuation value. A smaller positive penalty c permits stronger disturbances and therefore more conservative ego planning. Non-ego agents respond through their own cost-minimizing strategies rather than having their controls selected directly by nature. The robust equilibrium equations describe the ego’s hypothetical augmented game, not a claim that every runtime agent solves or shares it. The explanatory mean-shift paragraph inconsistently writes an N-indexed control where the surrounding equations label nature N+1; the original equations remain unchanged.

Equations 5a–6c
arg⁡minutN+1\displaystyle\underset{u^{N+1}_{t}}{\arg\min} c​‖utN+1‖2−CtN​(btN,ut)+Vt+1N+1​(bt+1)\displaystyle\;c\|u^{N+1}_{t}\|^{2}-C^{N}_{t}(b^{N}_{t},u_{t})+V^{N+1}_{t+1}(b_{t+1}) (5a)
s.t.bt+1¬N\displaystyle\mathrm{s.t.}\;b^{\neg N}_{t+1} =β^¬N​(bt¬N,ut,utN+1),\displaystyle=\hat{\beta}^{\neg N}(b^{\neg N}_{t},u_{t},u^{N+1}_{t}), (5b)
bt+1N\displaystyle b^{N}_{t+1} =βN​(btN,ut).\displaystyle=\beta^{N}(b^{N}_{t},u_{t}). (5c)
γti​(bt)=arg⁡minuti\displaystyle\gamma^{i}_{t}(b_{t})=\underset{u^{i}_{t}}{\arg\min} Cti​(bti,ut)+Vt+1i​(bt+1)\displaystyle\;C^{i}_{t}(b^{i}_{t},u_{t})\!+\!V^{i}_{t+1}(b_{t+1}) (6a)
γtN+1​(bt)=arg⁡minutN+1\displaystyle\gamma^{N+1}_{t}(b_{t})=\underset{u^{N+1}_{t}}{\arg\min} c​‖utN+1‖2−\displaystyle\;c\|u^{N+1}_{t}\|^{2}\!-
CtN​(btN,ut)+Vt+1N+1​(bt+1)\displaystyle\!C^{N}_{t}(b^{N}_{t},u_{t})\!+\!V^{N+1}_{t+1}(b_{t+1}) (6b)
s.t.bt+1=β^​(bt,\displaystyle\mathrm{s.t.}\;b_{t+1}=\hat{\beta}(b_{t}, ut,utN+1)\displaystyle u_{t},u_{t}^{N+1}) (6c)

IV-B Algorithm

A modified game-theoretic iLQG algorithm alternates forward simulation and backward dynamic programming of local quadratic approximations. Forward propagation uses the adversarial belief update. Backward cost derivatives pass through each agent’s own belief and joint actions rather than directly through all other beliefs, further reducing artificial coupling. The paper reports O((N+1) times state-dimension to the sixth power) per iteration when action and observation dimensions are comparable to state dimension; the covariance representation drives the high exponent. Polynomial iteration cost is not a bound on total iterations, convergence or global equilibrium quality. Full implementation and parameters are delegated to the companion repository; the paper contains no standalone pseudocode listing.

Refer to caption
Fig. 2: Here, we have two activist groups who attempt to influence 3 lawmakers’ opinions by fundraising and campaigning. In the experiment, the two causes are not directly opposed. However, as the activist groups have different informational channels, the groups may incorrectly model the other’s beliefs. Thus, in our experiment, we investigate robustness as a method to mitigate epistemic uncertainty about the information-gathering capabilities of non-robust activists.

V Results

Both case studies execute receding-horizon policies and measure realized ego costs under true game parameters and actual trajectories, rather than predicted planning costs. Unless varied in the mutual-robustness study, non-ego agents use the nominal belief-space baseline. Each plotted condition uses 50 Monte Carlo trials; line-plot ribbons represent standard deviations, not confidence intervals. These are computational experiments, with no physical robot or traffic deployment.

Original figure
Fig. 3: Distribution of cost difference from nominal (non-robust) incurred by the robust activist (2), across 50 Monte Carlo trials. The solid black lines denote the means of each distribution, and the baseline’s performance is the horizontal axis marked with a gray dotted line. Notice that as cc decreases and activist 2 becomes more robust, its cost decreases relative to the non-robust baseline.

V-A Example 1: Activism

The first case study is a stylized two-agent opinion-control game. It tests errors in the opponent’s noise and drift model, then examines what happens when both agents independently choose robust planning.

V-A 1 Experiment Setup

Two activists each choose six influence inputs for three lawmakers’ two-dimensional opinions. Opinion evolution adds both controls, an agent-specific drift and Gaussian noise; interpersonal influence terms from the cited model are removed. Costs combine ellipsoidal preference error with squared control effort weighted by lambda. The two causes need not be directly opposed. Agent 2’s drift is zero, and observations are Gaussian around the true opinions. The original dynamics and preference-cost equation remain available alongside the notes.

Equations 7xt+1i=xti+ut1+ut2+σdi​d+σmi​mtx^{i}_{t+1}=x^{i}_{t}+u^{1}_{t}+u^{2}_{t}+\sigma^{i}_{d}d+\sigma^{i}_{m}m_{t}
Cti​(bti,ut):=∑j(xti,j−xi,∗)⊤​Qi​(xti,j−xi,∗)+λi​‖uti‖22C^{i}_{t}(b^{i}_{t},u_{t}):=\sum_{j}(x^{i,j}_{t}-x^{i,*})^{\top}Q^{i}(x^{i,j}_{t}-x^{i,*})+\lambda^{i}||u^{i}_{t}||^{2}_{2} (7)

V-A 2 Experimental Results

In the noise-mismatch study, robust Activist 2 assumes the opponent’s noise scale is 0.75 when it is 0.02. Lower c reduces its cost relative to the nominal baseline, even though the stated costs contain no covariance-dependent term. The authors interpret this as resilience to unexpected actions as well as mistaken beliefs. In the drift sweep, true drift is zero while the assumed drift varies from −0.4 to 0.4. Positive assumed drift makes the opponent more antagonistic than predicted, and robustness helps; negative assumed drift can make extra defensive control wasteful. At assumed drift 0.4, c=0.5 reduces the additional cost relative to the zero-mismatch reference by 48%. A footnote reports that increasing c does not recover the nominal equilibrium in this activism case, even in a zero-disturbance ablation; numerical instability or equilibrium sensitivity is suggested, not resolved. With both activists robust and only Activist 2 misspecified, lowering its c helps against a fixed opponent profile, but mutual robustness raises both players’ costs relative to the non-robust pair in the tested grids. That is an empirical Pareto inefficiency, not a universal theorem.

Original figure
Fig. 4: Cost incurred by a robust activist for varying degrees of misspecified belief dynamics (σd1\sigma^{1}_{d}) and Activist 2 robustness cc. Solid lines and ribbons depict the means and standard deviations respectively. Robustness consistently lowers costs in unexpectedly adversarial environments (σd1>0\sigma^{1}_{d}>0) but can increase cost in unexpectedly collaborative environments (σd1<0\sigma^{1}_{d}<0). As cc decreases, the slope of the cost-drift curve deviates from the nominal baseline; effectively, selecting cc balances a tradeoff between the worst-case cost gain and the best-case cost reduction. The c=0.5c=0.5 level of robustness reduces the additional incurred cost at σ^d1=0.4\hat{\sigma}^{1}_{d}=0.4 by 48%, as compared to the additional cost incurred by the non-robust baseline at σ^d1=0.4\hat{\sigma}^{1}_{d}=0.4 from σ^d1=0.0\hat{\sigma}^{1}_{d}=0.0.
Refer to caption
Fig. 5: Cost change for Activist 2 under three different magnitudes of misspecified belief dynamics (σd1\sigma^{1}_{d}) across different paired robustness profiles of both activists, (c1,c2)(c_{1},c_{2}). Across all three drift mismatch grids shown, Activist 2 can reduce costs by being more robust (lower c2c_{2}) regardless of the robustness of Activist 1. However, Activist 2 suffers a higher cost when both activists are robust in comparison to when both are non-robust (baseline NR). This trend can also be observed in Activist 1’s cost despite the lack of belief dynamics mismatch for Activist 1.

V-B Example 2: Hockey

The second case study changes the attacker’s true action-effort cost while the defender keeps its nominal estimate fixed. This tests robustness to behavioral-model mismatch beyond the drift/noise changes in the activism example.

V-B 1 Experiment Setup

The attacker and defender are planar double integrators with position and velocity states and Gaussian sensing noise. Costs combine a shot-blocking geometry proxy, a distance-based stealing proxy and squared control effort. Here effort is weighted by one divided by lambda, whereas the activism example uses lambda directly. The original three-part cost definition is retained below, including the negative normalized dot-product shot term and negative squared-distance steal term. Chosen component weights and timestep are not enumerated in the paper; they are delegated to the companion implementation.

Equationhi​(xti,nti)=xti+ntih^{i}(x^{i}_{t},n^{i}_{t})=x_{t}^{i}+n^{i}_{t}
  1. a shot term: sshot​(xt1,xt2):=−w^t⊤​gtw^t⊤​w^t+gt⊤​gts_{\mathrm{shot}}(x_{t}^{1},x_{t}^{2}):=-\frac{\hat{w}_{t}^{\top}g_{t}}{\hat{w}_{t}^{\top}\hat{w}_{t}+g_{t}^{\top}g_{t}} where w^t:=wt2−wt1\hat{w}_{t}:=w_{t}^{2}-w_{t}^{1} and gt:=wgoal−wt1g_{t}:=w^{\textrm{goal}}-w_{t}^{1}. The term wgoal∈ℝ2w^{\text{goal}}\in\mathbb{R}^{2} is the fixed position of the goal. The term is a proxy for the magnitude of overlap between the attacker’s view of the goal and the defender;

  2. a steal term ssteal​(xt1,xt2):=−w^t⊤​w^ts_{\mathrm{steal}}(x_{t}^{1},x_{t}^{2}):=-\hat{w}^{\top}_{t}\hat{w}_{t} which is a distance based proxy for the likelihood of a successful defensive play, such as a steal; and

  3. a quadratic control effort term ‖uti‖22\|u^{i}_{t}\|^{2}_{2} weighted by 1λi\frac{1}{\lambda^{i}}.

V-B 2 Experimental Results

The defender assumes the attacker’s inverse effort-cost multiplier is one. Increasing the true multiplier makes attack effort cheaper and generally produces larger controls. Lower c then reduces defender costs more strongly than the nominal planner. When the attacker is more passive than expected, the extra conservatism cost stays small in the tested range. Unlike activism, increasing c approaches the nominal costs here. The observed bounded response is a property of these simulations, not a proved bound for all games or perturbation magnitudes.

Original figure
Fig. 6: Cost incurred by robust and non-robust defenders for varying inverse control cost multipliers of the attacker λ1\lambda^{1} that the defender is not aware of: believed λ^1=1\hat{\lambda}^{1}=1. Solid lines and ribbons depict the means and standard deviations. The baseline’s performance is depicted with a black dashed line. As the attacker becomes more aggressive, λ1>1\lambda^{1}>1, the more robust the defender, the lower cost it incurs. Even at low values where the attacker is passive, λ1<1\lambda^{1}<1, the cost increase incurred by the robust defender is small.

VI Conclusion

Adversarial mean perturbations in predicted non-ego beliefs improve the tested ego planner when nominal models understate opposing behavior, while exposing a clear cost of unnecessary caution. The method retains Gaussian beliefs, known sensor models, nominal future observations and a locally solved augmented-information game. Results establish a tradeoff in two stylized scenarios, not a general collision-avoidance or convergence guarantee. The paper leaves deliberate exploitation of others’ belief-model uncertainty as future work and reports an unresolved nominal-limit anomaly in the activism experiment.

Concept demonstration

Open image ↗