Literature
Display
← All papers

Scaling Nonlinear Optimization: Many Problems One GPU

John Viljoen · Johanna Haffner · Lasse Peters · Masayoshi Tomizuka · Negar Mehr

Preprint posted · arXiv (preprint) ↗

Robotics5 min text · CondensedOriginal paper ↗
Contents

Overview

Read this as a throughput-oriented reconstruction of IPOPT for many independent nonlinear programs. The original diagrams, plots, tables and mathematical displays remain available alongside the condensed argument. The reviewed source is arXiv v2, dated 3 November 2026.

Original figure
Fig. 1: We summarize jaxipm: each color represents a different optimization problem being executed in batch. At each iteration, every problem can take different paths, including but not limited to: backtracking line search, second-order correction, and feasibility restoration. Each of these heterogeneous paths shares some common quantities in their calculation, which we compute in parallel, followed by the remaining unique quantities sequentially.

At a glance

jaxipm implements generally constrained nonlinear optimization in JAX so hundreds of independent problems can share one GPU. Heterogeneous iteration fusion preserves each problem’s branch choices within a uniform computation graph; iteration-level batching replaces finished problems immediately. Across three quadrotor control benchmarks, one L40S GPU delivers 30.7–69.9% of the throughput of 64 concurrent CPU IPOPT instances. The gain is over sequential solvers, while the 64-core baseline remains faster.

I Introduction

Sampling-based robot learning and control can run many simulations on a GPU, but embedding a conventional nonlinear-program solver introduces CPU calls and transfers. jaxipm rebuilds IPOPT’s interior-point algorithm around two sources of divergence: different fallback branches and different numbers of iterations. Shared expensive calculations run together; remaining branch-specific calculations run for all problems before selecting the appropriate outputs. Completed slots receive new problems at iteration boundaries. This supports constrained trajectory generation on the GPU; integration into a complete learning or sampling-based control pipeline remains future work.

II Related Works

Existing nonlinear solvers provide mature constraint handling on CPUs. GPU-accelerated approaches such as MadNLP/ExaModels move derivative evaluation and linear algebra to the GPU while retaining CPU control flow. Earlier batched solvers mainly address convex programs or much smaller bound-constrained nonconvex problems. jaxipm targets general nonlinear equality/inequality constraints with independent branch decisions across a batch. Optimization-generated imitation datasets motivate the application, but dataset-training gains are not evaluated here.

III Problem Statement

Each problem minimizes a twice-differentiable objective subject to twice-differentiable equalities and elementwise bounds; general inequalities enter through slack variables. A batch shares the objective/constraint structure and bounds while varying its parameters and initial conditions. Its decision variables remain independent, making it mathematically equivalent to separate solver calls. This is concurrent solution of many programs, rather than a coupled optimization over all samples.

Predicate True branch Branch-unique computation
(1) Is the trial rejected? Backtracking line search shortens the step and retests. This occurs repeatedly until a convergence condition is met. Re-evaluation of ff and cc, filter acceptance test, reduction of α\alpha.
(2) Does the trial have surprising characteristics? A second-order correction changes the step direction based on information gathered at the original trial. This occurs repeatedly until a convergence condition is met. Modified right hand side of Equation (3), solved against the existing factorization of the left hand side of Equation (3), re-evaluation of ff and cc.
(3) Is a short run of rejected iterates hiding longer-horizon progress? The watchdog accepts a small sequence of trial steps unconditionally, either accepting the resultant trial, or reverting to the entry iterate. Copy and restore of the reference iterate and direction, counters. Scalar and vector arithmetic only.
(4) Are the steps really small? A tiny-step rule accepts it unconditionally. A convergence check using scalar arithmetic at the current trial point only.
(5) Are there repeated trial rejections? Soft feasibility restoration changes the acceptance criteria, and continues until original criteria are fulfilled or an iteration limit is reached. Re-evaluation of ff, cc and their derivatives, acceptance test.
(6) Is the optimizer unable to make progress? Hard feasibility restoration solves a new problem whose objective is the constraint violation, returning to the original problem once an acceptable point is found. None beyond the shared work; the next iteration instantiates the restoration.
(7) Is the optimization unstable? Barrier parameter update switches from an adaptive rule to a monotone schedule. Both updates are computed for every member regardless of the update rule. Scalar arithmetic only.
TABLE I: A simplified overview of jaxipm’s control flow. Branch-unique quantities and predicates are computed unconditionally, but only selected when the predicate is true.
Equations 1a–2
minx∈ℝnx\displaystyle\min_{x\in\mathbb{R}^{n_{x}}}\quad f⁡(x,θ)\displaystyle f(x,\theta) (1a)
s.t. c⁡(x,θ)=0\displaystyle c(x,\theta)=0 (1b)
xL≤x≤xU.\displaystyle x_{L}\leq x\leq x_{U}. (1c)
minXi∈ℝnxf⁡(Xi,Θi)s.t.c⁡(Xi,Θi)=0XiL≤Xi≤XiU,}∀i∈{1,…,N}\left.\begin{aligned} \min_{X_{i}\in\mathbb{R}^{n_{x}}}&\;f(X_{i},\Theta_{i})\\ \text{s.t.}&\;c(X_{i},\Theta_{i})=0\\ &\;X_{i_{L}}\leq X_{i}\leq X_{i_{U}},\end{aligned}\right\}\quad\forall i\in\{1,\dots,N\} (2)

IV IPOPT Background

IPOPT uses a shrinking logarithmic barrier, primal/dual variables and Newton steps from the constraint Jacobian and Lagrangian Hessian. Robustness requires backtracking, second-order corrections, watchdog steps, tiny-step acceptance, soft/hard feasibility restoration and alternative barrier updates. Table I preserves the conditions and work of every listed branch. Simply stacking problems into one larger IPOPT call forces shared branch choices and can increase iterations or cause failure; separate CPU calls retain independent branches but do not exploit a single GPU batch.

Equations 3
φμ​(x):=f⁡(x)−μ​∑i=1nxln⁡(x(i)−xL(i))−μ​∑i=1nxln⁡(xU(i)−x(i)),\varphi_{\mu}(x):=f(x)-\mu\sum_{i=1}^{n_{x}}\ln(x^{(i)}-x_{L}^{(i)})-\mu\sum_{i=1}^{n_{x}}\ln(x_{U}^{(i)}-x^{(i)}),
ℒ⁡(x,yc,zL,zU):=f⁡(x)+yc⊤​c​(x)−zL⊤​(x−xL)−zU⊤​(xU−x).\mathcal{L}(x,y_{c},z_{L},z_{U}):=f(x)+y_{c}^{\top}c(x)-z_{L}^{\top}(x-x_{L})-z_{U}^{\top}(x_{U}-x).
[W+ΣA⊤A0]​[Δ​xΔ​yc]=−[∇φμ​(x)+A⊤​ycc⁡(x)].\begin{bmatrix}W+\Sigma&A^{\top}\\ A&0\end{bmatrix}\begin{bmatrix}\Delta x\\ \Delta y_{c}\end{bmatrix}=-\begin{bmatrix}\nabla\varphi_{\mu}(x)+A^{\top}y_{c}\\ c(x)\end{bmatrix}. (3)
Original figure
Fig. 2: We visualize the throughput advantages of iteration-level batching versus solve-level batching. Left: We show solve-level batching achieving 3 solutions (optimizations A, B, and C) within 5 iterations, with substantial compute idle time. Right: We show iteration-level batching achieving 5 solutions (optimizations A, B, C, D, and E) within 5 iterations, with no compute idle time and a one-iteration head start on optimization F.

V Batched Interior-Point Method

The implementation computes shared functions, derivatives, Newton steps and fraction-to-boundary quantities once per iteration. It evaluates the remaining branch-specific work unconditionally and selects results by each problem’s predicate. Padding makes the ordinary and nested hard-restoration problems share a computation graph. Conditional line-search and second-order-correction loops continue across the batch until every member finishes. Iteration-level batching then replaces completed problems with fresh IPOPT-style initializations while other problems continue. This removes waiting across complete solves, although conditional inner loops still synchronize their members.

VI Results

Evaluation measures throughput, success, objective value, iteration counts and deviation from IPOPT. All experiments use double precision, a convergence tolerance of 10⁻⁸ and a 500-iteration limit on one 48 GB NVIDIA L40S and a 64-core, dual-socket AMD EPYC 9354 host. Three quadrotor benchmarks probe heterogeneous difficulty, tracking difficulty and problem dimension; no physical-robot deployment is reported.

VI-A Experimental Setup

The quadrotor model has 13 state coordinates and four rotor inputs. A 0.1-second step and 30-step horizon yield at least 506 optimization variables per vehicle before inequality slacks. IPOPT-default solves one problem at a time with a multithreaded linear solver; IPOPT-64 runs 64 independent single-core instances. MadNLP accelerates sequential solves on the GPU. The jaxipm-noIL ablation disables iteration-level replacement. Every method receives the same parameter/initialization pool, containing four times the selected GPU batch size.

Solver solves/s equiv. instances success (%) cost iters mean/max
IPOPT-default 6.99 ±\pm 0.28 1.00 100 507.6 ±\pm 17.1 21.5 / 71
IPOPT-64 310.9 ±\pm 14.8 44.5 100 507.6 ±\pm 17.1 21.5 / 71
MadNLP 2.74 ±\pm 0.02 0.39 100 507.6 ±\pm 17.2 32.2 / 184
jaxipm-noIL 14.08 ±\pm 1.12 2.01 99.55 507.6 ±\pm 17.2 71.6 / 468
jaxipm 95.45 ±\pm 1.24 13.7 100 507.6 ±\pm 16.5 51.9 / 252
TABLE II: We demonstrate our solver’s throughput advantage over IPOPT-default and MadNLP in terms of solves per second. We also note the average final cost function value and iteration count of every optimization to demonstrate that the quality of our solutions is comparable with baselines in batch, and we do not require excessive iterations to achieve them.
Original figure
Fig. 3: Left: The top-down view of a batch of optimization solutions (80/2000) from different initial positions, all of which must navigate to a common goal position whilst avoiding obstacles. We initialize quadrotors in a 180∘180^{\circ} arc around the destination. Right: The corresponding iteration count distribution.

VI-B What throughput does jaxipm achieve?

Navigation starts lie along a 180° arc around a common goal with obstacles, producing widely different solve lengths. With batch size 500, jaxipm reaches 95.45 solves/s versus 6.99 for IPOPT-default, 2.74 for MadNLP and 310.9 for IPOPT-64. That is 13.7× the sequential CPU baseline and 30.7% of the parallel CPU baseline. Removing iteration-level replacement drops throughput to 14.08 solves/s, about 14.8% of the full method. Costs are comparable; full jaxipm succeeds on all tested instances.

VI-C How does problem difficulty affect jaxipm ?

Tracking a periodic reference among known moving obstacles uses batch size 250; increasing reference speed raises difficulty. jaxipm achieves 15.3–28.9× IPOPT-default throughput and 39.4–69.9% of IPOPT-64 throughput. Its batch linear solver cannot reliably perform iterative refinement, so less accurate steps affect easy problems disproportionately and sometimes reach different local minima. Without slot replacement, success falls to 86.1% at the highest speed. The authors attribute this to converged, ill-conditioned systems remaining in the batch and affecting other linear solves; replacement removes them. The full variant reports 100% success in these tests, not a general robustness guarantee.

Original figure
Fig. 4: We visualize the reference tracking with time-varying obstacles task at v¯=1.5\bar{v}=1.5 m/s.
Solver solves/s equiv. instances success (%) cost iters mean/max
v¯=1.0\bar{v}=1.0 m/s
IPOPT-default 6.06 ±\pm 0.30 1.00 100 265.7 ±\pm 98.7 25.4 / 77
IPOPT-64 234.7 ±\pm 9.5 38.7 100 265.7 ±\pm 98.7 25.4 / 77
MadNLP 2.97 ±\pm 0.99 0.49 100 261.7 ±\pm 92.0 38.0 / 84
jaxipm-noIL 13.00 ±\pm 0.71 2.15 99.2 249.8 ±\pm 79.4 54.2 / 472
jaxipm 92.49 ±\pm 0.37 15.3 100 244.0 ±\pm 76.3 43.1 / 181
v¯=1.5\bar{v}=1.5 m/s
IPOPT-default 4.61 ±\pm 0.27 1.00 100 556.7 ±\pm 140.9 30.3 / 71
IPOPT-64 185.4 ±\pm 3.8 40.2 100 556.7 ±\pm 140.9 30.3 / 71
MadNLP 2.02 ±\pm 0.01 0.44 100 555.8 ±\pm 138.3 47.4 / 116
jaxipm-noIL 11.85 ±\pm 0.07 2.57 96.2 543.2 ±\pm 126.5 54.3 / 494
jaxipm 81.76 ±\pm 0.37 17.7 100 539.2 ±\pm 121.4 44.4 / 237
v¯=2.0\bar{v}=2.0 m/s
IPOPT-default 2.55 ±\pm 0.14 1.00 100 1146.7 ±\pm 126.9 47.8 / 125
IPOPT-64 105.4 ±\pm 3.7 41.3 100 1146.7 ±\pm 126.9 47.8 / 125
MadNLP 1.65 ±\pm 0.00 0.65 100 1147.5 ±\pm 121.2 57.8 / 136
jaxipm-noIL 11.32 ±\pm 0.22 4.44 86.1 1140.5 ±\pm 125.3 46.9 / 165
jaxipm 73.71 ±\pm 0.87 28.9 100 1142.4 ±\pm 131.5 45.8 / 129
TABLE III: Per-solve throughput and solution quality on the quadrotor reference-tracking problem with three time-varying obstacles. We control difficulty by varying the average velocity v¯\bar{v} of the reference trajectory.

VI-D How does dimensionality affect jaxipm ?

Two or four quadrotors swap positions while maintaining pairwise separation, giving 1,012 or 2,024 variables and batch size 75. jaxipm achieves 18.42 and 4.54 solves/s respectively, equivalent to 31.9 and 15.7 sequential IPOPT instances, or 69.8% and 32.5% of IPOPT-64 throughput. Larger problems reduce the useful batch size within GPU memory. Iteration replacement helps less for these relatively homogeneous initial conditions, though the four-vehicle ablation succeeds on 95% versus 100% for the full method. Four-vehicle costs and iteration counts also differ from IPOPT, consistent with the linear-solve limitations.

Original figure
Fig. 5: We visualize the multi-quadrotor consensus task running on our method with 2 and 4 quadrotors. Each colored trajectory shows one quadrotor’s path from its start to its assigned goal. The quadrotors must swap positions while maintaining a minimum separation distance with every other quadrotor throughout.
Solver solves/s equiv. instances success (%) cost iters mean/max
2 quadrotors
IPOPT-default 0.578 ±\pm 0.018 1.00 100 755.5 ±\pm 0.8 112.6 / 135
IPOPT-64 26.38 ±\pm 0.98 45.6 100 755.5 ±\pm 0.8 112.7 / 135
MadNLP 1.07 ±\pm 0.14 1.84 100 755.5 ±\pm 0.8 127.7 / 162
jaxipm-noIL 17.31 ±\pm 0.46 29.9 100 755.5 ±\pm 0.8 115.7 / 176
jaxipm 18.42 ±\pm 0.20 31.9 100 755.5 ±\pm 0.8 113.6 / 182
4 quadrotors
IPOPT-default 0.289 ±\pm 0.008 1.00 100 1529.7 ±\pm 4.7 95.5 / 159
IPOPT-64 13.99 ±\pm 0.13 48.4 100 1529.7 ±\pm 4.7 93.3 / 140
MadNLP 0.596 ±\pm 0.008 2.06 100 1529.7 ±\pm 4.7 184.1 / 217
jaxipm-noIL 2.62 ±\pm 0.03 9.06 95 1533.9 ±\pm 8.3 241.7 / 498
jaxipm 4.54 ±\pm 0.05 15.7 100 1533.7 ±\pm 8.9 227.3 / 484
TABLE IV: Per-solve throughput and solution quality on the multi-quadrotor consensus problem, where every quadrotor maintains a minimum separation from every other in the fleet.

VI-E Does jaxipm produce the same results as IPOPT?

The parity experiment saves each complete IPOPT state on difficult navigation solves, loads that state into jaxipm, executes one step and compares with IPOPT’s next state using the scaled maximum difference in Equation 4. All measured deviations across five optimizations are below 10⁻⁸, including activated restoration, correction and barrier-update branches. This is a state-reinitialized one-step comparison with conservative single-instance iterative refinement. It does not establish identical uninterrupted batched trajectories or cover every possible branch/problem; reliable iterative refinement is explicitly unavailable in the reported batch setting.

Equation 4
ev=maxi⁡|vijaxipm−viIPOPT|1+|viIPOPT|.e_{v}\;=\;\max_{i}\;\frac{\bigl\lvert v_{i}^{\texttt{jaxipm}}-v_{i}^{\text{IPOPT}}\bigr\rvert}{1+\bigl\lvert v_{i}^{\text{IPOPT}}\bigr\rvert}. (4)
Original figure
Fig. 6: We visualize the ECDF of the step-wise discrepancies between our method jaxipm and IPOPT across five full optimizations.

VII Conclusion

The contribution is a GPU-batched reconstruction of a robust nonlinear solver, with high throughput from branch fusion and prompt replacement of finished instances. The experiments support benefits over sequential IPOPT and MadNLP but show a remaining gap to 64 parallel CPU instances. GPU memory, inner-loop synchronization and batch linear-solve accuracy constrain performance. Generating trajectory datasets and embedding constrained optimization within sampling-based learning/control are proposed applications rather than demonstrated downstream results.

Concept demonstration

Open image ↗