Literature
Display
← All papers

Shape Formation for the Cooperative Transportation of Arbitrary Objects Using Multi-Agent Reinforcement Learning

Mohamed Sayed · Wolfram Burgard · Tanja Katharina Kaiser

Preprint posted · arXiv (preprint) ↗

Robotics8 min text · CondensedOriginal paper ↗
Contents

At a glance

This edition follows the complete eight-page, 8 June 2026 v1, retaining all six figures and their panels, four tables, 27 numbered equations and 22 references.

Shared multi-agent policies learn to arrange mobile robots beneath an irregular object while avoiding obstacles and balancing an estimated load. Training combines local motion sensing with team positions/loads and a centralized critic. The simulation results favor load-aware formations over geometry-only baselines. The study concerns preparation for transport; lifting, carried-object dynamics and physical deployment are not evaluated.

I INTRODUCTION

Cooperative transport can use pushing, grasping, caging or support from below. Kiva-style systems rely on matched robot/object interfaces, while flexible support sheets add compliance and tension. This work instead asks robots to choose their own support positions beneath a rigid object with an irregular footprint and shifted center of mass.

The object is already suspended above the robots. The learned task is to enter its footprint and distribute an algebraic load estimate evenly; a balanced formation is a prerequisite for later transport, not evidence that the object has been lifted or carried safely.

Complete original PDF block S1.F1
Fig. 1: Overview of formation task for object carrying with (a) the scene description, as well as examples of (b) a valid formation and (c) invalid formations.
Original panel labels
(a)
(b)
(c)

II RELATED WORK

II-A Cooperative Object Transport

Prior transport controllers use hierarchical optimization, finite-state behaviors or decentralized negotiation, often with predefined attachment points or geometric formations. Pattern-formation methods fill or outline target shapes; image-moment control includes the center of mass but does not necessarily keep every robot inside the footprint. The proposed reward explicitly adds load balance rather than prescribing target positions.

II-B Multi-Agent Reinforcement Learning

Multi-agent reinforcement learning must address changing teammates, credit assignment and scaling. Centralized training with decentralized execution gives actors their own observation vectors while a critic sees the joint observations. Earlier Pattern-RL and swarm-learning methods target geometric arrangements; this work adds object-load information and omits fixed robot goal poses.

Complete original PDF block S2.F2
Fig. 2: Centralized Training Decentralized Execution Paradigm

III METHODS

III-A Task Description

Circular mobile robots begin at random positions among static obstacles. The target is a suspended rigid object represented by a two-dimensional polygon; robots cannot collide with the object itself. Tests progress from uniform polygons to shifted centers of mass and footprints with cavities. Geometry and load information come from the simulator, rather than onboard estimation.

Complete original PDF block S3.F3
Fig. 3: Setting description. Information about teammates oiagentso_{i}^{\text{agents}} , obstacles oienvo_{i}^{\text{env}} and target object oiobjo_{i}^{\text{obj}} are green, gray, and orange lines, respectively. Current velocity viv_{i} is red arrow. Vectors are represented under robots’ local frames.
Original panel labels
(a) Local Observation
(b) Local Frame

III-B Problem Formulation

A partially observable Markov game defines joint transitions, individual rewards and local policy inputs. Each robot maximizes discounted reward under its stochastic policy. Equations 1–2 retain the full game specification and objective; the partial-observation formulation does not itself restrict communication range.

Equations 1–2
𝒢=⟨𝒩,𝒮,{𝒜i},P,{ℛi},{Ωi},O,γ⟩,\mathcal{G}=\langle\mathcal{N},\mathcal{S},\{\mathcal{A}_{i}\},P,\{\mathcal{R}_{i}\},\{\Omega_{i}\},O,\gamma\rangle, (1)
Ji​(π)=𝔼π​[∑t=0∞γt​ℛi​(st,at)],π=(πθ1,…,πθN).J_{i}(\pi)=\mathbb{E}_{\pi}\!\left[\sum_{t=0}^{\infty}\gamma^{t}\,\mathcal{R}_{i}(s^{t},a^{t})\right],\,\pi=(\pi_{\theta_{1}},\dots,\pi_{\theta_{N}}). (2)

III-C Agents, Observations and Actions

Unicycle robots output linear and angular velocities. Their inputs include position, velocity, estimated current load, every other robot’s relative position and load, twelve obstacle/robot distance rays, twelve target-boundary rays and a team load-similarity score. Figure 3 expresses vectors in each robot’s local coordinate frame.

Obstacle sensing is range-limited, but teammate information covers all other robots and the score aggregates team loads. Decentralized action selection therefore does not mean communication-free or strictly local-neighbor sensing. The observation dimension changes with team size.

Equations 3–9
x˙i=vicosψi,y˙i=visinψi,ψ˙i=ωi.\dot{x}_{i}=v_{i}\cos\psi_{i},\quad\dot{y}_{i}=v_{i}\sin\psi_{i},\quad\dot{\psi}_{i}=\omega_{i}. (3)
oi=[oiself,oirobots,oienv,oiobj,oiSl​o​a​d]o_{i}=[o_{i}^{\text{self}},o_{i}^{\text{robots}},o_{i}^{\text{env}},o_{i}^{\text{obj}},o_{i}^{S_{load}}] (4)
oiself={pi,vi,ℓi}o_{i}^{\text{self}}=\{\,p_{i},\;v_{i},\;\ell_{i}\,\} (5)
oirobots={Δ​pi​j,ℓj}j≠i,o_{i}^{\text{robots}}=\{\,\Delta p_{ij},\;\ell_{j}\,\}_{\,j\neq i}, (6)
oienv=Dioo_{i}^{\text{env}}=D_{i}^{o} (7)
oiobj=Di𝒫o_{i}^{\text{obj}}=D_{i}^{\mathcal{P}} (8)
dr𝒫=min∥rayr∩∂𝒫−pi∥2,r=1,…,nrd_{r}^{\mathcal{P}}=\min\bigl\|\,\text{ray}_{r}\cap\partial\mathcal{P}-p_{i}\,\bigr\|_{2},\qquad r=1,\dots,n_{r} (9)

III-D Load Computation and Sharing

Only robot centers inside the polygon participate. The load model enforces total vertical force and zero moments about the object’s center of mass, solving the linear system by least squares or a pseudoinverse. It normalizes the resulting loads and pads outside robots with zeros. A scaled cosine similarity to uniform loads supplies the team score; α=2 and exponent β=10 emphasize near-uniform configurations.

This is an unconstrained static surrogate. Negative loads are possible and would require pulling the object downward; nonnegative contact forces, actuator limits and exact equilibrium feasibility are not enforced by the stated solve. The simplification in Equations 16–18 assumes unit-sum normalized loads and suppresses the preceding numerical stabilizers. The score is not a physical support certificate.

Equations 10–18
pic=pi−C​o​M=(xic,yic),i∈ℐ.p_{i}^{c}=p_{i}-CoM=(x_{i}^{c},y_{i}^{c}),\quad i\in\mathcal{I}. (10)
∑i∈ℐℓi=w,∑i∈ℐxic​ℓi=0,∑i∈ℐyic​ℓi=0,\displaystyle\sum_{i\in\mathcal{I}}\ell_{i}=w,\quad\sum_{i\in\mathcal{I}}x_{i}^{c}\,\ell_{i}=0,\quad\sum_{i\in\mathcal{I}}y_{i}^{c}\,\ell_{i}=0, (11)
𝐀​𝐋ℐ=B\mathbf{A}\,\mathbf{L}_{\mathcal{I}}=B
[11⋯1xℐ1cxℐ2c⋯xℐkcyℐ1cyℐ2c⋯yℐkc]⏟𝐀⁡(3×k)​[ℓℐ1ℓℐ2ℓℐk]⏟𝐋ℐ​(k×1)=[w00]⏟𝐁⁡(3×1)\underbrace{\begin{bmatrix}1&1&\cdots&1\\ x_{\mathcal{I}_{1}}^{c}&x_{\mathcal{I}_{2}}^{c}&\cdots&x_{\mathcal{I}_{k}}^{c}\\ y_{\mathcal{I}_{1}}^{c}&y_{\mathcal{I}_{2}}^{c}&\cdots&y_{\mathcal{I}_{k}}^{c}\end{bmatrix}}_{\mathbf{A}\,(3\times k)}\underbrace{\begin{bmatrix}\ell_{\mathcal{I}_{1}}\\ \ell_{\mathcal{I}_{2}}\\ \vdots\\ \ell_{\mathcal{I}_{k}}\end{bmatrix}}_{\mathbf{L}_{\mathcal{I}}\,(k\times 1)}=\underbrace{\begin{bmatrix}w\\ 0\\ 0\end{bmatrix}}_{\mathbf{B}\,(3\times 1)} (12)
𝐋^ℐ=𝐋ℐ∑𝐋ℐ+ε.\mathbf{\hat{L}}_{\mathcal{I}}=\frac{\mathbf{L}_{\mathcal{I}}}{\sum\mathbf{L}_{\mathcal{I}}+\varepsilon}\,. (13)
𝐋^=[𝐋^ℐ𝟎n−k].\mathbf{\hat{L}}=\begin{bmatrix}\mathbf{\hat{L}}_{\mathcal{I}}\\ \mathbf{0}_{n-k}\end{bmatrix}. (14)
Sl​o​a​d\displaystyle S_{load} =α⋅(𝑳^T​𝐮‖𝑳^‖2⋅‖𝐮‖2+ε)β,\displaystyle=\alpha\cdot\left(\frac{\boldsymbol{\hat{L}}^{T}\mathbf{u}}{\|\boldsymbol{\hat{L}}\|_{2}\cdot\|\mathbf{u}\|_{2}+\varepsilon}\right)^{\beta}, (15)
∵𝑳^T​𝐮\displaystyle\because\boldsymbol{\hat{L}}^{T}\mathbf{u} =∑i1n​𝑳^i=1n,\displaystyle=\sum_{i}\frac{1}{n}\boldsymbol{\hat{L}}_{i}=\frac{1}{n},\quad (16)
&‖𝐮‖2\displaystyle\&\quad\|\mathbf{u}\|_{2} =n​(1n)2=1n,\displaystyle=\sqrt{n\left(\frac{1}{n}\right)^{2}}=\frac{1}{\sqrt{n}}, (17)
∴Sl​o​a​d\displaystyle\therefore S_{load} =α​(1n​‖𝑳^‖2)β.\displaystyle=\alpha\left(\frac{1}{\sqrt{n}\ \|\boldsymbol{\hat{L}}\|_{2}}\right)^{\beta}. (18)

III-E Reward Design

Rewards combine footprint entry/progress, collision penalty, load similarity and a bonus when all robots are inside and the score exceeds a team-size threshold. The stated intention is to attract robots toward the object while encouraging balanced support.

A source sign discrepancy is retained: Equation 20 uses αₚ(dₚᵗ−dₚᵗ⁻¹) outside the footprint, with positive αₚ=0.1. For the nonnegative distance defined in the text, that term rewards moving farther away, contrary to the prose. The displayed formula alone does not establish what the implementation executes.

Equations 19–21
Ri=Ripos+Ricollide+RiloadShare+Risuccess.R_{i}=R_{i}^{\text{pos}}+R_{i}^{\text{collide}}+R_{i}^{\text{loadShare}}+R_{i}^{\text{success}}. (19)
Rpos={Ri​nif ​p∈𝒫,αp⋅(d𝒫t−d𝒫t−1)otherwise.R^{\text{pos}}=\begin{cases}R_{in}&\text{if }p\in\mathcal{P},\\ \alpha_{p}\cdot(d_{\mathcal{P}}^{t}-d_{\mathcal{P}}^{t-1})&\text{otherwise}.\end{cases} (20)
Rsuccess={Rs∑in1pi∈𝒫=n&Sl​o​a​d≥τs,0otherwise.R^{\text{success}}=\begin{cases}{R_{s}}&\sum_{i}^{n}1_{p_{i}\in\mathcal{P}}=n\;\&\;S_{load}\geq\tau_{s},\\ 0&\text{otherwise}.\end{cases} (21)
Complete original PDF block S3.T1
TABLE I: Environment Parameters. Spatial quantities are expressed in simulator units.
Original selectable table
Parameter Value
Agents, Observations and Actions
rar_{a} (N∈{3,5})(N\in\{3,5\}) 0.05
rar_{a} (N∈{10,15,20})(N\in\{10,15,20\}) 0.02
Dm​a​xoD^{o}_{max} 0.35
nrn_{r} 12
Dm​a​x𝒫D^{\mathcal{P}}_{max} 1.4
ror_{o} 0.05
Load Computation and Sharing
α\alpha 2
β\beta 10
Reward Design
Ri​nR_{in} 0.1
αp\alpha_{p} 0.1
Rc​o​l​l​i​d​eR^{collide} -0.5
RsR_{s} 5
Similarity Score Threshold
τs3,τs5,τs10,τs15,τs20\tau_{s}^{3},\tau_{s}^{5},\tau_{s}^{10},\tau_{s}^{15},\tau_{s}^{20} [1.6049, 1.2999, 0.7954, 0.5086, 0.3374]
Obstacles
nOn_{O} 10
rOr_{O} 0.05
rOcr_{O}^{c} Uniform(0.06, 0.16)
wOcw_{O}^{c} Uniform(0.12, 0.35)
hOch_{O}^{c} Uniform(0.06, 0.20)
Objects
𝒫s\mathcal{P}_{s} 0.4
𝒫v\mathcal{P}_{v} Uniformint​(5,25)\text{Uniform}_{\text{int}}(5,25)
𝒫c\mathcal{P}_{c} 0.2
nc​a​vn_{cav} Uniformint​(1,2)\text{Uniform}_{\text{int}}(1,2)
kc​a​vk_{cav} Uniform​(0.1,0.25)\text{Uniform}(0.1,0.25)
Complete original PDF block S3.T2
TABLE II: MAPPO Hyperparameters
Original selectable table
Parameter Search Space Chosen Value
η\eta LogUniform(1​e−5, 3​e−41e-5,\,3e-4) 9​e−59e-5
ϵ\epsilon Uniform(0.05, 0.30.05,\,0.3) 0.20.2
βe​n​t\beta_{ent} LogUniform(1​e−4, 3​e−21e-4,\,3e-2) 4​e−34e-3
λ\lambda Uniform(0.85, 0.970.85,\,0.97) 0.9350.935
γ\gamma {0.95,0.97,0.98,0.99}\{0.95,0.97,0.98,0.99\} 0.980.98
KK {3,5,8}\{3,5,8\} 88
MM {1000,2000,3000,6000}\{1000,2000,3000,6000\} 10001000
‖∇‖max\|\nabla\|_{\max} {0.5,1.0}\{0.5,1.0\} 1.01.0
LπL_{\pi} {2,3}\{2,3\} 33
LVL_{V} {2,3}\{2,3\} 22
HπH_{\pi}, HVH_{V} {256,512,1024}\{256,512,1024\} 512512 (N∈{3,5,10}(N\in\{3,5,10\}),
10241024 (N∈{15,20})(N\in\{15,20\})

III-F Similarity Score Threshold Choice

The paper derives a similarity threshold from a desired maximum normalized-load spread d=0.15. Its intermediate case assumes one high load and all remaining loads equally low. The original endpoint cases and derivation are preserved below.

That threshold is not equivalent to bounding the largest load difference. For five robots, the derivation uses squared norm 0.218 and threshold about 1.2999. Loads [0.29, 0.11, 0.20, 0.20, 0.20] sum to one and pass the score test, although their spread is 0.18. Conversely, [0.29, 0.29, 0.14, 0.14, 0.14] have allowed spread 0.15 but lower similarity and fail. Thus the reported score criterion must be distinguished from a proven 0.15 spread guarantee; these counterexamples concern the formula, not reconstructed experimental trajectories.

Equations 22–26
d=max⁡(𝑳^)−min⁡(𝑳^),(0≤d≤2).d=\max(\boldsymbol{\hat{L}})-\min(\boldsymbol{\hat{L}}),\quad(0\leq d\leq 2). (22)
‖𝑳^‖2,m​a​x={N(N​mod​2=1),N−1(N​mod​2=0).\|\boldsymbol{\hat{L}}\|_{2,max}=\begin{cases}\sqrt{N}&(N~\text{mod}~2=1),\\ \sqrt{N-1}&(N~\text{mod}~2=0).\end{cases} (23)
a+(n−1)​(a−d)=1,∴a=1+d⁡(n−1)n.a+(n-1)(a-d)=1,\quad\therefore a=\frac{1+d(n-1)}{n}. (24)
‖𝑳^‖2,m​a​x=a2+(n−1)​(a−d)2\displaystyle\|\boldsymbol{\hat{L}}\|_{2,max}=\sqrt{a^{2}+(n-1)(a-d)^{2}} (25)
=(1+(n−1)​d)2+(n−1)​(1−d)2n2.\displaystyle=\sqrt{\frac{(1+(n-1)d)^{2}+(n-1)(1-d)^{2}}{n^{2}}}. (26)

III-G Policy Optimization

MAPPO extends PPO with a centralized value function. Homogeneous robots share a three-layer ReLU actor producing a diagonal Gaussian action, squashed through tanh to respect velocity bounds. A two-layer critic receives concatenated team observations and predicts a value for each robot; generalized advantage estimates drive policy updates. Sharing actor parameters is within a fixed-size team, not demonstrated transfer of one policy across sizes.

IV EXPERIMENTAL SETUP

IV-A Environment

VMAS supplies the two-dimensional dynamics. E1 uses random star-shaped polygons without obstacles; E2 adds circles; E3 uses denser circular/rectangular clutter. E4 shifts the center of mass and E5 adds cavities. Random radial perturbations control polygon concavity; the center-of-mass shift models nonuniform weight without simulating a lifted three-dimensional load.

Table I retains all scene and reward parameters. Robot radius changes from 0.05 for three/five robots to 0.02 for larger teams, so scaling comparisons also change physical packing difficulty. These tests sample the same families used in training, rather than unrestricted arbitrary objects.

Complete original PDF block S4.F4
Fig. 4: Environments E1 - E5 (purple circles: robots, red cross: CoM, orange lines: object footprint, black circles and rectangles: obstacles)
Original panel labels

(a) E1

(b) E2

(c) E3

(d) E4

(e) E5
Equation 27
ri=1+0.9​𝒫c​𝒰​(−1,1)r_{i}=1+0.9\ \mathcal{P}_{c}\ \mathcal{U}(-1,1) (27)

IV-B Hyperparameters

Separate policies are trained for 3, 5, 10, 15 and 20 robots. Training uses 150 parallel scenes, 125 updates and seed zero, with randomized clutter, shifted centers of mass and cavities appearing with probability 0.35. Two hundred steps per rollout gives 3.75 million environment steps; the larger teams use 300-step rollouts, so the same update count would imply 5.625 million steps rather than the stated common total.

A Ray Tune search tests 64 configurations per team size for fifty iterations, selecting by mean episode reward. Adam jointly updates actor/critic with per-robot advantage normalization and gradient clipping. The chosen widths are 512 for smaller teams and 1,024 for fifteen/twenty robots. Table II preserves the full search spaces and selected settings; multi-seed robustness is not reported.

IV-C Post-Evaluation

Each team/environment combination is evaluated in 150 runs capped at 400 steps. Reported success uses the footprint/score threshold; mean execution steps measures time to the successful formation. The paper describes this as equivalent to maximum load spread below 0.15, but the preceding threshold derivation does not justify that equivalence. No transported distance or carried-object collision metric is measured.

Baselines are random placements and DUD+EM: an attraction/repulsion field fills the shape, followed by centroid-based refinement. A separate comparison runs each method one hundred times on obstacle-free unions of two to five rectangles with shifted centers of mass. A mixed-integer linear program finds the closest equal-load, minimum-spacing formation using total Manhattan distance. This restricted benchmark measures formation correction cost, not a transport trajectory; objects are doubled in size for fifteen/twenty robots.

V RESULTS

The learning curves and all tabulated outcomes below are retained as reported. Interpretation depends on the fixed-size policies, simulator load surrogate and score-based success rule.

Complete original PDF block S5.F5
Fig. 5: Learning Curves

V-A Policy Optimization

Returns improve for all five team sizes. Three robots have less geometric redundancy for balancing moments; larger teams can redistribute the estimated load more easily. Fifteen/twenty robots also train with longer episodes, accumulating more shaping and success reward, so their raw returns are not directly comparable with smaller teams. The curves provide one-seed training evidence rather than confidence intervals over independent training runs.

V-B Post-Evaluation

Table III reports 92–100% success over E1–E4. Cavities reduce E5 success to 73.67%, 84.67%, 86.67%, 93.33% and 91.33% for increasing team sizes. Corresponding mean formation steps are 82.48, 105.10, 94.62, 74.27 and 77.31. These are formation outcomes under the reported threshold, not successful physical transports.

In the separate rectangle-union benchmark, learned formations have lower mean correction cost than both baselines for every size: 0.03/0.07/0.42/0.84/1.11 versus DUD+EM’s 0.25/0.40/0.75/2.48/3.23. Success is 90/83/93/95/97%, versus 14/49/79/91/96%. However, at twenty robots DUD+EM has slightly higher mean similarity, 1.30 versus 1.27; the proposed method does not lead every metric.

Figure 6 shows five robots, matching its caption despite the body’s “ten robots” wording. Several Table III percentages, including 99.8%, do not correspond to integer successes out of the stated 150 runs; the paper does not explain that aggregation. The figures and values remain unchanged. Lower correction costs support the sampled formations’ quality, without establishing an optimality guarantee for the policy.

Complete original PDF block S5.T3
TABLE III: Success Rate (SR) and Mean Execution Steps (MES) for different MRS sizes
Original selectable table
Environment No. of Robots
3 5 10 15 20
SR MES SR MES SR MES SR MES SR MES
E1 94.00% 62.6 100% 59.23 98.67% 62.69 99.33% 47.23 99.33% 49.54
E2 93.07% 62.8 96.67% 66.38 96.00% 63.62 99.8% 47.63 99.33% 50.09
E3 92.33% 66.17 98.68% 66.26 94.00% 63.43 96.67% 49.52 95.33% 58.03
E4 92.00% 85.78 98.67% 80.83 93.33% 83.11 94.67% 70.71 94.60% 70.37
E5 73.67% 82.48 84.67% 105.10 86.67% 94.62 93.33% 74.27 91.33% 77.31
Complete original PDF block S5.T4
TABLE IV: Comparison with baselines across different MRS sizes
Original selectable table
Metric Method No. of Robots
3 5 10 15 20
Avg. 𝙻𝟷\mathtt{L_{1}} Cost Random 0.56±\pm0.29 0.90±\pm0.45 1.49±\pm0.81 3.89±\pm1.73 4.53±\pm2.37
DuD+EM 0.25±\pm0.25 0.40±\pm0.23 0.75±\pm0.41 2.48±\pm1.11 3.23±\pm1.66
Ours 0.03±\pm0.03 0.07±\pm0.08 0.42±\pm0.29 0.84±\pm0.42 1.11±\pm0.78
Success Rate Random 5.00% 9.00% 33.00% 49.00% 77.00%
DuD+EM 14.00% 49.00% 79.00% 91.00% 96.00%
Ours 90.00% 83.00% 93.00% 95.00% 97.00%
Avg. Sl​o​a​dS_{load} Random 0.32±\pm0.55 0.39±\pm0.54 0.63±\pm0.62 0.72±\pm0.59 0.93±\pm0.58
DuD+EM 1.03±\pm0.54 1.19±\pm0.56 1.27±\pm0.51 1.23±\pm0.47 1.30±\pm0.46
Ours 1.84±\pm0.41 1.63±\pm0.63 1.55±\pm0.44 1.28±\pm0.44 1.27±\pm0.56
Complete original PDF block S5.F6
Fig. 6: Visualization of the transportation cost to the closest optimal formation for N=5N=5 robots carrying an object with non-uniform mass distribution.
Original panel labels
(a) Random
(b) DUD+EM
(c) Our method

VI CONCLUSION

Adding load-aware reward improves simulated support placement beyond geometric shape filling. The evidence is limited to two-dimensional formation, estimated unconstrained loads, one training seed and separately trained team sizes. Collision penalties and favorable success rates do not certify safe support or navigation.

The authors identify full transport after lifting, onboard geometry/mass estimation and graph-based policies for larger teams as future work. Those additions would be needed to test the method’s assumptions on physical robots and more scalable communication structures.

ACKNOWLEDGEMENTS

The authors acknowledge support from the German research ministry through Robotics Institute Germany and computing resources from NHR@FAU under BayernKI.

Concept demonstration

Open image ↗