Literature
Display
← All papers

Robot Control Stack: A Lean Ecosystem for Robot Learning at Scale

Tobias Jülg · Pierre Krack · Seongjin Bien · Yannik Blei · Khaled Gamal · Ken Nakahara · Johannes Hechtl · Roberto Calandra · Wolfram Burgard · Florian Walter

· ICRA 2026 ↗

Robotics9 min text · CondensedOriginal paper ↗
Contents

Overview

RCS connects hardware, simulation and learning applications through composable interfaces. The original feature matrix below records the authors’ comparison of related software at publication time; detailed performance and picking-task evidence follow.

TABLE I: Feature matrix of software packages for robot learning most closely related to RCS
Robot Support API Scalability & Deployment Robot Learning Features
Name Hardware Simulation Gymnasium Low Level Distributed Parallel pip Online Offline (Data Collection)
Isaac Lab [1] ROS2 Isaac Sim ✓ ✗ ✓ ✓ ✓ ✓ ✓
frankx [2] Panda ✗ ✗ ✗ ✗ ✗ ✓ ✗ ✗
LeRobot [3] Universal MuJoCo ✓ ✗ ✓ ✓ ✓ ✓ ✓
Deoxys [4] Panda robosuite ✗ ✗ ✓ ✗ ✗ ✗ ✓
Polymetis [5] Panda Multiple ✗ ✗ ✓ ✗ ✓ ✗ ✗
Ark [6] Universal Multiple ✓ C++ ✓ ✗ ✓ ✗ ✓
RCS (ours) Universal MuJoCo ✓ C++ ✓ ✓ ✓ ✓ ✓

At a glance

Robot Control Stack (RCS) combines a low-level C++ robot interface, composable Gymnasium wrappers, MuJoCo simulation and Python applications for data collection, control and policy evaluation. VLAgents separates policy dependencies from robot software through remote procedure calls.

The authors demonstrate four physical setups, a matched Franka simulation, VLA fine-tuning and an RL training pipeline. The experiments support practical integration across these setups, while the picking-task results also expose model and simulation limitations. Code, datasets and videos accompany the paper.

I Introduction

Large generalist policies place the robot inside a data/training/inference workflow, whereas conventional robotics stacks often treat learned components as modules inside a fixed system. Simulation libraries scale training but may leave hardware integration and calibration to separate code.

RCS uses a shared robot interface and wrapper composition to reuse collection and evaluation applications across real and simulated environments. Its contribution is software integration plus concrete use cases, not a new VLA architecture or proof that all robotics middleware is unnecessary.

Complete original Figure 1.
Fig. 1: The layered architecture of RCS: Applications at the top layer access robots, sensors, and actuators through a Gymnasium-based Python API for easy switching between hardware and a MuJoCo simulation. The lower layers expose a C++ API for performance-critical features, making RCS equally suitable for end-to-end policy learning and low-level control.

II Related Work

The comparison spans general middleware such as ROS and YARP, specialized Panda libraries, Ark, Isaac Lab and LeRobot. RCS emphasizes low-level access, synchronous execution, optional distribution, parallel simulation and collection/training tools in one interface.

Table I records the authors’ feature comparison at the paper’s release. It is a historical snapshot, not a current audit of rapidly changing packages. The paper does not supply controlled developer-effort or end-to-end latency comparisons establishing a universal usability or overhead advantage.

Complete original Figure 2.
Fig. 2: The architecture of RCS. The applications on the left side interface with the environment on the right side, which can be a simulation or a real robot, through the Gymnasium interface. Sensors, actuators, and data observers wrap the environment by mutating the action, and/or the observation space.

III Methodology

A wrapper W=⟨f:S→S′,g:A′→A,P′,R′⟩W=\langle f:S\to S^{\prime},g:A^{\prime}\to A,P^{\prime},R^{\prime}\rangle maps observations outward through f and incoming actions inward through g, optionally changing transition or reward functions. It transforms M=⟨S,A,P,R⟩M=\langle S,A,P,R\rangle into M′=⟨S′,A′,P′,R′⟩M^{\prime}=\langle S^{\prime},A^{\prime},P^{\prime},R^{\prime}\rangle, written M′=W⊳MM^{\prime}=W\rhd M.

Identity mappings retain state/action values: St′=St=fI​(St)S^{\prime}_{t}=S_{t}=f_{I}(S_{t}) and At′=At=gI​(At)A_{t}^{\prime}=A_{t}=g_{I}(A_{t}). Such wrappers can record or transport data without changing the control semantics. The same directional convention extends to partially observable environments.

III-A Library Architecture

A C++ interface with Python bindings exposes abstract robot control, including Cartesian/joint and synchronous/asynchronous modes. The base environment can target hardware or MuJoCo. Wrappers add components such as a gripper dimension, camera images, transformed actions or recording.

An agent supplies At=At(n)A_{t}=A_{t}^{(n)} to M(n)=W(n)⊳⋯⊳W(1)⊳MM^{(n)}=W^{(n)}\rhd\dots\rhd W^{(1)}\rhd M. Actions traverse the wrappers inward as At(0)=g(1)​(…​g(n)​(At(n)))A_{t}^{(0)}=g^{(1)}(\dots g^{(n)}(A_{t}^{(n)})); returned observations traverse outward as St(n)=f(n)​(…​f(1)​(St(0)))S_{t}^{(n)}=f^{(n)}(\dots f^{(1)}(S_{t}^{(0)})). The original formulas retain the order that makes composition work.

III-A 1 Hardware Abstraction

Common sensor/actuator interfaces provide reusable camera polling and end-effector wrappers. New devices implement these interfaces or a custom wrapper. By default, a step waits until the requested action state is reached; asynchronous operation can return immediately for uses such as teleoperation.

The step interface coordinates sensor observations, but high recording frequency does not imply equally frequent fresh camera exposures. The performance experiment explicitly reuses the most recent frame above the cameras’ 30 fps rate.

III-A 2 Simulation

RCS exposes MuJoCo data structures while adding an object-oriented scene interface. Step callbacks wait for relevant entities to converge, or interrupt when conditions such as collision occur. This lets synchronous gripper/arm operations use the same high-level semantics as hardware.

III-A 3 Robotics Tool Kit

Pinocchio supplies kinematics and IK from the MuJoCo MJCF robot description; OMPL supplies basic motion planning and scripted data-collection diversity. Supported robots ship with base simulation scenes.

A digital twin can execute the same commands in parallel for prototyping or checking proposed actions. A camera-calibration interface accepts setup-specific strategies such as AprilTags. A simulation check is a useful tool, not a validated hardware-safety guarantee under model mismatch.

III-B Applications and Ecosystem

Applications compose wrappers for teleoperation, trajectory generation, Parquet recording or learned-policy execution. VLAgents installs a lightweight server in a policy’s own Python environment and a client alongside RCS, separating incompatible dependency stacks.

The connection uses TCP between machines and shared memory on one machine. Observations arrive grouped by timestep; the package reports integration with seven policies and three simulation environments. Distribution is optional rather than built into every component.

IV Results

Results follow the workflow from hardware/simulation setup through data collection, policy evaluation and reinforcement learning.

Complete original Figure 3.
Fig. 3: We deployed RCS in four different setups. From top left to bottom right: FR3 with Franka Hand gripper, wrist and side cameras; xArm7 with Tilburg Hand and one side camera; UR5e with Robotiq gripper, wrist and side cameras; SO101 with wrist and side cameras.
TABLE II: Components implemented in RCS. Bold font indicates availability in hardware and simulation.
Robots End effectors Sensors User Input
- FR3 - Franka Hand - D400s - HTC Vive
- xArm7 - Tilburg Hand - Webcams - Meta Quest 3
- UR5e - Robotiq 2F85 - DIGIT [26] - SpaceMouse
- SO101 - SO101 Gripper - Tacto [27] - Keyboard
Complete original Figure 4.
Fig. 4: Configured control vs. measured data frequency during teleoperation, averaged over more than 1000 steps. The shaded area denotes the standard deviation. FR3 2 Cams: Two RealSense cameras. FR3 4+2 Cams: Four RealSense cameras and two DIGIT sensors. Dual FR3 4+2 Cams: Like FR3 4+2 Cam but with two FR3 robots.

IV-A Deployed Setups

Four combinations of robots, end effectors and cameras operate across three institutions: Franka FR3, xArm7, UR5e and SO101. The FR3 picking setup is also reconstructed in MuJoCo. Table II preserves the supported component list and bold marks for hardware/simulation availability.

IV-A 1 Hardware

FR3 uses a seven-joint arm, Franka Hand, wrist D405 and side D435 cameras plus fixed lighting. xArm7 pairs a seven-joint arm with a sixteen-joint Tilburg Hand and one D435i side camera.

UR5e has six arm joints, a Robotiq 2F-85 gripper and wrist/overhead D405 cameras. SO101 has five arm joints, a one-joint gripper, an InnoMaker wrist webcam and a D435 side view. Different optics, workspace and end effectors make their policy results an integration demonstration rather than a controlled embodiment ranking.

IV-A 2 Simulation

The FR3 simulation matches the real table CAD, arm placement, background, texture and color. Calibrated camera poses, intrinsics and resolutions are reproduced. This deliberate visual matching supports the transfer experiment; arbitrary uncalibrated simulation is not tested.

IV-B System Performance

Teleoperation tests compare configured frequency with recorded data frequency over more than one thousand steps, using one/two FR3 arms, up to four 1280×720 RealSense cameras and two 320×240 DIGIT sensors. All cameras run at 30 fps, so faster recording uses their latest frames while proprioception can update faster.

All configurations sustain about 90 Hz and the two-camera FR3 about 120 Hz; additional I/O causes higher-frequency jitter. These measurements show collection/control-loop throughput under these configurations, not fresh multi-camera capture at 120 Hz or learned-policy inference speed.

Complete original Figure 5.
Fig. 5: Replicated simulation scene. Left: The simulation scene of FR3, with calibrated camera poses visible in light green. Right: Images from the wrist and side cameras in real setup (above) vs. in simulation (below).

IV-C Task and Dataset

Pick-Cuboid asks the robot to grasp a randomly placed, oriented green printed cuboid. The task is intentionally simple and reproducible; xArm7 substitutes a grasp-friendly toy. Seeded placement controls simulation, while physical placement is manual or robot-mediated.

All demonstrations record at 30 Hz to Parquet: joint angles, Cartesian end-effector pose in a unified robot-base convention, gripper state and RGB/depth images, plus calibration matrices when available. Each embodiment’s collection details matter for interpreting the comparisons.

FR3

The FR3 dataset has 143 HTC Vive teleoperated demonstrations in a 30×40 cm workspace using operational-space control. The robot places the cuboid itself, reducing manual-placement bias and recording its pose for simulation replay.

xArm7

xArm7 supplies one hundred Meta Quest teleoperated demonstrations in a 60×30 cm workspace. A deformable toy replaces the undersized cuboid; it is manually placed, grasped and lifted to the home pose. This task does not require the same angle-specific grasp as Pick-Cuboid.

UR5e

UR5e provides 167 SpaceMouse demonstrations using Cartesian velocity control in a 50×50 cm workspace. The operator grasps a manually placed cuboid, lifts it and returns home while holding it.

SO101

SO101 uses a smaller cuboid and 35×20 cm workspace. Its leader–follower teleoperation setup produces 120 demonstrations.

Scripted Simulated FR3

A scripted FR3 simulator linearly interpolates toward the known cuboid pose and checks success using object height. Parallel generation produces roughly one thousand episodes per twenty minutes on a consumer GPU laptop. Of three thousand generated episodes, 2,193 succeed (about 73%) and enter the filtered dataset.

Replaying recorded real robot states in the matched simulation produces only 73 successful episodes from 143 real demonstrations. Contact/model mismatch therefore remains substantial despite visual matching. The known object poses and calibrated cameras enable this replay.

IV-D VLA Experiments

VLA experiments use the shared RCS applications to compare setup-specific fine-tuning, low-frequency next-step prediction, checkpoint transfer and synthetic-data mixtures.

IV-D 1 π 0 on Different Setups

Each robot receives its own independently fine-tuned π0 weights. Evaluation runs at 30 Hz with twenty-step action chunks and nominally fifty physical rollouts per model. Figure 6 reports success rates 0.96 FR3, 0.84 xArm7, 0.79 UR5e and 0.62 SO101.

The UR5e value 0.79 cannot be an exact fraction of fifty binary trials; the paper does not reconcile that denominator or rounding detail. These are not results from one jointly trained cross-embodiment policy.

FR3

FR3 has the highest reported π0 success. The authors attribute some of its advantage to Franka data in pretraining, but do not isolate that explanation experimentally.

xArm7

The xArm7 result shows that the software/policy workflow supports a multifingered hand after fine-tuning. The larger deformable toy and lack of a prescribed grasp angle also simplify the task; the score alone does not measure embodiment generalization.

UR5e

UR5e/Robotiq hardware also appears in π0 pretraining. A larger workspace, nearby singularities and manual-placement bias are proposed explanations for its lower result than FR3; these remain hypotheses.

SO101

SO101 performs worse. Suggested causes are its smaller, lower-DoF embodiment, collisions on the fixed-finger side, deflection and motor backlash. No ablation separates their contributions.

Complete original Figure 6.
Fig. 6: VLA success rates in different settings. Top left: π0\pi_{0} fine-tuned on four datasets from different setups. Top right: Octo, OpenVLA and π0\pi_{0} fine-tuned on our FR3 setup (real) with a down-sampled frequency of 5 Hz5\text{\,}\mathrm{H}\mathrm{z} and evaluated on the real-world setup and the replicated simulated scene. Bottom: Different data mixes of synthetic and real data evaluated on the real-world setup and the simulated scene. The number in the experiment name denotes how many episodes from the respective domain were used in the training mix.

IV-D 2 Different VLAs on Next Step Prediction

The FR3 comparison fine-tunes Octo, OpenVLA and π0 on demonstrations downsampled to 5 Hz, then uses synchronous one-step prediction rather than action chunks. This matches OpenVLA’s capability but differs from the 30 Hz chunked π0 experiment.

Across thirty physical and one hundred simulated rollouts, Figure 6 reports real/sim success 0.03/0.00 for Octo, 0.20/0.00 for OpenVLA and 0.93/0.37 for π0. Models see the same cuboid placements. Pretraining distribution and low-frequency action semantics may affect performance; these values are not a universal ranking across tasks.

IV-D 3 π 0 Evaluation in Simulation

Following the motivation of SIMPLER, each π0 training checkpoint is tested on twenty physical and one hundred simulated rollouts. For real-only training, simulation success lies below physical success and loosely tracks it.

That observed relationship is specific to this calibrated picking setup; it is not a guaranteed lower bound for arbitrary policies, checkpoints or tasks. Figure 7 also retains the loss curve rather than equating training loss with execution success.

IV-D 4 π 0 with Synthetic Data Mixes

With ten real demonstrations, π0 reaches 0.16 real/0.08 simulated success. One hundred synthetic demonstrations yield 0.22/0.93. Combining both yields 0.41/0.89: real success rises, but simulated success drops slightly from synthetic-only training.

A larger mix of 143 real and five hundred synthetic episodes reaches observed 100% physical success by ten thousand training steps in Figure 7. Each checkpoint has only twenty real trials, and later checkpoints are not uniformly perfect. Added synthetic data improves this task’s transfer without establishing universal gains.

The authors explain lower simulation success by contact mechanics requiring grasps nearer the cuboid center. The modest ten-real-only result is evidence of weak few-shot performance here, not absence of all real-to-sim capability.

Complete original Figure 7.
Fig. 7: Evaluation success rates of all training checkpoints. Every checkpoint is evaluated on 20 real and 100 simulated rollouts. Left: Trained on 143 episodes on the FR3 dataset. Right: Trained on a mix of 143 episodes from the FR3 dataset and 500 episodes from the scripted dataset of the replicated simulated domain.

IV-E RL Training Pipeline

RCS runs parallel Gymnasium environments with Stable Baselines3. A one-camera low-resolution configuration exceeds two thousand steps/s over twenty-four environments; rendering contributes much of the runtime and the result depends on scene and algorithm configuration.

For learning Pick-Cuboid, vanilla PPO uses two RGB views and proprioception, predicts relative joint angles and receives ManiSkill3-style potential shaping plus a discrete success reward. Object distances and grasp state come from the simulator. The authors report 100% success within three hours and 8.5 million steps on an RTX 4080/12-core CPU, without hyperparameter tuning.

The success-evaluation trial count and repeated-seed variation are not supplied. The one-camera throughput measurement and two-camera learning run are distinct configurations, so their timing should not be conflated.

V Conclusion and Future Work

RCS demonstrates reuse of control, collection and evaluation code across several hardware setups and a matched simulation, with enough throughput for the tested VLA/RL workflows. Synthetic data improves physical π0 picking performance in the reported mixtures.

The paper supplies case studies rather than a controlled usability study or broad generalization benchmark. Planned additions include ROS integration and bimanual/mobile manipulation; their mention is future work, not validation in these experiments. Support includes Genius-Robot 01IS24083, RIG 16ME1006, BMFTR and NHR@FAU/BayernKI v106be, with code contributions acknowledged.

Concept demonstration

Open image ↗