Overview
RCS connects hardware, simulation and learning applications through composable interfaces. The original feature matrix below records the authors’ comparison of related software at publication time; detailed performance and picking-task evidence follow.
| Robot Support | API | Scalability & Deployment | Robot Learning Features | ||||||
| Name | Hardware | Simulation | Gymnasium | Low Level | Distributed | Parallel | pip | Online | Offline (Data Collection) |
| Isaac Lab [1] | ROS2 | Isaac Sim | ✓ | ✗ | ✓ | ✓ | ✓ | ✓ | ✓ |
| frankx [2] | Panda | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | ✗ | ✗ |
| LeRobot [3] | Universal | MuJoCo | ✓ | ✗ | ✓ | ✓ | ✓ | ✓ | ✓ |
| Deoxys [4] | Panda | robosuite | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ | ✓ |
| Polymetis [5] | Panda | Multiple | ✗ | ✗ | ✓ | ✗ | ✓ | ✗ | ✗ |
| Ark [6] | Universal | Multiple | ✓ | C++ | ✓ | ✗ | ✓ | ✗ | ✓ |
| RCS (ours) | Universal | MuJoCo | ✓ | C++ | ✓ | ✓ | ✓ | ✓ | ✓ |
At a glance
Robot Control Stack (RCS) combines a low-level C++ robot interface, composable Gymnasium wrappers, MuJoCo simulation and Python applications for data collection, control and policy evaluation. VLAgents separates policy dependencies from robot software through remote procedure calls.
The authors demonstrate four physical setups, a matched Franka simulation, VLA fine-tuning and an RL training pipeline. The experiments support practical integration across these setups, while the picking-task results also expose model and simulation limitations. Code, datasets and videos accompany the paper.
I Introduction
Large generalist policies place the robot inside a data/training/inference workflow, whereas conventional robotics stacks often treat learned components as modules inside a fixed system. Simulation libraries scale training but may leave hardware integration and calibration to separate code.
RCS uses a shared robot interface and wrapper composition to reuse collection and evaluation applications across real and simulated environments. Its contribution is software integration plus concrete use cases, not a new VLA architecture or proof that all robotics middleware is unnecessary.
II Related Work
The comparison spans general middleware such as ROS and YARP, specialized Panda libraries, Ark, Isaac Lab and LeRobot. RCS emphasizes low-level access, synchronous execution, optional distribution, parallel simulation and collection/training tools in one interface.
Table I records the authors’ feature comparison at the paper’s release. It is a historical snapshot, not a current audit of rapidly changing packages. The paper does not supply controlled developer-effort or end-to-end latency comparisons establishing a universal usability or overhead advantage.
III Methodology
A wrapper maps observations outward through f and incoming actions inward through g, optionally changing transition or reward functions. It transforms into , written .
Identity mappings retain state/action values: and . Such wrappers can record or transport data without changing the control semantics. The same directional convention extends to partially observable environments.
III-A Library Architecture
A C++ interface with Python bindings exposes abstract robot control, including Cartesian/joint and synchronous/asynchronous modes. The base environment can target hardware or MuJoCo. Wrappers add components such as a gripper dimension, camera images, transformed actions or recording.
An agent supplies to . Actions traverse the wrappers inward as ; returned observations traverse outward as . The original formulas retain the order that makes composition work.
III-A 1 Hardware Abstraction
Common sensor/actuator interfaces provide reusable camera polling and end-effector wrappers. New devices implement these interfaces or a custom wrapper. By default, a step waits until the requested action state is reached; asynchronous operation can return immediately for uses such as teleoperation.
The step interface coordinates sensor observations, but high recording frequency does not imply equally frequent fresh camera exposures. The performance experiment explicitly reuses the most recent frame above the cameras’ 30 fps rate.
III-A 2 Simulation
RCS exposes MuJoCo data structures while adding an object-oriented scene interface. Step callbacks wait for relevant entities to converge, or interrupt when conditions such as collision occur. This lets synchronous gripper/arm operations use the same high-level semantics as hardware.
III-A 3 Robotics Tool Kit
Pinocchio supplies kinematics and IK from the MuJoCo MJCF robot description; OMPL supplies basic motion planning and scripted data-collection diversity. Supported robots ship with base simulation scenes.
A digital twin can execute the same commands in parallel for prototyping or checking proposed actions. A camera-calibration interface accepts setup-specific strategies such as AprilTags. A simulation check is a useful tool, not a validated hardware-safety guarantee under model mismatch.
III-B Applications and Ecosystem
Applications compose wrappers for teleoperation, trajectory generation, Parquet recording or learned-policy execution. VLAgents installs a lightweight server in a policy’s own Python environment and a client alongside RCS, separating incompatible dependency stacks.
The connection uses TCP between machines and shared memory on one machine. Observations arrive grouped by timestep; the package reports integration with seven policies and three simulation environments. Distribution is optional rather than built into every component.
IV Results
Results follow the workflow from hardware/simulation setup through data collection, policy evaluation and reinforcement learning.
| Robots | End effectors | Sensors | User Input |
|---|---|---|---|
| - FR3 | - Franka Hand | - D400s | - HTC Vive |
| - xArm7 | - Tilburg Hand | - Webcams | - Meta Quest 3 |
| - UR5e | - Robotiq 2F85 | - DIGIT [26] | - SpaceMouse |
| - SO101 | - SO101 Gripper | - Tacto [27] | - Keyboard |
IV-A Deployed Setups
Four combinations of robots, end effectors and cameras operate across three institutions: Franka FR3, xArm7, UR5e and SO101. The FR3 picking setup is also reconstructed in MuJoCo. Table II preserves the supported component list and bold marks for hardware/simulation availability.
IV-A 1 Hardware
FR3 uses a seven-joint arm, Franka Hand, wrist D405 and side D435 cameras plus fixed lighting. xArm7 pairs a seven-joint arm with a sixteen-joint Tilburg Hand and one D435i side camera.
UR5e has six arm joints, a Robotiq 2F-85 gripper and wrist/overhead D405 cameras. SO101 has five arm joints, a one-joint gripper, an InnoMaker wrist webcam and a D435 side view. Different optics, workspace and end effectors make their policy results an integration demonstration rather than a controlled embodiment ranking.
IV-A 2 Simulation
The FR3 simulation matches the real table CAD, arm placement, background, texture and color. Calibrated camera poses, intrinsics and resolutions are reproduced. This deliberate visual matching supports the transfer experiment; arbitrary uncalibrated simulation is not tested.
IV-B System Performance
Teleoperation tests compare configured frequency with recorded data frequency over more than one thousand steps, using one/two FR3 arms, up to four 1280×720 RealSense cameras and two 320×240 DIGIT sensors. All cameras run at 30 fps, so faster recording uses their latest frames while proprioception can update faster.
All configurations sustain about 90 Hz and the two-camera FR3 about 120 Hz; additional I/O causes higher-frequency jitter. These measurements show collection/control-loop throughput under these configurations, not fresh multi-camera capture at 120 Hz or learned-policy inference speed.
IV-C Task and Dataset
Pick-Cuboid asks the robot to grasp a randomly placed, oriented green printed cuboid. The task is intentionally simple and reproducible; xArm7 substitutes a grasp-friendly toy. Seeded placement controls simulation, while physical placement is manual or robot-mediated.
All demonstrations record at 30 Hz to Parquet: joint angles, Cartesian end-effector pose in a unified robot-base convention, gripper state and RGB/depth images, plus calibration matrices when available. Each embodiment’s collection details matter for interpreting the comparisons.
FR3
The FR3 dataset has 143 HTC Vive teleoperated demonstrations in a 30×40 cm workspace using operational-space control. The robot places the cuboid itself, reducing manual-placement bias and recording its pose for simulation replay.
xArm7
xArm7 supplies one hundred Meta Quest teleoperated demonstrations in a 60×30 cm workspace. A deformable toy replaces the undersized cuboid; it is manually placed, grasped and lifted to the home pose. This task does not require the same angle-specific grasp as Pick-Cuboid.
UR5e
UR5e provides 167 SpaceMouse demonstrations using Cartesian velocity control in a 50×50 cm workspace. The operator grasps a manually placed cuboid, lifts it and returns home while holding it.
SO101
SO101 uses a smaller cuboid and 35×20 cm workspace. Its leader–follower teleoperation setup produces 120 demonstrations.
Scripted Simulated FR3
A scripted FR3 simulator linearly interpolates toward the known cuboid pose and checks success using object height. Parallel generation produces roughly one thousand episodes per twenty minutes on a consumer GPU laptop. Of three thousand generated episodes, 2,193 succeed (about 73%) and enter the filtered dataset.
Replaying recorded real robot states in the matched simulation produces only 73 successful episodes from 143 real demonstrations. Contact/model mismatch therefore remains substantial despite visual matching. The known object poses and calibrated cameras enable this replay.
IV-D VLA Experiments
VLA experiments use the shared RCS applications to compare setup-specific fine-tuning, low-frequency next-step prediction, checkpoint transfer and synthetic-data mixtures.
IV-D 1 π 0 on Different Setups
Each robot receives its own independently fine-tuned π0 weights. Evaluation runs at 30 Hz with twenty-step action chunks and nominally fifty physical rollouts per model. Figure 6 reports success rates 0.96 FR3, 0.84 xArm7, 0.79 UR5e and 0.62 SO101.
The UR5e value 0.79 cannot be an exact fraction of fifty binary trials; the paper does not reconcile that denominator or rounding detail. These are not results from one jointly trained cross-embodiment policy.
FR3
FR3 has the highest reported π0 success. The authors attribute some of its advantage to Franka data in pretraining, but do not isolate that explanation experimentally.
xArm7
The xArm7 result shows that the software/policy workflow supports a multifingered hand after fine-tuning. The larger deformable toy and lack of a prescribed grasp angle also simplify the task; the score alone does not measure embodiment generalization.
UR5e
UR5e/Robotiq hardware also appears in π0 pretraining. A larger workspace, nearby singularities and manual-placement bias are proposed explanations for its lower result than FR3; these remain hypotheses.
SO101
SO101 performs worse. Suggested causes are its smaller, lower-DoF embodiment, collisions on the fixed-finger side, deflection and motor backlash. No ablation separates their contributions.
IV-D 2 Different VLAs on Next Step Prediction
The FR3 comparison fine-tunes Octo, OpenVLA and π0 on demonstrations downsampled to 5 Hz, then uses synchronous one-step prediction rather than action chunks. This matches OpenVLA’s capability but differs from the 30 Hz chunked π0 experiment.
Across thirty physical and one hundred simulated rollouts, Figure 6 reports real/sim success 0.03/0.00 for Octo, 0.20/0.00 for OpenVLA and 0.93/0.37 for π0. Models see the same cuboid placements. Pretraining distribution and low-frequency action semantics may affect performance; these values are not a universal ranking across tasks.
IV-D 3 π 0 Evaluation in Simulation
Following the motivation of SIMPLER, each π0 training checkpoint is tested on twenty physical and one hundred simulated rollouts. For real-only training, simulation success lies below physical success and loosely tracks it.
That observed relationship is specific to this calibrated picking setup; it is not a guaranteed lower bound for arbitrary policies, checkpoints or tasks. Figure 7 also retains the loss curve rather than equating training loss with execution success.
IV-D 4 π 0 with Synthetic Data Mixes
With ten real demonstrations, π0 reaches 0.16 real/0.08 simulated success. One hundred synthetic demonstrations yield 0.22/0.93. Combining both yields 0.41/0.89: real success rises, but simulated success drops slightly from synthetic-only training.
A larger mix of 143 real and five hundred synthetic episodes reaches observed 100% physical success by ten thousand training steps in Figure 7. Each checkpoint has only twenty real trials, and later checkpoints are not uniformly perfect. Added synthetic data improves this task’s transfer without establishing universal gains.
The authors explain lower simulation success by contact mechanics requiring grasps nearer the cuboid center. The modest ten-real-only result is evidence of weak few-shot performance here, not absence of all real-to-sim capability.
IV-E RL Training Pipeline
RCS runs parallel Gymnasium environments with Stable Baselines3. A one-camera low-resolution configuration exceeds two thousand steps/s over twenty-four environments; rendering contributes much of the runtime and the result depends on scene and algorithm configuration.
For learning Pick-Cuboid, vanilla PPO uses two RGB views and proprioception, predicts relative joint angles and receives ManiSkill3-style potential shaping plus a discrete success reward. Object distances and grasp state come from the simulator. The authors report 100% success within three hours and 8.5 million steps on an RTX 4080/12-core CPU, without hyperparameter tuning.
The success-evaluation trial count and repeated-seed variation are not supplied. The one-camera throughput measurement and two-camera learning run are distinct configurations, so their timing should not be conflated.
V Conclusion and Future Work
RCS demonstrates reuse of control, collection and evaluation code across several hardware setups and a matched simulation, with enough throughput for the tested VLA/RL workflows. Synthetic data improves physical π0 picking performance in the reported mixtures.
The paper supplies case studies rather than a controlled usability study or broad generalization benchmark. Planned additions include ROS integration and bimanual/mobile manipulation; their mention is future work, not validation in these experiments. Support includes Genius-Robot 01IS24083, RIG 16ME1006, BMFTR and NHR@FAU/BayernKI v106be, with code contributions acknowledged.