At a glance
VLAgents provides a common interface between robot environments and vision-language-action policies. It uses shared memory on one host and compressed image transport between hosts. Seven policies are integrated. The benchmark measures communication round-trip time with model inference skipped, so its reported rates describe transport capacity rather than complete robot-policy inference speed.
I Introduction
Model-specific interfaces and incompatible simulator/model dependencies make deployment and evaluation expensive to maintain. VLAgents separates environments from policies with a Gymnasium-style client/server interface [8]. The same wrapper can connect hardware or a simulator, and supporting tools run checkpoint evaluations on Slurm clusters.
II Related Work
The comparison covers OpenVLA’s HTTP server, OpenPi’s WebSocket server, and LeRobot’s asynchronous gRPC service. VLAgents adds typed observation/action fields, an explicit translation layer, and transport chosen for local or remote execution. These comparisons describe the software versions evaluated in the paper; they do not establish that other systems cannot add equivalent capabilities.
III Methodology
Policies expose initialization, reset, and action-generation methods. Observations and actions have dedicated fields plus an extensible information dictionary. RPyC exposes the interface remotely. A connection-aware client uses shared memory locally and JPEG-compressed RGB imagery over TCP remotely. Optional environment loops, recording, and Slurm utilities support automated evaluations. Figure 2 retains the complete interface listing.
class Obs:
cameras: dict[str, np.ndarray] = {}
gripper: float | None = None
info: dict[str, Any] = {}
class Act:
action: np.ndarray
done: bool = False
info: dict[str, Any] = {}
class Agent:
def initialize(self):
"""heavy intitilzation e.g. model loading"""
def act(self, obs: Obs) -> Act:
"""forward pass"""
def reset(self, obs: Obs, instruction: Any, **kwargs) -> dict[str, Any]:
"""reset state, e.g. history"""
IV Results and Conclusion
The implementation integrates seven policies and supports ManiSkill 3 [16] and the Robot Control Stack [4]. With two 224-by-224 RGB camera inputs, the measured mean communication round trip is 0.3 ms locally and 4.4 ms across a 1-Gbit local Ethernet network. The latter corresponds to roughly 220 requests per second only when model execution is excluded. The measurements support lower serialization and transport overhead in this setup; they do not measure full policy latency, wide-area-network behavior, task success, or the effect of JPEG compression on policy accuracy. All original timing results and error bars remain in Figure 3.