Literature
Display
← All papers

VLAgents: A Policy Server for Efficient VLA Inference

Tobias Jülg · Khaled Gamal · Nisarga Nilavadi · Pierre Krack · Seongjin Bien · Michael Krawez · Florian Walter · Wolfram Burgard

Preprint posted · arXiv (preprint) ↗

Robotics2 min text · CondensedOriginal paper ↗
Contents

At a glance

VLAgents provides a common interface between robot environments and vision-language-action policies. It uses shared memory on one host and compressed image transport between hosts. Seven policies are integrated. The benchmark measures communication round-trip time with model inference skipped, so its reported rates describe transport capacity rather than complete robot-policy inference speed.

I Introduction

Model-specific interfaces and incompatible simulator/model dependencies make deployment and evaluation expensive to maintain. VLAgents separates environments from policies with a Gymnasium-style client/server interface [8]. The same wrapper can connect hardware or a simulator, and supporting tools run checkpoint evaluations on Slurm clusters.

Original figure
Fig. 1: The architecture of VLAgents. Each environment implements a thin wrapper that translates observations and actions into the expected data format (see Fig. 2). A central control loop takes the state and forwards it via either shared memory or a TCP connection with JPEG-encoding to the policy server. The policy server uses the interface from Fig. 2 to run an inference pass with the corresponding model. This yields an action, which is then returned and used by the environment loop to step the environment to obtain the next state.

II Related Work

The comparison covers OpenVLA’s HTTP server, OpenPi’s WebSocket server, and LeRobot’s asynchronous gRPC service. VLAgents adds typed observation/action fields, an explicit translation layer, and transport chosen for local or remote execution. These comparisons describe the software versions evaluated in the paper; they do not establish that other systems cannot add equivalent capabilities.

III Methodology

Policies expose initialization, reset, and action-generation methods. Observations and actions have dedicated fields plus an extensible information dictionary. RPyC exposes the interface remotely. A connection-aware client uses shared memory locally and JPEG-compressed RGB imagery over TCP remotely. Optional environment loops, recording, and Slurm utilities support automated evaluations. Figure 2 retains the complete interface listing.

class Obs:
    cameras: dict[str, np.ndarray] = {}
    gripper: float | None = None
    info: dict[str, Any] = {}

class Act:
    action: np.ndarray
    done: bool = False
    info: dict[str, Any] = {}

class Agent:
    def initialize(self):
        """heavy intitilzation e.g. model loading"""

    def act(self, obs: Obs) -> Act:
        """forward pass"""

    def reset(self, obs: Obs, instruction: Any, **kwargs) -> dict[str, Any]:
        """reset state, e.g. history"""
Fig. 2: VLA policy interface.

IV Results and Conclusion

The implementation integrates seven policies and supports ManiSkill 3 [16] and the Robot Control Stack [4]. With two 224-by-224 RGB camera inputs, the measured mean communication round trip is 0.3 ms locally and 4.4 ms across a 1-Gbit local Ethernet network. The latter corresponds to roughly 220 requests per second only when model execution is excluded. The measurements support lower serialization and transport overhead in this setup; they do not measure full policy latency, wide-area-network behavior, task success, or the effect of JPEG compression on policy accuracy. All original timing results and error bars remain in Figure 3.

Original figure
Fig. 3: Mean Round-Trip Time (RTT) for different policy servers with two 224×224224\times 224 RGB cameras. Localhost indicates that the client and server are running on the same machine, while network indicates execution across different machines. For the network setting, the machines were connected in a Local Area Network with a 1 Gbit Ethernet connection.
Concept demonstration

Open image ↗