Overview
TARC learns when to request the next policy decision as well as what action to execute, allocating more frequent decisions to demanding states.
At a glance
A reinforcement-learning policy jointly selects an action and its holding duration. A fixed query penalty encourages sparse decisions; an adaptive multiplier targets an expected discounted query budget. Experiments on a radio-controlled car, a Go1 quadruped, and simulated vision-language-action tasks reduce policy calls while preserving task performance in the tested settings. Reported inference and command-travel reductions are deployment proxies, not measured energy or wear savings. Held actions remain open-loop until the next scheduled decision.
1 Introduction
Fixed control rates must accommodate difficult moments even when most states need fewer corrections. TARC makes the decision interval state-dependent, combining task return with a query-count constraint. Its contributions are the duration-aware objective, dynamic hardware transfer, and an extension to selecting how much of a generative action chunk to execute before replanning.
2 Related Work
Variable action repetition and MOSEAC [32] already trade interaction cost against performance. TARC distinguishes a fixed soft penalty from a learned multiplier targeting the rate itself. It also adapts generative policy chunk execution through diffusion steering [31]. Unlike classical event-triggered control, it learns timing from reward; it does not inherit a hand-designed stability trigger or guarantee.
3 Method
The method begins with a query-constrained control objective and expresses it as rewards at variable-length decision epochs.
3.1 Problem statement
A binary indicator marks whether the policy is queried at each underlying environment step. The objective maximizes discounted task return subject to an expected discounted query count bounded by K/(1−γ). K=1 permits querying every step; lower values require skipped decisions. This normalization assumes a discount below one, and the budget is an expectation rather than a strict cap on every rollout.
Equation 1
| (1) |
3.2 Time-Adaptive Policies
The policy chooses both a control command and an integer repetition duration, with task rewards accumulated across the held interval.
Policy Class.
The input augments state with episode time. Each action is held for one to i steps, giving a decision rate between the maximum rate divided by i and the maximum rate. Increasing i allows larger computational savings but also longer periods without a new observation-conditioned policy decision.
Equations 2–3
| (2) |
| (3) |
Lagrangian relaxation.
Grouping the discounted return by decision epochs yields a per-decision reward equal to the within-hold discounted task rewards minus one query cost. The budget contributes a constant independent of the policy for a fixed multiplier. Consecutive decisions must be discounted by γ raised to their actual duration, as detailed in the appendix; using ordinary per-decision discounting would change the objective.
Equations 4–5
| (4) |
| (5) |
3.3 Soft and Hard Query Budget Constraints
The same objective supports either a manually selected query cost or an automatically adjusted budget penalty.
Soft constraint (fixed c ).
A fixed cost encourages fewer calls but does not enforce a particular realized rate. Its effect depends on task dynamics and reward scale, so it remains a tuning parameter. This is related to time-adaptive continuous-time reinforcement learning [29], adapted here to discrete control and chunk execution.
Hard constraint (auto-tuned λ ).
The nonnegative multiplier increases when estimated discounted query use exceeds the target and decreases otherwise. Updating it alongside the policy empirically tracks chosen budgets. The source calls this the hard-constraint variant, but finite-sample training and an expected constraint do not provide a guaranteed per-episode query cap or safety guarantee.
Equation 6
| (6) |
4 Experimental Setup
The car performs a roughly two-meter reverse-parking maneuver with a 180-degree turn, usually involving a drift, at a maximum 30 Hz decision rate. State comes from OptiTrack, and randomized tire/motor dynamics support transfer. Episodes last about 6.6 seconds. The Go1 uses a maximum 50 Hz high-level policy and an unchanged lower-level joint controller; three twenty-second command scenarios test steady turning, changing speed, and abrupt run-to-turn transitions.
The VLA experiment uses a frozen LIBERO-fine-tuned π₀ model [4], a learned lightweight steering policy, and variable execution lengths up to fifty actions. Its main simulated task compares fixed chunks of twenty and thirty-two actions, with ten rollouts per checkpoint. In contrast to holding one motor command on the hardware tasks, the VLA executes a prefix of an action sequence. Most results use five training seeds. Unpenalized task reward and reward after a common query penalty are reported separately.
Equation 7
| (7) |
5 Results
Evaluation separates budget tracking, matched-rate comparisons, generative-policy inference, and hardware deployment. Statistical analysis treats the training seed as the random unit and uses bootstrap/permutation checks. Claimed task-reward equivalence uses two one-sided tests with a prespecified margin of 5% of baseline reward, rather than treating a nonsignificant difference as proof of equality.
5.1 Simulation Results
Simulation tests the expected query-budget constraint and whether adaptive timing offers benefits beyond simply lowering a fixed rate.
Automatic query budget enforcement.
For Go1 query ratios of 0.3, 0.4, 0.6, and 0.7, realized discounted rates track the targets with reported mean deviations below 5%. Tight budgets reduce task reward; at ratios of at least 0.6 it returns within the high-rate baseline’s confidence interval. This demonstrates a controllable trade-off, not cost-free query reduction at every budget.
Adaptive vs. fixed low-frequency control.
TARC meets the specified equivalence criterion against 50 Hz PPO while using fewer than half as many decisions. Against a fixed 20 Hz controller, its mean reward difference is +1.93 with confidence interval −0.8 to +5.4, so a mean improvement is not established. The stronger finding is consistency: per-seed rewards span 29.7–30.6 for TARC versus 21.5–31.7 for fixed 20 Hz, with a reported 12.6-fold standard-deviation ratio.
Vision-language-action model.
On the main LIBERO task, average executed chunk length is 32.1±7.8, versus twenty for the baseline, reducing transformer calls by about 38% while retaining success. TARC also exceeds a fixed thirty-two-action comparison. This supports adaptive replanning within the evaluated simulated task and post-training method, not a measured 38% reduction in total robot execution time.
5.2 Hardware Results
Simulation-trained car and quadruped policies are deployed without hardware fine-tuning, with task reward, decision frequency, and control-command variation measured.
RC Car.
Five car rollouts per policy show similar unpenalized reward and lower variance for TARC. Total absolute throttle-command variation falls by 21%. That is a measured command-travel proxy; mechanical wear and energy consumption are not directly measured in these short experiments.
Unitree Go1.
Across three Go1 scenarios, task reward improves by 2.58 with confidence interval 0.5–5.2. The policy uses 365±2 queries per one thousand environment steps, about 64% fewer than the 50 Hz baseline. Identical per-query inference graphs make call count a useful inference-load proxy. A perturbation example switches from 16.7 to 50 Hz after detecting the disturbance, but a disturbance arriving mid-hold must wait for the next scheduled decision.
6 Conclusion
Duration selection can reduce high-level policy inference while retaining performance across the tested tasks. The expected-budget formulation and duration-aware return provide a principled training interface; matched-frequency comparisons show the clearest adaptive benefit on the quadruped and simulated VLA tasks. Car results also show that some tasks can be handled just as well at a suitably chosen fixed low rate.
7 Limitations
One global duration applies across all controlled degrees of freedom, which may be restrictive for subsystems with different dynamics. Short hardware episodes do not measure energy savings. Safety is only implicit in task reward, and held actions cannot react until the next decision, bounded by the maximum hold duration. Explicit safety constraints and per-subsystem timing remain future work.
Acknowledgments
The work acknowledges Swiss National Science Foundation support through NCCR Automation.
Appendix A Proof of Proposition 1
The proof groups ordinary environment timesteps into variable-length decision epochs and preserves the discounted objective exactly for the stated policy class.
Setup.
A policy call occurs at the start of each epoch. Its command persists for the selected duration, and the next epoch starts after precisely that many environment steps. The query indicator is zero between epochs.
Decomposing the task reward.
The task-return sum is regrouped into within-epoch discounted rewards multiplied by the discount at each epoch’s absolute start time. Equation 8 retains the complete decomposition.
Equation 8
| (8) |
Decomposing the constraint.
Because each epoch contains exactly one call, the discounted query count becomes a sum of discounts at epoch start times, as shown in Equation 9.
Equation 9
| (9) |
Combining into the Lagrangian.
Combining both sums produces the per-decision task reward minus the multiplier, plus the policy-independent budget constant. Equation 10 preserves the algebra. The resulting control process uses duration-dependent transitions and discounting.
Equation 10
| (10) |
Remark (training).
Training uses γ raised to the held duration in both temporal-difference residuals and generalized-advantage recursion. This semi-Markov adjustment is essential when comparing commands of different lengths.
Appendix B Experimental Details and Hyperparameters
The implementation appendix records observations, actions, dynamics randomization, rewards, network architectures, and complete training settings for both hardware platforms.
B.1 RC Car
The car setup combines delayed state/action information with randomized drifting dynamics and a position-orientation parking objective.
B.1.1 State Space
Six physical state values cover position, heading, local velocity, and angular velocity. Three recent two-dimensional actions compensate for approximately 80 ms transmission delay. The fixed baseline therefore sees twelve values; TARC adds episode time as a thirteenth.
| Dimension(s) | Description |
|---|---|
| 0, 1 | Global position () |
| 2 | Orientation () |
| 3, 4 | Local velocity () |
| 5 | Angular velocity () |
| 6, 7 | Action at () |
| 8, 9 | Action at () |
| 10, 11 | Action at () |
| 12 | Episode timestep (TARC only) |
B.1.2 Action Space
The two continuous controls are steering and throttle, normalized to −1 through 1 and rescaled to physical limits. TARC adds a duration choice to this control interface.
B.1.3 Reward Function
A smooth tolerance function combines positional and angular parking errors, peaks at the goal, and decays with distance. The main reward subtracts a small action-magnitude penalty. Equation 11 retains the exact tolerance function, avoiding conflation of task shaping with query cost.
Equation 11
| (11) |
B.1.4 Simulation Dynamics
A Pacejka tire model is randomized over mass, inertia, front/rear grip, motor constant, and steering limit. Table 2 preserves the ranges that support the claimed transfer.
| Parameter | Range |
|---|---|
| Mass () | [1.6, 1.7] kg |
| Inertia () | [0.03, 0.18] |
| Front tire grip () | [0.25, 0.6] |
| Rear tire grip () | [0.15, 0.45] |
| Motor constant () | [10.0, 40.0] |
| Steering limit | [0.4, 0.75] rad |
B.1.5 Hyperparameters
Car training uses seventy-five million timesteps, 2,048 parallel environments, and five seeds. The maximum rate is 30 Hz with soft query cost 0.1; repetition limits four, five, and ten allow minima of 7.5, 6, and 3 Hz. Table 3 retains PPO settings and the five-layer policy/four-layer critic sizes.
| Parameter | Value |
|---|---|
| Environment | |
| Episode steps | 200 |
| Parallel environments | 2048 |
| Total timesteps | |
| Random seeds | 5 |
| TARC-specific | |
| 30 Hz | |
| Switch cost | 0.1 |
| variants | Hz |
| PPO | |
| Learning rate | 3e-4 |
| Batch size | 1024 |
| Minibatches | 32 |
| Updates per batch | 4 |
| Unroll length | 10 |
| Entropy cost | 0.01 |
| Discount | 0.9 |
| GAE | 0.95 |
| Clip | 0.3 |
| Optimizer | Adam |
| Network architecture | |
| Policy hidden layers | |
| Critic hidden layers | |
| Activation | Swish |
B.2 Unitree Go1
The Go1 implementation keeps the underlying joint-tracking controller and task reward fixed while changing high-level action timing.
B.2.1 Environment and Policy Architecture
MuJoCo Playground randomizes sensing and physical properties. A twelve-action policy produces joint targets tracked by an unchanged proportional–derivative controller at the underlying rate. An asymmetric critic receives privileged state during training; hardware inference uses ONNX on the onboard CPU. Thus policy-query reduction does not mean all low-level feedback or actuator updates stop.
B.2.2 Go1 Evaluation Scenarios
The three held-out command schedules probe smooth low-speed turning, varying straight-line speed, and running followed by in-place turning. Their twenty-second durations compare different responsiveness demands rather than a long-horizon energy or reliability benchmark.
B.2.3 Hyperparameters
Go1 training uses six hundred million timesteps, 8,192 parallel environments, five seeds, and a 50 Hz maximum policy rate. The soft query cost is 0.005; both actor and critic use widths 512, 256, and 128. Table 4 retains the full optimization configuration.
| Parameter | Value |
|---|---|
| Environment | |
| Episode steps | 1000 |
| Parallel environments | 8192 |
| Total timesteps | |
| Random seeds | 5 |
| TARC-specific | |
| 50 Hz | |
| Switch cost (soft constraint) | 0.005 |
| variants | Hz |
| PPO | |
| Learning rate | 3e-4 |
| Batch size | 256 |
| Minibatches | 32 |
| Updates per batch | 4 |
| Unroll length | 20 |
| Entropy cost | 0.01 |
| Discount | 0.97 |
| GAE | 0.95 |
| Clip | 0.2 |
| Optimizer | Adam |
| Network architecture | |
| Policy hidden layers | |
| Critic hidden layers | |
| Activation | Swish |
Appendix C Supplementary Results
The supplementary ablations vary maximum hold counts of four, five, and ten, examining both frequency savings and the consequences of longer open-loop intervals.
C.1 Simulation Results
Simulation ablations separate fixed query-cost sensitivity, adaptive-versus-fixed scheduling, and task-dependent VLA chunk lengths.
Switch Cost Sensitivity
A fixed cost changes the reward trade-off but does not directly specify the realized rate. The cost sweeps test whether practical operation follows the intended direction without destroying task performance.
RC Car.
For the car, each repetition-limit variant settles near a task-specific rate—approximately 10, 7.5, or 4 Hz—across the tested cost sweep. Reward remains high. Here, dynamics determine a sufficient rate more strongly than the chosen soft penalty.
Go1.
For Go1, very small penalties give little frequency reduction, intermediate penalties lower the rate while retaining reward, and costs of at least 0.05 cause reward collapse. The chosen 0.005 cost sits in the useful region; it is a platform-tuned setting rather than an automatically enforced budget.
Adaptive vs. Fixed Low-Frequency Control
Matched-frequency comparisons ask whether state-dependent allocation improves performance beyond reducing the average number of policy calls.
Go1.
All three Go1 repetition variants retain high-rate-baseline task reward at lower average rates, and their plotted means exceed the fixed 20 Hz controller. The main-text confidence interval remains the appropriate qualification on the individual mean-reward comparison, while the reduced seed variability is more clearly established.
RC Car.
For the car, TARC-4 and TARC-5 perform comparably to fixed 10 and 7.5 Hz controllers. The task admits a suitable constant low rate, so adaptive timing is not necessary for a reward advantage in this comparison.
Multi-Task VLA Generalization
Two additional LIBERO tasks bring the evaluation to three. Success remains about 96–100% relative to fixed twenty-action chunks, while average selected lengths differ by task. This supports task-dependent scheduling in the evaluated setup; it does not by itself establish transfer of one timing policy to arbitrary unseen tasks.
C.2 Hardware Results
Hardware ablations check whether conclusions persist across repetition limits and whether simulated timing survives deployment.
Go1 Hardware Results
All three Go1 variants perform consistently across the tested command scenarios, matching or exceeding the high-rate baseline’s task reward. Figure 14 retains the per-scenario reward and frequency comparisons with five-seed uncertainty.
Sim-to-Real Transfer on the RC Car
Car query rates transfer closely, but TARC-10 shows a larger task-reward gap between simulation and hardware. A ten-step hold lasts up to about 330 ms, allowing model errors and disturbances to accumulate before correction. The authors suggest limits no greater than five for this fast platform; that is an empirical choice for these experiments rather than a universal bound.