Literature
Display
← All papers

Temporal and Object Quantification Networks

Jiayuan Mao · Zhezheng Luo · Chuang Gan · Joshua B. Tenenbaum · Jiajun Wu · Leslie Pack Kaelbling · Tomer D. Ullman

· IJCAI 2021 ↗

Reasoning9 min text · CondensedOriginal paper ↗
Contents

At a glance

TOQ-Nets classify events using learned relational and temporal features with explicit pooling over objects and time. Shared operations let the same network accept different object counts and sequence lengths. Experiments test few-example action learning, transfer to different player counts, and changes in event timing. The model’s logical bias improves several generalization settings, but it represents a bounded subset of temporal logic and relies on structured input features.

1 Introduction

Events such as a high soccer pass depend on relations among agents and on their ordering over time. A relational-temporal description can remain valid as the number of players or detailed timing changes. TOQ-Nets encode that idea in differentiable tensor operations: learned feature combinations express predicates, while pooling approximates finite-domain quantifiers. High-level action labels train the classifier; experiments use simulated soccer, human-activity datasets, and robot manipulation trajectories.

Original figure
Figure 1: (a) An input sequence composed of relational states: each column represents the state of an entity that changes over time. A logic formula describes a complex concept or feature that is true of this temporal sequence using object and temporal quantification. The sequence is segmented into three stages: throughout the first stage, red holds for at least one entity, until the second stage, in which each entity is always either green or blue, until the third stage, in which yellow eventually becomes true for at least one of the entities. (b) Such events can be described using first-order linear temporal logic expressions.

2 TOQ-Nets

A task-specific feature extractor produces global, object, and pairwise features at each time. Relational layers combine them and pool across objects; temporal layers then summarize how the resulting events evolve. A final classifier uses the initial-time summary of the complete sequence. Quantification is over the finite objects and frames supplied to the network, and network depth bounds expression complexity. This architecture is a structural bias rather than a complete first-order temporal-logic engine.

Refer to caption
Figure 2: A TOQ-Net contains three modules: (i) an input feature extractor, (ii) relational reasoning layers, and (iii) temporal reasoning layers. To illustrate the model’s representational power, we show that logical forms of increasing complexity can be realized by stacking multiple layers.

2.1 Temporal-Relational Feature Representation

Nullary features describe the whole scene; unary features describe individual entities; binary features describe ordered entity pairs. Their per-frame shapes are D₀, N×D₁, and N×N×D₂. Shared transformations and reductions preserve compatibility with changing N and sequence length. Soccer inputs include entity positions and ball, target-player, and team indicators.

Original figure
Figure 3: Illustration of (i) relational reasoning layers and (ii) temporal reasoning layers. We provide two illustrative running traces. (i) The first relational reasoning layer takes unary predicates q1q_{1} and q2q_{2} as input and its output Q1Q_{1} is able to represent q1∧q2q_{1}\wedge q_{2}. The max⁡(Q1,d​i​m=0)\max(Q_{1},dim=0) in layer 2 can represent ∃x.q1​(x,t)∧q2​(x,t)\exists x.~q_{1}(x,t)\wedge q_{2}(x,t). (ii) Assume PKP_{K} encodes the occurance of events e1e_{1} and e2e_{2} at each time step. The first temporal reasoning layer can realize always e2e_{2} with a temporal pooling from time step 33 to time step TT. In the second temporal reasoning layer, the temporal pooling summarizes that e1e_{1} holds true from time step 1 to 2. Thus, the NN should be able to realize e1e_{1} until (always e2e_{2}).

2.2 Relational Reasoning Layers

Following Dong et al. (2019)’s Neural Logic Machines, each relational layer combines existing features with expanded lower-arity features and max-reduced higher-arity features. Fully connected sigmoid transformations learn the feature combinations. Max reduction behaves like soft existential quantification; learned negation can express universal conditions through the same operations. Parameters are shared across entities, pairs, and frames. The experiments restrict relations to arity at most two.

Equation
Pl\displaystyle P_{l} =\displaystyle= NNP​(Concat⁡[Pl−1;max⁡(Ql−1,dim=0)]),\displaystyle\mathrm{NN}_{P}\left(\mathrm{Concat}\left[P_{l-1};\max(Q_{l-1},\mathrm{dim}=0)\right]\right),
Ql\displaystyle Q_{l} =\displaystyle= NNQ(Concat[Ql−1;max(Rl−1,dim=0);\displaystyle\mathrm{NN}_{Q}\left(\mathrm{Concat}\left[Q_{l-1};\max(R_{l-1},\mathrm{dim}=0);\right.\right.
OPENOPENmax⁡(Rl−1,dim=1);expand⁡(Pl−1,dim=1)]),\displaystyle\left.\left.\max(R_{l-1},\mathrm{dim}=1);\mathrm{expand}(P_{l-1},\mathrm{dim}=1)\right]\right),
Rl\displaystyle R_{l} =\displaystyle= NNR(Concat[Rl−1;expand(Ql−1,dim=0);\displaystyle\mathrm{NN}_{R}\left(\mathrm{Concat}\left[R_{l-1};\mathrm{expand}(Q_{l-1},\mathrm{dim}=0);\right.\right.
expand(Ql−1,dim=1)]).\displaystyle\left.\left.\mathrm{expand}(Q_{l-1},\mathrm{dim}=1)\right]\right).

2.3 Temporal Reasoning Layers

Temporal layers pool over future intervals and candidate transition times. Learned transformations combine interval summaries with events at an endpoint, enabling bounded compositions resembling always, eventually, and until. Residual connections retain earlier events. The first temporal layer is a simpler suffix-pooling operation; the worked table and algorithm show how a later layer can express p until always q. In the tested ordering, object quantification occurs inside temporal quantification, so it cannot directly express every formula that tracks the same existentially chosen object across all times. Interleaving layer types is proposed as an extension. The selected temporal construction uses quadratic space in sequence length.

Equations 1
PK+l(t)=maxt′>t⁡NNl​(Concat⁡[PK+l−1(t′);maxt≤t′′<t′⁡PK+l−1(t′′)]).P_{K+l}^{(t)}=\max_{t^{\prime}>t}\mathrm{NN}_{l}\left(\mathrm{Concat}\left[P_{K+l-1}^{(t^{\prime})};\max_{t\leq t^{\prime\prime}<t^{\prime}}P_{K+l-1}^{(t^{\prime\prime})}\right]\right). (1)
PK+1(t)=NN1​(maxt≤t′′<T⁡PK(t′′)),\displaystyle P_{K+1}^{(t)}=\mathrm{NN}_{1}\left(\max_{t\leq t^{\prime\prime}<T}P_{K}^{(t^{\prime\prime})}\right),
Input 1-st Layer 2-nd Layer
tt p⁡(t)p(t) q⁡(t)q(t) Gpp Fpp Gqq Fqq pp U (Gqq)
1 T T F T F T T
2 T F F T F T T
3 F T F F T T F
4 F T F F T T F
Table 1: A running example of different temporal quantification formulas that TOQ-Nets can realize. For clarity, we use T and F for True/False. In the actual computation, they are “soft” Boolean values ranges in [0,1][0,1]. G means always; F means Eventually; U means until.
Algorithm 1 An example temporal structure that the second temporal reasoning layer can recognize.
1: p⁡(t)p(t), q⁡(t)q(t), (G​p)​(t)(\textit{{G}}p)(t), (F​p)​(t)(\textit{{F}}p)(t), (G​q)​(t)(\textit{{G}}q)(t), and (F​q)​(t)(\textit{{F}}q)(t)
2: OPEN(p​U​(G​q)))​(t)\left(p\textit{{U}}(\textit{{G}}q))\right)(t), which is true if pp holds true from time step tt until qq becomes always true.
3: for t←1t\leftarrow 1 to TT do
4:   for t′←t+1t^{\prime}\leftarrow t+1 to TT do
5:    if ∀t′′∈[t,t′).p⁡(t′′)\forall t^{\prime\prime}\in[t,t^{\prime}).~p(t^{\prime\prime}) and (G​q)​(t′)(\textit{{G}}q)(t^{\prime}) then
6:      OPEN(p​U​(G​q)))​(t)←𝑡𝑟𝑢𝑒\left(p\textit{{U}}(\textit{{G}}q))\right)(t)\leftarrow\mathit{true}
7:    end if
8:   end for
9: end for
Equations
p1Up2Up3U⋯Upk,p_{1}~\textit{{U}}~p_{2}~\textit{{U}}~p_{3}~\textit{{U}}~\cdots~\textit{{U}}~p_{k},
Fp1XFp2XFp3XF⋯XFpk,\textit{{F}}p_{1}~\textit{{XF}}~p_{2}~\textit{{XF}}~p_{3}~\textit{{XF}}~\cdots~\textit{{XF}}~p_{k},

3 Experiments

The experiments emphasize macro accuracy, scarce examples for held-out action categories, and generalization under changed object counts and timing. Standard in-distribution accuracy and robustness are reported separately.

3.1 Baseline Approaches

Baselines include STGCN (Yan et al., 2018), a max-pooling variant, a graph-plus-LSTM variant, Space-Time Region Graphs, and Non-Local Networks. Their relational and temporal operations differ. Appendix comparisons vary both capacity and input features, helping separate the proposed inductive bias from those confounds.

3.2 Trajectory-Based Soccer Event Detection

Model Reg. Few-Shot Full
STGCN 73.2±1.6 26.0±5.7 62.8±0.6
STGCN-MAX 73.6±1.5 28.6±5.0 63.6±0.7
STGCN-LSTM 72.7±1.4 23.8±5.9 61.9±0.6
Space-Time 74.8±1.5 31.7±6.1 65.2±0.6
Non-Local 76.5±2.4 45.0±6.3 69.5±2.4
TOQ-Net (ours) 87.7±1.3 52.2±6.3 79.8±0.8

Table 2: Results on the soccer event dataset. Different columns correspond to different action sets (the regular, few-shot, and full action sets). The performance is measured by per-action (macro) accuracy, averaged over nine few-shot splits. The ±{\pm} values indicate standard errors. TOQ-Net significantly outperforms all baseline methods on the few-shot action set.

Dataset and setup.

The soccer dataset uses simulator-generated trajectories and nine action categories after excluding absent or uninformative actions. Most categories have 5,000 examples, while sliding has 4,000. Each split exposes all training examples for seven regular actions and only 50 clips for each of two few-shot actions. The setup contains an internal length inconsistency: it first mentions 25-frame context, then describes ordinary 17-frame clips at 8 fps with the target action at frame 9. The later timing-shift test explicitly uses 25-frame clips.

Model 3v3 4v4 6v6 6v6 (Time Warp) 8v8 11v11
STGCN
40.7±1.0  (-40.4%)
63.2±4.9  (-7.4%)
68.2±2.8
52.8±7.0  (-22.6%)
55.4±3.3  (-18.8%)
44.4±2.1  (-34.9%)
STGCN-MAX
47.4±3.2  (-33.7%)
68.8±2.0  (-3.8%)
71.5±1.9
56.5±4.5  (-21.0%)
59.1±0.7  (-17.3%)
45.6±2.5  (-36.2%)
STGCN-LSTM
39.7±1.1  (-43.1%)
60.4±0.2  (-13.5%)
69.8±0.1
30.6±0.6  (-56.1%)
55.8±2.0  (-20.0%)
44.1±0.7  (-36.8%)
Space-Time
29.0±1.6  (-60.4%)
53.5±3.2  (-27.0%)
73.3±0.3
70.7±0.3  (-3.5%)
33.9±2.8  (-53.7%)
15.2±1.8  (-79.3%)
Non-Local
45.9±5.1  (-41.2%)
70.7±5.3  (-9.5%)
78.1±5.8
77.7±5.0  (-0.5%)
58.5±10.8  (-25.1%)
41.8±13.6  (-46.5%)
TOQ-Net
77.4±3.5  (-12.4%)
88.3±0.7  (-0.0%)
88.4±0.6
86.9±0.4  (-1.7%)
81.3±1.7  (-8.0%)
77.1±1.7  (-12.8%)
Table 3: Results on generalization to scenarios with more agents and temporally warped trajectories on the soccer event dataset. All models are trained only on 6v6 games. The standard errors indicated by the ±\pm signs are computed with three random seeds.

Input features.

Raw entity inputs contain three-dimensional positions and four ball/player/team indicators. A Gaussian temporal indicator marks the frame of interest. The soccer feature extractor additionally computes velocities, acceleration, and pairwise distance, then learns threshold-like features; the model therefore learns predicates over supplied physical feature templates.

Results.

Across the reported nine few-shot splits, TOQ-Net achieves the strongest macro accuracy on the regular, few-shot, and combined action sets. The few-shot result supports reuse of learned relational-temporal features for categories with limited labeled clips; it is supervised low-data classification, not recognition of wholly untrained labels.

Generalization to more players.

After training on six-player teams, evaluation changes the teams to three, four, eight, or eleven players. TOQ-Net degrades less than the compared models, but still confuses unsuccessful shots with short passes when an opponent gains the ball. Shot recall falls from 97% to 60% in the described larger-team analysis. This exposes a limitation of outcome-based physical cues without a representation of player intent.

Generalization to temporally warped trajectories.

Timing tests place the sole target action between frames 6 and 19 in a 25-frame clip. Temporal convolution and LSTM baselines lose substantial accuracy; TOQ-Net retains much more. Space-Time and Non-Local models change little under this shift partly because they are insensitive to temporal ordering, so invariance alone should not be read as evidence of detailed temporal reasoning.

3.3 Extension to Real-World Datasets

The real-world experiments use structured skeleton or annotated player inputs rather than raw-video recognition.

3.3.1 Toyota Smarthome

Toyota Smarthome provides about 16,100 videos in 19 classes. Thirty sampled frames supply joint positions, velocities, limb lengths, and angles. Standard accuracy is similar: TOQ-Net 42.0%, STGCN 43.0%. Under doubled-speed sequences, TOQ-Net retains 41.2% while STGCN falls to 32.3%, illustrating robustness without an in-distribution accuracy advantage.

3.3.2 Volleyball Activity

The volleyball experiment uses 4,830 clips, eight group-activity classes, and manual player boxes, involvement indicators, and individual-action annotations as inputs. Standard accuracy is 73.3% for TOQ-Net and 73.6% for STGCN. Under temporal shifts and doubled speed, TOQ-Net scores 70.3% and 70.7%, versus 39.5% and 48.6%. These results depend on the substantial structured annotations supplied to the classifiers.

4 Related Work

Action concept representations and learning.

Symbolic activity models can express temporal relations but often require annotated primitives, rules, or combinatorial search. TOQ-Nets replace explicit rule search with gradient-trained feature combinations and quantification operations.

Temporal and relational reasoning.

Related approaches include temporal-logic learners, logical neural networks, hidden-state models, graph networks, and recurrent models. TOQ-Nets share structured relational inputs with several baselines but use a different temporal aggregation mechanism.

5 Conclusion and Discussion

Pooling-based object and time quantification can improve generalization across entity counts and temporal variation. Remaining limits include restricted logical expressiveness, dependence on task-specific physical or annotated features, quadratic temporal computation, and ambiguity about goals or intentions. Broader transfer between object kinds and integration of mental-state variables remain future work.

Acknowledgements

The authors thank Tomás Lozano-Pérez and acknowledge the listed research sponsors.

Supplementary Material for Temporal and Object Quantification Networks

The supplement adds robot-manipulation evaluation, complete model and training specifications, capacity and input-feature comparisons, scale-transfer experiments, and interpretability examples.

Appendix A Manipulation Concept Learning from 6D Poses

The manipulation setting tests whether temporal-relational representations transfer beyond soccer into object interactions by a robot arm.

Dataset and setup.

RLBench generates 100 trajectories for each of 24 manipulation actions, with randomized object poses and built-in motion planning. Lengths range from 30 to 1,000 frames, usually near 50. The stated split is 60% training, 15% validation, and 25% test. The subsequent input paragraph says regular actions average 65 training trajectories; this differs from the stated 60-example training split.

Input.

Objects are represented by position, quaternion orientation, and axis-aligned box coordinates; the robot gripper includes pose and an open/closed indicator. Articulated or deformable items are split into meaningful pieces, such as a separate box lid. These structured state inputs carry information unavailable from raw images alone.

Input features.

TOQ-Net adds a learned sigmoid projection of the raw state vectors. The recurrent baseline uses temporal kernel size one in its graph front end, isolating the LSTM’s contribution; the supplement also compares kernel size three.

Original figure
Model Reg. 1-Shot Full
STGCN 99.89±0.05 94.92±1.03 98.79±0.23
STGCN-LSTM 99.92±0.03 95.48±1.67 98.86±0.43
TOQ-Net 99.96±0.02 98.04±0.97 99.48±0.24
Table 1: Few-shot learning on the RLBench dataset, measured by per-action (macro) accuracy and averaged of four 1-shot splits and four random seeds per split. The ±{\pm} values indicate standard errors. On the right we shows the sampled performance of different models on each individual 1-shot split.

l

Refer to caption
Figure 1: Comparing different models with different time stretching factors on the RLBench dataset.

Results.

The 24-way experiment stretches trajectories from one-eighth to eight times their original duration. The retained plot shows TOQ-Net’s strong performance across these changes compared with convolutional and recurrent baselines.

Generalization to new action concepts.

A one-shot protocol withholds half the opening and closing action types, providing one example of each held-out type. Classification groups them as Open-X, Close-X, or Other. Performance varies across splits: closing jars and drawers is harder, plausibly because rotation or pushing differs from other closing motions. The test concerns learning with one example of a new type, not zero-shot object transfer.

Appendix B Implementation Details

B.1 TOQ-Nets

The implementation section specifies three object and three temporal layers, with relational/temporal widths 16/48 for soccer and 48/144 for manipulation. A later depth study recommends three relational and four temporal layers; the source does not reconcile those two specifications.

Input feature extractor.

Soccer features derive ground speed, vertical speed, acceleration, and pairwise distance from trajectories. Each normalized physical property produces five trainable sigmoid-threshold features; the temperature decays from 1 to 0.001. Manipulation instead uses a 64-dimensional sigmoid linear projection. The thresholds are learned, while the physical feature family is designed.

Equation
σ⁡(X−θiβ),\sigma\left(\frac{X-\theta_{i}}{\beta}\right),

B.2 STGCN

STGCN follows the cited architecture, with the complete layer specification retained. Average pooling over time and entities yields a 256-dimensional classification feature.

Input feature extractor.

The original soccer STGCN cannot consume pairwise-distance predicates, unlike TOQ-Net. Manipulation input formats are shared. The separate STGCN-2D experiment checks whether supplying binary features closes the gap.

B.3 STGCN-MAX

STGCN-MAX changes neighbor aggregation from averaging to elementwise maxima, making its relational update closer to the proposed model while retaining temporal convolutions.

B.4 STGCN-2D

STGCN-2D adds binary inputs such as distance. It improves standard and few-shot soccer accuracy somewhat but does not recover TOQ-Net’s generalization to different player counts or shifted timing. Both complete comparison tables are retained.

Reg. New All
TOQ-Net 87.7 52.2 79.8
STGCN-2D 84.1 46.9 75.8
Table 2: Ablation study of the STGCN-2D model on the soccer event dataset, evaluated by its few-shot learning performance.
9-way 3v3 11v11 6v6(Time warp)
TOQ-Net 88.4 77.4 77.1 86.9
STGCN-2D 84.5 17.5 16.6 39.7
Table 3: Ablation study of the STGCN-2D model on the soccer event dataset, evaluated by its generalization to different number of players and time-warped trajectories.

B.5 STGCN-LSTM

STGCN-LSTM pools graph features over entities and passes the resulting sequence to a two-layer bidirectional LSTM with hidden width 256. Soccer uses temporal kernel size three and manipulation size one, both at stride one. The manipulation ablation finds similar performance for kernel sizes one and three.

B.6 Space-Time Region Graph Networks

Space-Time Region Graphs use the cited graph-convolutional formulation. The full architectural table preserves each layer and its dimensions.

B.7 Non-Local Neural Networks

Non-Local Networks apply attention over the same space-time graph instead of graph convolution. The full layer specification is retained.

Model Reg. 1-Shot Full
STGCN 99.89±0.05 94.92±1.03 98.79±0.23
STGCN-LSTM
(kernel size 3)
99.89±0.05 96.16±1.39 99.04±0.31
STGCN-LSTM
(kernel size 1)
99.92±0.03 95.48±1.67 98.86±0.43
TOQ-Net 99.96±0.02 98.04±0.97 99.48±0.24
Table 4: Ablation study of different kernel sizes for the STGCN-LSTM model on the RLBench dataset, measured by per-action (macro) accuracy and averaged of four 1-shot splits and four random seeds per split. The ±{\pm} values indicate standard errors.

Soccer event detection.

Soccer batches contain 128 examples, balanced by first sampling an action category. Adam starts at 0.002, remains fixed for 50 epochs, then decays by 0.9 after six epochs without improved validation loss; training lasts 200 epochs. The separate input-scale adaptation task fine-tunes pretrained models for 10,000 iterations using a small new dataset.

Manipulation concept learning.

Manipulation training uses Adam at 0.003 with the analogous plateau schedule for 150 epochs. Batch sizes differ: 32 for STGCN-based models and four for TOQ-Net.

Model Input Dim. Output Dim. Kernel Size Stride Dropout Residual
STGCN - 64 7 1 0 False
64 64 7 1 0.5 True
64 64 7 1 0.5 True
64 64 7 1 0.5 True
64 128 7 2 0.5 True
128 128 7 1 0.5 True
128 128 7 1 0.5 True
128 256 7 2 0.5 True
256 256 7 1 0.5 True
256 256 7 1 0 True
STGCNS - 16 7 1 0 False
16 16 7 1 0.5 True
16 32 7 2 0.5 True
32 32 7 1 0.5 True
32 64 7 2 0.5 True
64 128 7 1 0 True
STGCNT - 8 7 1 0 False
8 12 7 2 0.5 True
12 64 7 1 0 True
Table 5: The STGCN architecture used in the paper.
Model Input Dim. Output Dim. Stride Residual
Space-Time - 64 1 False
64 64 1 True
64 64 1 True
64 64 1 True
64 128 2 True
128 128 1 True
128 128 1 True
128 256 2 True
256 256 1 True
256 256 1 True
Space-TimeS - 16 1 False
16 16 1 True
16 32 2 True
32 64 2 True
64 128 1 True
Space-TimeT - 32 1 False
32 64 2 True
64 64 2 True
Table 6: The Space-Time Region Graph architecture used in the paper.
Model Input Dim. Output Dim. Stride Residual
Non-Local - 64 1 False
64 64 1 True
64 64 1 True
64 64 1 True
64 128 2 True
128 128 1 True
128 128 1 True
128 256 2 True
256 256 1 True
256 256 1 True
Non-LocalS - 32 1 False
32 32 2 True
32 64 2 True
64 128 1 True
Non-LocalT - 32 1 False
32 32 2 True
32 64 2 True
Table 7: The Non-local Neural Networks architecture used in the paper.

Appendix C Additional Ablation Study

C.1 Network Architecture

Capacity ablations compare original, small, and tiny baselines. Most best-performing variants are smaller; main comparisons use the selected best variant for each baseline. TOQ-Net has about 35,000 parameters and still leads the reported soccer few-shot comparison.

Model #params Reg. Few-Shot Full
STGCN 3.42M 72.0±1.5 22.6±5.2 61.0±0.8
STGCNS 263K 73.0±1.5 22.5±4.9 61.7±0.5
STGCNT 34K 73.2±1.6 26.0±5.7 62.8±0.6
STGCN-LSTM 2.08M 72.5±1.6 20.2±5.0 60.9±0.7
STGCN-LSTMS 233K 72.1±1.7 20.5±5.9 60.6±0.7
STGCN-LSTMT 33K 72.7±1.4 23.8±5.9 61.9±0.6
STGCN-MAX 3.42M 73.5±4.8 20.2±6.0 61.7±4.1
STGCN-MAXS 263K 73.6±1.5 28.6±5.0 63.6±0.7
STGCN-MAXT 34K 74.5±1.2 25.4±6.1 63.6±0.7
Space-Time 263K 71.3±1.4 26.9±7.1 61.4±1.2
Space-TimeS 24K 74.7±1.4 30.7±6.8 64.9±0.6
Space-TimeT 14K 74.8±1.5 31.7±6.1 65.2±0.6
Non-Local 1.23M 74.5±4.2 44.3±7.0 67.8±3.7
Non-LocalS 108K 76.5±2.4 45.0±6.3 69.5±2.4
Non-LocalT 32K 75.6±1.2 44.8±6.4 68.8±1.0
TOQ-Net 35K 87.7±1.3 52.2±6.3 79.8±0.8

Table 8: Results on the soccer event dataset for baselines with different capacities, where modelS and modelT denote the small and the tiny variant for each model. The performance is measured by per-action (macro) accuracy, averaged over nine few-shot splits. The ±\pm values indicate standard errors. For each baseline we showed the best performance over three levels of capacities. In fact, performances of most of the models are not affected much by the capacity. We highlight the best-performing variants of all baselines, and use them in all comparisons in the main text.

C.2 Generalization to different input scales

A separate adaptation experiment enlarges the soccer field and changes temporal/spatial scale. All models are pretrained on the original nine-class task, then fine-tuned with few examples from the changed setting. TOQ-Net benefits especially when that adaptation set is small. These curves measure sample-efficient adaptation, unlike the direct player-count transfer test.

Original figure
Figure 2: Generalization to soccer environments with a different court size and agent speeds. The standard errors are computed based on three random seeds.

C.3 Full Results for the Time-Warping Generalization

The full table crosses all five tested team sizes with the timing-shift condition, retaining each model’s accuracy and relative degradation.

C.4 #Layers of TOQ-Net

The depth study shows that relational layers are required to aggregate entity-only inputs and that temporal layers support timing robustness. The plot recommends three relational and four temporal layers as a computation/performance tradeoff, while the earlier implementation paragraph states three of each.

C.5 Interpretability

Gradient-based dependency plots trace learned feature channels supporting high-pass and short-pass predictions. They illustrate compositional use of low-level properties, but are qualitative explanations rather than proof that every feature has a stable symbolic interpretation.

Model Time Warp 3v3 4v4 6v6 8v8 11v11
STGCN N
40.7±1.0
(-40.4%)
63.2±4.9
(-7.4%)
68.2±2.8
(0.0%)
55.4±3.3
(-18.8%)
44.4±2.1
(-34.9%)
STGCN Y
32.6±2.8
(-52.3%)
50.2±4.4
(-26.5%)
52.8±7.0
(-22.6%)
43.2±4.9
(-36.7%)
34.0±3.7
(-50.2%)
STGCN-MAX N
47.4±3.2
(-33.7%)
68.8±2.0
(-3.8%)
71.5±1.9
(0.0%)
59.1±0.7
(-17.3%)
45.6±2.5
(-36.2%)
STGCN-MAX Y
37.5±7.2
(-47.5%)
52.3±4.2
(-26.9%)
56.5±4.5
(-21.0%)
46.6±3.7
(-34.8%)
36.9±1.9
(-48.4%)
STGCN-LSTM N
39.7±1.1
(-43.1%)
60.4±0.2
(-13.5%)
69.8±0.1
(0.0%)
55.8±2.0
(-20.0%)
44.1±0.7
(-36.8%)
STGCN-LSTM Y
21.8±0.8
(-68.8%)
27.8±1.3
(-60.2%)
30.6±0.6
(-56.1%)
25.8±1.0
(-63.1%)
22.6±0.8
(-67.6%)
Space-Time N
29.0±1.6
(-60.4%)
53.5±3.2
(-27.0%)
73.3±0.3
(0.0%)
33.9±2.8
(-53.7%)
15.2±1.8
(-79.3%)
Space-Time Y
29.7±3.1
(-59.5%)
51.6±2.5
(-29.6%)
70.7±0.3
(-3.5%)
33.8±2.2
(-53.9%)
14.9±1.6
(-79.7%)
Non-Local N
45.9±5.1
(-41.2%)
70.7±5.3
(-9.5%)
78.1±5.8
(0.0%)
58.5±10.8
(-25.1%)
41.8±13.6
(-46.5%)
Non-Local Y
46.7±4.1
(-40.2%)
69.9±4.7
(-10.5%)
77.7±5.0
(-0.5%)
58.7±12.6
(-24.9%)
41.3±13.6
(-47.1%)
TOQ-Net N
77.4±3.5
(-12.4%)
88.3±0.7
(-0.0%)
88.4±0.6
(0.0%)
81.3±1.7
(-8.0%)
77.1±1.7
(-12.8%)
TOQ-Net Y
76.0±2.6
(-14.0%)
87.7±1.4
(-0.8%)
86.9±0.4
(-1.7%)
80.3±1.1
(-9.1%)
74.9±2.3
(-15.2%)
Table 9: Full results on generalization to scenarios with all combinations of #agents and temporally warping on the soccer event dataset. The standard error of all values are smaller than 2.5%, computed based on three random seeds.
Original figure
Figure 3: Comparing # of reasoning layers. Accuracy is tested on 6v6 9-way classification. When # of relational reasoning layers vary (blue), temporal reasoning layers are fixed at 4. When # of temporal reasoning layers vary (orange), relational reasoning layers are fixed at 3. When we have 0 temporal reasoning layers, we predict the sequence label based on the feature of the frame of interest. The purple line shows the performance on temporally warped trajectories when we have the # of relational reasoning layers fixed and vary the # of temporal reasoning layers. Relational layers are required for the prediction task (# of relational reasoning layer must be greater than 0), because there is no nullary feature input to the network: we need at least one relational reasoning layer to gather information across the entities. Overall, adding reasoning layers improves results, and 3 relational layers + 4 temporal layers is a good balance between computation and performance.
Refer to caption
Figure 4: Relevant features in temporal layers. Feature dependencies are computed by gradient. These dependencies and thresholds are learned end-to-end from data. Insets detail features for events in the first and the second stage of the high pass and short pass.
Concept demonstration

Open image ↗