Overview
The supplementary material specifies network architectures, typed program operations, tracking ablations, transfer-dataset construction, and training losses. Its implementation details qualify the main claims: object and event concepts avoid dense scene labels, but learning still uses a pretrained proposal network, program annotations, a supplied operation library, and staged training.
At a glance
Dynamic Concept Learner grounds objects and physical events from videos and question–answer pairs, then executes parsed programs over object tracks and learned dynamics. It improves the reported CLEVRER question-answering results without using ground-truth attribute or collision labels for its main variant. The demonstrations concern synthetic object-interaction videos and a small real block-tower dataset; they do not establish general physical understanding or robot control.
1 Introduction
The key idea is to separate tracking, concept grounding, dynamics prediction, and structured language reasoning so these components can support one another. Static attributes learned through question answering refine object associations; the refined tracks then train a dynamics predictor. Future and object-removal counterfactual questions use predicted scenes. Compared with the annotation-heavy CLEVRER/NS-DR setup, the main gain is reducing attribute and event supervision, not eliminating all supervision or hand-designed structure. Transfer to retrieval and grounding reuses learned visual concepts but trains parsers for the new expressions.
2 Related Work
The method extends the static concept-learning approach of the Neuro-Symbolic Concept Learner to temporal features, interactions, and predicted scenes. A learned Propagation Network supplies dynamics rather than a ground-truth simulator. Modular programs expose intermediate reasoning operations, while latent concepts and perception remain learned.
The related-work discussion distinguishes physical object events from broad human activities and benchmarks lacking language. Industrial manipulation is motivation rather than an evaluated application. Comparisons with prior visual-reasoning systems must account for different supervision, scene representations, and dynamics models.
3 Dynamic Concept Learner
Five components form the system: a trajectory detector, video feature extractor, dynamics predictor, language-to-program parser, and neural-symbolic executor. Objects and events are represented in a latent concept space. Static attributes refine trajectories; predicted object positions and image patches allow the same grounding machinery to inspect future or counterfactual scenes. The language side supplies symbolic composition, while learned features determine which objects and events satisfy each operation.
3.1 Model Details
The following components connect adjacent-frame tracking to concept recognition and compositional reasoning. Their interfaces impose substantial structure: object boxes, finite concept families, a graph dynamics model, and a fixed inventory of typed operations.
Object Detection and Tracking.
A pretrained region proposal network detects candidate boxes. Adjacent-frame connection scores combine proposal confidence with intersection over union, and linear sum assignment chooses a consistent matching for that frame pair. This improves on greedy linking but is not a global optimization over all video trajectories. Tracks contain normalized center, width, and height at each frame.
Equation 1
| (1) |
Video Feature Extraction.
Three representations serve different concepts. Averaged regional visual features describe stable attributes; concatenated box coordinates describe temporal motion and entering or exiting; pairwise union-region appearance and relative trajectory features describe collisions. Thus event grounding depends on both appearance and tracked geometry rather than a single pooled video descriptor.
Grounding Object and Event Concepts.
Concept embeddings are compared with transformed object or event features. Static concepts use visual features, motion and unary events use trajectories, and collisions use pairwise features. The resulting soft scores support differentiable filtering and question-answer losses. Their interpretation as probabilities is architectural terminology, without a calibration evaluation.
Trajectory Refinement.
Learned color, shape, and material predictions add an appearance-consistency term to the linking score. Agreement is averaged across the history of a trajectory and combined with spatial overlap. This lets question-answer supervision improve tracking indirectly, though incorrect concept predictions can also affect associations.
Equations 2–3
| (2) |
| (3) |
Dynamic Prediction.
Propagation Network predicts each object’s next box and cropped image patch from recent object states and pairwise relations. Predicted patches are placed into a scene and re-encoded to recognize future events. Counterfactual rollouts remove the specified object and its incident graph edges. Repeated prediction accumulates error, and this mechanism specifically models the benchmark’s object-removal interventions rather than unrestricted causal interventions.
Language Program Parsing.
An attention-based sequence-to-sequence parser maps questions and answer choices separately into executable programs. It is trained with one thousand annotated programs. The fixed operation vocabulary and these annotations are important qualifications to the claim of learning physical concepts from videos and question–answer pairs.
Symbolic Execution.
The executor applies typed filtering, event selection, temporal comparison, counting, and querying to learned representations. Predictive and counterfactual programs inspect generated scenes. The appendix also uses argmax, thresholding, and event sorting, so the implementation should not be described as smoothly differentiable at every decision. Causal ancestry follows the benchmark’s collision-event graph rather than learning an unconstrained causal model.
3.2 Training and inference
Training proceeds in stages. Initial trajectories use spatial matching; descriptive and explanatory questions train concepts; static attributes refine tracking; refined trajectories train dynamics; and all question categories then train the unified system. Dynamics first learns locations and subsequently learns locations plus image patches. This staged optimization is central to the reported result and is more specific than training every component from scratch jointly in one pass.
4 Experiments
Experiments evaluate four CLEVRER question categories, learned object and event concepts, transfer to grounding and retrieval, and a real block-tower extension. Tables distinguish option-level from whole-question multiple-choice accuracy and models trained with or without scene labels. These distinctions materially affect the interpretation of the headline scores.
| Methods | Extra Labels | Descriptive | Explanatory | Predictive | Counterfactual | ||||
| Attr. | Prog. | per opt. | per ques. | per opt. | per ques. | per opt. | per ques. | ||
| CNN+MLP | No | No | 48.4 | 54.9 | 18.3 | 50.5 | 13.2 | 55.2 | 9.0 |
| CNN+LSTM | 51.8 | 62.0 | 17.5 | 57.9 | 31.6 | 61.2 | 14.7 | ||
| Memory | 54.7 | 53.7 | 13.9 | 50.0 | 33.1. | 54.2 | 7.0 | ||
| HCRN | 55.7 | 63.3 | 21.0 | 54.1 | 21.0 | 57.1 | 11.5 | ||
| MAC (V) | 85.6 | 59.5 | 12.5 | 51.0 | 16.5 | 54.6 | 13.7 | ||
| TVQA+ | Yes | No | 72.0 | 63.3 | 23.7 | 70.3 | 48.9 | 53.9 | 4.1 |
| MAC (V+) | 86.4 | 70.5 | 22.3 | 59.7 | 42.9 | 63.5 | 25.1 | ||
| IEP (V) | No | Yes | 52.8 | 52.6 | 14.5 | 50.0 | 9.7 | 53.4 | 3.8 |
| TbD-net (V) | 79.5 | 61.6 | 3.8 | 50.3 | 6.5 | 56.1 | 4.4 | ||
| DCL (Ours) | 90.7 | 89.6 | 82.8 | 90.5 | 82.0 | 80.4 | 46.5 | ||
| NS-DR | Yes | Yes | 88.1 | 87.6 | 79.6 | 82.9 | 68.7 | 74.1 | 42.4 |
| NS-DR (NE) | 85.8 | 85.9 | 74.3 | 75.4 | 54.1 | 76.1 | 42.0 | ||
| DCL-Oracle (Ours) | 91.4 | 89.8 | 82.0 | 90.6 | 82.1 | 80.7 | 46.9 | ||
4.1 Implementation Details
The implementation uses ResNet-34 features and Adam with learning rate 10⁻⁴ for twenty epochs. Thirty-two frames are sampled per clip, and the dynamics system predicts states at four-frame intervals. The parser uses one thousand program annotations; static attributes and event labels are withheld from the main DCL variant. DCL-Oracle adds those labels to test their effect.
4.2 Comparisons on Temporal and Causal Reasoning
DCL reaches 90.7% descriptive accuracy. Explanatory accuracy is 89.6% per option and 82.8% per whole question; predictive accuracy is 90.5% and 82.0%; counterfactual accuracy is 80.4% and 46.5%. Corresponding NS-DR figures are 88.1%, 87.6%/79.6%, 82.9%/68.7%, and 74.1%/42.4%. The particularly low whole-question counterfactual score shows considerable remaining difficulty. DCL-Oracle is not a uniform empirical upper bound: explanatory whole-question accuracy is 82.0%, below DCL’s 82.8%.
4.3 Evaluation of Object and Event Concept Grounding in Videos
Attribute recognition reaches 99.7% color, 99.2% shape, and 99.6% material accuracy. Reported event or dynamic-concept accuracy is 89.7% moving, 93.3% stationary, 99.2% entering, 98.9% exiting, and 96.9% collision. These evaluations aggregate whether a concept or event occurs over frames; they do not by themselves establish precise event timing. Slowly moving objects visible only briefly are a noted source of stationary/moving confusion.
| Methods | Static Attributes | Dynamic Attributes | Events | |||||
| Color | Shape | Material | Moving | Stationary | In | Out | Collision | |
| DCL | 99.7 | 99.2 | 99.6 | 89.7 | 93.3 | 99.2 | 98.9 | 96.9 |
| Methods | Spatial Acc. | Spatial mIoU. | Frame Diff. | ||||
| Obj. | Col. | Obj. | Col. | In | Out | Col. | |
| WSSTG | 34.4 | 10.3 | 34.9 | 15.6 | 21.4 | 50.4 | 37.5 |
| DCL | 91.9 | 88.3 | 90.0 | 79.0 | 5.5 | 4.4 | 4.5 |
4.4 Generalization
Two datasets derived from the same CLEVRER video family test whether learned visual concepts support new language tasks. Grounding identifies a referenced trajectory or event, and retrieval ranks videos or expressions. Visual concept modules are reused, while task-specific parsers are trained on generated expression/program pairs. Consequently, the result supports transfer of visual representations with new language supervision rather than zero-training transfer to a new visual domain.
CLEVRER-Grounding.
Grounding expressions refer to objects, entry or exit moments, and collisions. An object or collision localization is counted correct above 0.5 trajectory overlap. DCL obtains 91.9% object and 88.3% collision spatial accuracy, with mean overlaps of 90.0% and 79.0%. Mean timing errors are 5.5 frames for entering, 4.4 for exiting, and 4.5 for collisions. The WSSTG baseline scores 34.4% and 10.3% spatial accuracy; all comparisons remain within this controlled synthetic dataset.
CLEVRER-Retrieval.
The video–expression match is the maximum grounding score across candidate proposals. DCL text-to-video mean average precision is 73.1% for objects, 81.9% for entry, 88.5% for exit, and 85.4% for collisions; video-to-text is 78.6%. The reported WSSTG and HGR baselines perform substantially worse under these evaluation conditions. The examples show compositional expressions and matched clips, not retrieval from an unrestricted real-world video collection.
4.5 Extension to real videos and the new concept
A real block-tower dataset contains 493 videos, split into 393 training and 100 testing videos, with generated question–answer pairs. DCL is trained to recognize the added falling concept; this is not zero-shot real-world transfer. Query, existence, and count accuracies are 97.0%, 95.5%, and 84.1%, averaging 92.6%, versus MAC’s 87.7% average. Color and falling concept accuracies are 98.5% and 91.8%. The small controlled setting leaves diverse scenes, object discovery, and broader physical interactions untested.
5 Discussion and future work
DCL demonstrates that structured reasoning and learned dynamics can acquire useful physical concept representations without dense attribute or event annotations. Its remaining dependencies include pretrained proposals, program supervision, manually specified operations, reliable tracking, and staged optimization. Performance still falls markedly on complete counterfactual questions. The experiments support controlled video reasoning and a small trained real-video extension, while broader real-world perception and physical planning remain future work.
Appendix A Implementation Details
The appendix gives implementation choices for the learned components and the grounding/retrieval baselines. These settings are necessary for reproducing the comparisons and understanding what remains fixed or supervised.
DCL Implementation Details.
The dynamics model uses a three-step history, two graph propagation steps, and 512-dimensional hidden states. Box prediction is trained before joint box-and-patch prediction. The parser uses 300-dimensional word embeddings and 256-dimensional hidden states. Concept scores use cosine similarity followed by a fixed shift and scale; architectural and training details are not inferred automatically from the question-answer corpus.
Baseline Implementation.
WSSTG receives the same object proposals and temporal candidates, including pairwise union regions for collisions. It is trained with the generated text–video associations using a triplet-ranking objective. HGR supplies a retrieval comparison. These are adapted baselines on the authors’ datasets, so the large margin should be interpreted in the context of the supplied candidate representations and supervision.
Appendix B Feature Extraction
Static features average ResNet representations from each object region together with scene context. Motion features concatenate all normalized box coordinates. Collision features combine a union-region representation with pairwise overlap, coordinate differences, and elementwise products of trajectory coordinates near a sampled frame. Missing objects receive zero-filled regional features. This design explicitly encodes geometry and interaction cues; it does not require the concept learner to discover those input relations from raw pixels alone.
Appendix C Dynamic Predictor
A directed graph represents objects as recent boxes and image patches and edges as relative coordinates. Object and relation encoders initialize latent states; two rounds of relation-to-object propagation update interaction effects. Separate predictors output the next normalized box and image patch. The original encoder, propagation, and prediction equations are retained.
Training minimizes location and pixel errors against observed future frames. Inference feeds predictions back into the history window. An object-removal counterfactual starts from the first three observed states with that vertex and its edges deleted. Rollout errors can compound, and this intervention relies on the given object-centric representation.
Equations 4–6
| (4) |
| (5) |
| (6) |
Appendix D Program Parser
An attention-based bidirectional long short-term memory encoder represents words; a decoder emits symbolic operations. Questions and options are parsed independently. Word embeddings have dimension 300 and hidden states dimension 256. Program labels train the parser explicitly; the video question-answer losses primarily ground the operations’ concepts rather than discovering an arbitrary program language.
Equations 7–8
| (7) |
| (8) |
Appendix E CLEVRER Operations and Program Execution
The supplied type system and operation library define the permitted reasoning programs. Tables 7–9 retain all original operation signatures, data types, and neural implementations, including filtering, querying, temporal selection, causal ancestry, counting, and future or counterfactual event handling. Original spelling and notation are preserved in these source tables.
| Type | Operation | Signature |
| Input Modules | Objects | |
| Returns all objects in the video | ||
| Events | ||
| Returns all events happening in the video | ||
| UnseenEvents | ||
| Returns all future events happening in the video | ||
| Start | ||
| Returns the special “start” event | ||
| end | ||
| Returns the special “end” event | ||
| Object Filter Modules | Filter_static_concept | |
| Select objects from the input list with the input static concept | ||
| Filter_dynamic_concept | ||
| Selects objects in the input frame with the dynamic concept | ||
| Event Filter Modules | Filter_in | |
| Select incoming events of the input objects | ||
| Filter_out | ||
| Select existing events of the input objects | ||
| Filter_collision | ||
| Select all collisions that involve an of the input objects | ||
| Get_col_partner | ||
| Return the collision partner of the input object | ||
| Filter_before | ||
| Select all events before the target event | ||
| Filter_after | ||
| Select all events after the target event | ||
| Filter_order | ||
| Select the event at the specific time order | ||
| Filter_ancestor | ||
| Select all ancestors of the input event in the causal graph | ||
| Get_frame | ||
| Return the frame of the input event in the video | ||
| Output Modules | Query_Attribute | |
| Returns the query attribute of the input object | ||
| Count | ||
| Returns the number of the input objects/ events | ||
| Exist | ||
| Returns “yes” if the input objects is not empty | ||
| Belong_to | ||
| Returns “yes” if the input event belongs to the input event sets | ||
| Negate | ||
| Returns the negation of the input boolean | ||
| Unique | ||
| Return the only event /object in the input list |
| Type | Semantics |
| object | A single object in the video. |
| objects | A set of objects in the video. |
| event | A single event in the video. |
| events | A set of events in the video. |
| order | The chronological order of an event, e.g. “First”, “Second” and “Last”. |
| static concept | Object-level static concepts like “Red”, “Sphere” and “Mental”. |
| dynamic concept | Object-level dynamic concepts like “Moving” and “Stationary”. |
| attribute | Static attributes including “Color”, “Shape” and “Material”. |
| frame | The frame number of an event. |
| int | A single integer like “0” and “1”. |
| bool | A single boolean value, “True” or “False”. |
| Type | Operation/ Signature | Implementation |
| Input Modules | Objects | |
| Events | for | |
| UnseenEvents | and | |
| Start | if else | |
| end | if else | |
| Object Filter Modules | Filter_static_concept | |
| Filter_dynamic_concept | ||
| Event Filter Modules | Filter_in | |
| Filter_out | ||
| Filter_collision | ||
| Get_col_partner | ||
| Filter_before | if | |
| Filter_after | if | |
| Filter_order | if | |
| Filter_ancestor | ||
| in the causal graph of event1} | ||
| Get_frame | ||
| Output Modules | Query_Attribute | |
| Count | ||
| Exist | ||
| Belong_to | ||
| Negate | False if bl else True | |
Representation for Objects, Events and Time.
Object sets and entry/exit event sets are vectors of scores over tracked objects. Collision events use an object-by-object-by-time tensor. Entry and exit times come from first and last appearance, while a binary mask selects valid time steps. Unique selects the largest-scoring object or event. These finite representations make the program operations explicit but assume a usable object inventory and trajectories.
Object and Event Concept Quantization.
A concept has a semantic embedding and an attribute-membership indicator. Transformed visual features are compared by cosine similarity, shifted by 0.15 and scaled by 0.2. Unary events instead use trajectory features, and collisions use pairwise features. Moment-specific moving or stationary queries retain a local coordinate window and zero the other times. The resulting confidence scores are learned from downstream answers rather than direct labels for each grounded concept.
Equation 9
| (9) |
Temporal and causal Reasoning.
Before/after operations update the time mask and suppress invalid trajectory coordinates. Order filters threshold event scores at zero and sort retained events by time. Ancestor selects valid events in the target collision’s causal graph. Table 9 also specifies first/last five-frame masks, minima or maxima for set filtering, and hard-threshold counting. These explicit temporal and causal conventions restrict the reasoning space and include discrete decisions.
Appendix F Trajectory Performance Evaluation.
Table 10 isolates tracking choices. At overlap threshold 0.9, greedy spatial linking yields 39.6% precision and 40.0% recall; linear assignment yields 88.5%/89.6%; adding learned attributes gives 91.5%/90.8%; and Kalman filtering gives 91.6%/90.9%. At threshold 0.5, adding attributes improves precision from 97.6% to 99.1% while recall changes from 98.8% to 98.4%, so improvements are not uniform on every metric. The largest gain comes from assignment, with attribute refinement and Kalman filtering providing smaller further changes.
| 0.5 | 0.6 | 0.7 | 0.8 | 0.9 | ||||||
| prec. | recall | prec. | recall | prec. | recall | prec. | recall | prec. | recall | |
| Greedy+IoU | 87.2 | 88.3 | 71.4 | 72.3 | 57.3 | 58.0 | 46.9 | 47.5 | 39.6 | 40.0 |
| Greedy+IoU+Attr. | 97.0 | 97.4 | 95.1 | 95.5 | 93.1 | 93.5 | 89.9 | 90.3 | 83.6 | 83.9 |
| LSM+IoU | 97.6 | 98.8 | 96.9 | 98.1 | 96.0 | 97.2 | 93.8 | 95.0 | 88.5 | 89.6 |
| LSM+IoU+Attr. | 99.1 | 98.4 | 98.6 | 97.9 | 97.9 | 97.2 | 96.2 | 95.5 | 91.5 | 90.8 |
| LSM+IoU+Attr.+KF | 99.1 | 98.4 | 98.6 | 97.9 | 98.0 | 97.3 | 96.2 | 95.6 | 91.6 | 90.9 |
Appendix G Statistics for CLEVRER-Grounding and CLEVRER-Retrieval
Grounding has an average of 10.2 generated expressions per video and is evaluated on all five thousand CLEVRER validation videos. Retrieval training averages 7.4 expressions per video; evaluation uses 1,129 unique expressions and the first one thousand validation videos as the gallery. Figures 5–7 retain all example frames, highlighted target regions, and query text. The language is generated within the same compositional object/event domain, limiting claims about natural unrestricted language or novel visual distributions.
Appendix H Training Objectives
Open-ended answers use cross-entropy, count answers use mean squared error, and each multiple-choice option becomes a separate Boolean training question. Program annotations train the parser with cross-entropy. Dynamics minimizes squared error on object coordinates and every pixel of the predicted cropped patch. All original loss equations remain below. These distinct objectives explain why the system’s supervision is richer than answer correctness alone, even though dense scene attribute and collision labels are omitted from the main model.
Equations 10–13
| (10) |
| (11) |
| (12) |
| (13) |