Overview
The complete article, annotation examples, implementation details, labeling interfaces, and dataset statement are included. The authors release the data under CC0 and describe versioned maintenance on the project website. The preprint identifies dataset version 3, updated May 12, 2023; counts below are those reported in this source.
At a glance
CLEVRER-Humans adds natural-language event descriptions and human causal judgments to simulated CLEVRER videos. Iterative cloze annotation supplies seed descriptions; trajectory-conditioned generators expand them; annotators verify events and rate causal relations. Causal event graphs are converted into question-answer pairs. Baselines perform near chance despite strong results on the original benchmark, exposing a combined challenge of language diversity, different causal labels, and limited training data. These labels measure human judgments in a restricted physical setting rather than objective causal mechanisms.
1 Introduction
Original CLEVRER descriptions cover entering, exiting, and colliding, with causal relations assigned by rules. Human event descriptions include richer motion and interaction distinctions, and human causal judgments need not match those heuristics. The proposed graph records natural-language events as nodes and graded causal links as edges. Iteratively reusing newly written events expands coverage; learned generation reduces annotation cost before human verification. The resulting benchmark targets physical grounding, causal attribution, and data-efficient learning, and can also support cognitive studies.
2 Related work
The paper distinguishes prediction/intervention benchmarks, tasks that search for initial conditions, and language-based physical reasoning. It connects human causal attribution to force dynamics, conserved quantities, causal models, and counterfactual simulation, without claiming that one theory fully determines the annotations. The chosen connectives, especially “because” and “responsible for,” influence the task. Video-language generation provides a practical augmentation tool; generated captions here are grounded in simulator trajectories and subsequently checked by people.
| Dataset | Video |
|
|
|
|
| ||||||||||
| CLEVR [12] | - | ✓ | ✓ | - | - | - | ||||||||||
| MovieQA [13] | ✓ | ✓ | - | ✓ | - | - | ||||||||||
| TGIF-QA [14] | ✓ | ✓ | - | ✓ | - | - | ||||||||||
| TVQA+ [15] | ✓ | ✓ | - | ✓ | - | - | ||||||||||
| AGQA [16] | ✓ | ✓ | - | ✓ | - | - | ||||||||||
| IntPhys [17] | ✓ | - | ✓ | - | ✓ | - | ||||||||||
| Galileo [18] | ✓ | - | ✓ | - | ✓ | - | ||||||||||
| PHYRE [19] | ✓ | - | ✓ | - | ✓ | - | ||||||||||
| CATER [8] | ✓ | ✓ | ✓ | - | ✓ | - | ||||||||||
| CoPhy [20] | ✓ | - | ✓ | - | ✓ | - | ||||||||||
| CRAFT [10] | ✓ | ✓ | ✓ | - | ✓ | - | ||||||||||
| CLEVRER [7] | ✓ | ✓ | ✓ | - | ✓ | - | ||||||||||
| ComPhy [21] | ✓ | ✓ | ✓ | - | ✓ | - | ||||||||||
| CLEVRER-Humans | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
3 CLEVRER-Humans
Videos contain at most five objects moving and colliding on a plane, drawn from three shapes, eight colors, and two materials. The benchmark focuses on explanatory questions: which listed event is responsible for a target event. It expands language and event categories while retaining the synthetic visual domain.
3.1 Delving into Causal Relation Annotations
On 50 videos, original heuristic relations and an intervention-based labeling method are compared with human judgments. Only 62% of heuristic-positive pairs and 61% of intervention-positive pairs are human-positive; the original heuristic covers 96% of human-positive pairs. Wording may partly explain this asymmetry. The source describes its intervention test using “B happens even if” relevant objects are removed, which is inconsistent with the usual difference-making interpretation; its reported labels and tables are retained without silently reversing that statement.
| P(X|Y) | Y = CLEVRER | Y = Counterfactual | Y = Human |
| X = CLEVRER | 1.00 | 0.74 | 0.96 |
| X = Counterfactual | 0.89 | 1.00 | 0.54 |
| X = Human | 0.62 | 0.61 | 1.00 |
3.2 Augment CLEVRER with Human Annotations
A causal event graph stores positive, negative, or unlabelled directed relations between event descriptions. These human-labeled graphs need not obey the inference rules of formal causal graphical models. Collection has three stages: seed human descriptions on a small video subset, generate candidate descriptions for more videos, and have annotators verify nodes and rate edges. The final human checks are essential because the candidate events are partly machine-generated.
3.3 Stage I: Event Cloze
Starting from a CLEVRER event, workers complete either an effect-before-“because” or a cause-after-“because” sentence. Each new response can seed a later prompt. The 100 training videos with the most collisions yield 1,724 descriptions: 740 single-object and 984 pairwise examples. Selecting collision-rich videos improves annotation density but also shapes the seed distribution.
| Object State | Relative Position | Collision |
| Object A comes/moves/is hurled from some direction. | Object A is in the path of object B. | Object A is hit/struck by object B |
| Object A is thrown/pushed from some direction. | Object A is aimed at object B. | Object A hits/collides with/bumps into/runs into object B |
| Object A changes direction (towards some direction) | Object A moves together with object B | Object A is avoided hitting object B. |
| Object A stops moving | Object A pushes object B into object C. |
3.4 Stage II: Trajectory-based Event Description Generation
Two attention-based GRU encoder–decoders generate single-object and pairwise descriptions from ground-truth simulator trajectories, including attributes, position, velocity, and angular velocity. A manually defined concept vocabulary links noun phrases to objects. Pairwise generation first uses rules for approaching, colliding, or moving together, since the seed data lacks negative interaction descriptions. On a manually labeled 100-video subset, this filter has 99.2% recall and raises human-judged caption accuracy from 80.6% to 87.2%. Generation considers ten likely sequences per object or pair, then removes grammar errors and nonexistent-object references, rebalances verbs to match the seed distribution, and samples at most ten descriptions per video. Thus event diversity remains influenced by the rule filter, vocabulary, and seed captions.
3.5 Stage III: CEG Condensation
Workers first decide whether both descriptions are understandable and occur in the video. If so, they rate whether the first event is responsible for the second on a five-point scale. Majority-rejected descriptions are removed; scores of at least four define positive edges. The authors also manually remove ambiguous descriptions and enrich negative edges. Raw graded judgments are retained alongside the binary task.
3.6 QA conversion
Eligible graph nodes become “Which event is responsible for…” questions. Positive and negative options are sampled to balance candidate labels, with a randomly chosen number of correct options. The split follows the original CLEVRER video split.
3.7 Dataset Statistics
The statistics section reports 1,099 videos, 5,538 descriptions, 23,032 relation annotations, and 1,413 question-answer pairs. It reports average graph sizes of 4.88 nodes and 20.68 labeled edges, description vocabulary 157 versus CLEVRER’s 82, and about 30 listed verbs. The average causal score is 1.70, reflecting many unrelated event pairs. Elsewhere the paper reports 219 vocabulary items and 1,076 question pairs without fully reconciling the scopes, so these should not be treated as one consistent release-size specification.
4 Experiments
Baselines test language-only cues, symbolic-program transfer, and end-to-end video-language learning.
4.1 Methods
The best-guess baseline always chooses the majority “No” label. An LSTM language-only classifier tests textual biases. NS-DR and VRDP parse text into programs using original CLEVRER supervision; there are no program labels for the new dataset. CNN+LSTM and CNN+BERT combine frame features with text encoders. ALOE uses object-centric MONet features and a multimodal transformer.
4.2 Results
On CLEVRER-Humans, per-option model accuracies range around 50–54%, versus 50.7% for majority guessing and 84.5% for people. Per-question human accuracy is 71.4%; the best listed model result is 34.2%. Strong original-CLEVRER performance does not transfer: pretrained ALOE drops from 98.5% to 54.0% per option. Training or fine-tuning on the small dataset yields little improvement and can overfit. The authors attribute the difficulty to unfamiliar language/events, different causal labels, and sample scarcity; these factors are not cleanly isolated by the comparisons. Confidence intervals cover five runs where available, while released pretrained checkpoints lack such intervals. Human evaluation uses 50 test videos and reports 90% of descriptions as natural. Question-level comparisons across datasets also differ in option count, about four versus two.
| CLEVRER | CLEVRER-Humans | ||||
| Model | Training | Per-Option | Per-Ques. | Per-Option | Per-Ques. |
| Best Guess | N/A | 50.2 | 16.5 | 50.7 | 30.1 |
| Lang-Only | Scratch | 59.7 | 13.6 | 51.9 ( 1.09) | 30.4 ( 1.90) |
| NS-DR [7] | Pretrain | 87.6 | 79.6 | 51.0 | 32.0 |
| VRDP [47] | Pretrain | 96.3 | 91.9 | 50.9 | 31.6 |
| CNN+LSTM | Pretrain | 62.0 | 17.5 | 50.3 | 30.0 |
| CNN+LSTM | Scratch | N/A† | N/A† | 51.7 ( 0.64) | 34.2 ( 1.69) |
| CNN+LSTM | Pretrain+Finetune | 62.0 | 17.5 | 51.5 ( 2.35) | 30.8 ( 0.69) |
| CNN+BERT | Pretrain | 55.1 | 11.5 | 52.9 | 32.0 |
| CNN+BERT | Scratch | N/A† | N/A† | 52.0 ( 2.34) | 30.2 ( 2.41) |
| CNN+BERT | Pretrain+Finetune | 55.1 | 11.5 | 50.1 ( 0.68) | 30.4 ( 3.09) |
| ALOE [43] | Pretrain | 98.5 | 96.0 | 54.0 | 26.9 |
| ALOE [43] | Scratch | N/A† | N/A† | 51.8 ( 1.00) | 31.7 ( 0.79) |
| ALOE [43] | Pretrain+Finetune | 98.5 | 96.0 | 52.7 ( 1.36) | 32.1 ( 1.36) |
| Human | N/A | N/A | N/A | 84.5 | 71.4 |
5 Conclusion
The dataset is intended chiefly for zero-shot evaluation or few-shot learning because of its size. Its main methodological contribution is economical annotation through iterative cloze prompts and checked neural caption generation. Extending the method to natural video, more complex events, and graded causal reasoning remains future work. The authors acknowledge the listed research support.
Appendix A Dataset Visualization and Analysis
The appendix shows the growth of cloze graphs and examples of final human-labeled graphs. The reported 29.4 descriptions per illustrated stage-I video concerns the iterative collection analysis, while the main seed totals and final condensed graph counts describe different reported quantities.
A.1 Dataset Statistics
Descriptions average 6.99 words versus 8.93 in original CLEVRER. Color and shape distributions account for synonyms such as ball/sphere and gray/silver. Verb families distinguish motion and collision nuances, and tense can change the event being attributed as a cause. The source lists 30 verb entries but repeats some words, so the displayed list alone does not establish 30 unique lemmas. The seven statistics panels retain graph sizes, sentence length, object properties, verb families, and graded edge scores.
| Y = | Y = | Y = | Y = | Y = | |
| P(X = Human | Y) | 0.62 | 0.61 | 0.23 | 0.34 | 0.34 |
| P(Y | X = Human) | 0.96 | 0.54 | 0.29 | 0.62 | 0.33 |
A.2 Comparison between Heuristic and Human Causal Judgments
Participants suggest an average binary cutoff of 3.6; the authors choose four to favor stronger causal judgments. Figure 10 varies that cutoff, and Table 5 compares Boolean combinations of the two heuristic labelers. The authors conclude that these combinations remain poor substitutes for human labels. Some conditional-probability entries in Table 5 do not respect the expected monotonicity of the stated union operation, so the original table is retained as reported rather than used to derive new quantitative claims.
Appendix B Implementation Details
Implementation details specify the privileged trajectory inputs used for annotation generation and the video/text inputs used by evaluation baselines.
B.1 Stage II Implementation
Each object contributes a 24-dimensional state over 128 frames. Pairwise rules inspect segments where every coordinate displacement is under 0.5 m; sustained proximity and velocity changes distinguish moving together, approach, and collision. Pairs without detected events are discarded. Separate attention-GRU models use hidden widths 256 for single objects and 128 for pairs, decoder dropout 0.3/0.1, and learning rate 0.001. Post-processing rejects repeated words and incomplete verb arguments.
B.2 Baseline Implementation
Language-Only models.
The language-only model uses GloVe embeddings, a 512-wide LSTM, dropout 0.2, Adam at 0.0004, weight decay 0.00001, and batch size four. Model selection holds out 20% of training pairs.
CNN+LSTM.
CNN+LSTM samples 25 frames, extracts pretrained ResNet-50 features, and uses separate 512-wide video and question LSTMs with GloVe initialization. Batch size is 128; Adam learning rates are 0.00001 from scratch and 0.001 for fine-tuning. Twenty percent of training pairs supplies validation.
BERT+LSTM
CNN+BERT keeps the same visual encoder and freezes pretrained uncased BERT-base text weights. Inputs are padded or truncated to 32 tokens. Batch size, learning-rate settings, and validation fraction follow the preceding baseline. The appendix heading says BERT+LSTM, while its prose describes CNN+BERT.
ALOE.
ALOE uses released pretrained MONet object embeddings. Because the release lacks training code, the authors implement training themselves, use Adam with weight decay 0.001 after comparison with nearby values, and select models on a 5% training holdout.
B.3 Error analysis
Transferred program parsers map unfamiliar events such as slowing or stopping to incorrect collision operations. Learned models struggle with temporal and linguistic distinctions, such as what part of a trajectory “comes up” denotes. Pretraining may also preserve heuristic causal associations that people reject. These qualitative examples support several plausible failure sources, while the small dataset limits the strength of causal attribution among them.
| Event | Parsed program |
| The purple sphere slows down from the right. | ["events", "objects", "purple", "filter_color", "sphere", "filter_shape", "unique", "filter_collision", "objects", "unique", "filter_color", "sphere", "filter_shape", "unique", "filter_collision", "unique"] |
| The red ball comes to a stop. | ["events", "objects", "red", "filter_color", "unique", "filter_collision", "objects", "red", "filter_color", "unique", "filter_collision", "unique"] |
| The yellow cube comes from the right side at a fast speed. | ["events", "objects", "yellow", "filter_color", "cube", "filter_shape", "unique", "filter_out", "unique"] |
Appendix C Labeling Interface
MTurk interfaces provide replayable videos, highlighted objects, examples, and explicit response instructions. Cloze workers complete a seven-question comprehension quiz with two attempts. The authors report about $3,500 total compensation: U.S.-restricted tasks paid an estimated $7.7/hour, while later recruitment outside the U.S. brought the overall estimated average to $6.1/hour. Figures 12 and 13 preserve both complete example interfaces.
Appendix D Dataset Release
The dataset is released under CC0 with a structured data statement.
Short form data statement
The texts combine English crowd-written seeds and machine-generated descriptions; correctness and causal labels come from workers watching videos and reading event pairs.
Curation Nationale
Curation aims for interpretable, correct, diverse descriptions through cloze collection, trajectory-based generation, grammar/object filtering, verb balancing, and human event and relation judgments.
Language Variety
The source includes mainstream U.S. English, but detailed language-variety information is unavailable.
Speaker Demographic
Seed cloze writers were restricted to U.S. workers. The authors expect many native English speakers but do not provide measured participant demographics.
Annotator Demographic
Annotators must have at least 1,000 approved tasks. Batches contain 30 question pairs, usually from one or two videos; English comprehension and annotation quality are checked. Broader demographic estimates are referenced rather than measured for this sample.
Speech Situation
Descriptions are written for a general audience. Workers may replay the roughly five-second videos while completing prompts or judging event pairs.
Text Characteristics
Sentences describe one or more objects and their motion or interactions. Generated language is intended to resemble the seed-caption vocabulary and structure.
Recording Quality
The source marks recording quality as not applicable.
Other
No additional information is supplied under this field.
Provenance Appendix
All source videos come from CLEVRER.
D.1 Intended Use
The intended use is evaluating physical-scene and causal-language reasoning in short, restricted synthetic videos; it does not establish competence on real-world causal inference.
D.2 Maintenance Plan
The authors state plans for persistent project-site hosting, release versioning, and regular archival backups.
D.3 Quality Check
Three annotators provide overlapping event-correctness judgments for majority voting. Relation-label batches include an author-curated quality-control video selected from a 30-video set; submissions are accepted only when control answers sufficiently agree with the authors. This checks consistency with the chosen annotation standard rather than making subjective causal labels uniquely correct.