Literature
Display
← All papers

CLEVRER-Humans: Describing Physical and Causal Events the Human Way

Jiayuan Mao · Xuelin Yang · Xikun Zhang · Noah D. Goodman · Jiajun Wu

· NeurIPS 2022 Datasets and Benchmarks Track ↗

Contents

Overview

The complete article, annotation examples, implementation details, labeling interfaces, and dataset statement are included. The authors release the data under CC0 and describe versioned maintenance on the project website. The preprint identifies dataset version 3, updated May 12, 2023; counts below are those reported in this source.

At a glance

CLEVRER-Humans adds natural-language event descriptions and human causal judgments to simulated CLEVRER videos. Iterative cloze annotation supplies seed descriptions; trajectory-conditioned generators expand them; annotators verify events and rate causal relations. Causal event graphs are converted into question-answer pairs. Baselines perform near chance despite strong results on the original benchmark, exposing a combined challenge of language diversity, different causal labels, and limited training data. These labels measure human judgments in a restricted physical setting rather than objective causal mechanisms.

1 Introduction

Original CLEVRER descriptions cover entering, exiting, and colliding, with causal relations assigned by rules. Human event descriptions include richer motion and interaction distinctions, and human causal judgments need not match those heuristics. The proposed graph records natural-language events as nodes and graded causal links as edges. Iteratively reusing newly written events expands coverage; learned generation reduces annotation cost before human verification. The resulting benchmark targets physical grounding, causal attribution, and data-efficient learning, and can also support cognitive studies.

Refer to caption
Figure 1: For (a) each video in the CLEVRER dataset, (b) CLEVRER-Humans annotates a human-labeled graphical representation of physical events and their causal relations, in the form of causal event graphs (CEGs). Each CEG composes of a collection of nodes associated with textual descriptions of events, and human-labeled directional edges indicating the causal relationship between objects. Each edge is also associated with a score indicating a human’s graded attribute of causal relations. (c) The compact representation of CEGs can be easily translated into question-answer pairs to evaluate video reasoning models.

2 Related work

The paper distinguishes prediction/intervention benchmarks, tasks that search for initial conditions, and language-based physical reasoning. It connects human causal attribution to force dynamics, conserved quantities, causal models, and counterfactual simulation, without claiming that one theory fully determines the annotations. The chosen connectives, especially “because” and “responsible for,” influence the task. Video-language generation provides a practical augmentation tool; generated captions here are grounded in simulator trajectories and subsequently checked by people.

Dataset Video
Question
Answering
Diagnostic
Annotation
Natural Language
Events
Causal
Reasoning
Human Causal
Judgements
CLEVR [12] - ✓ ✓ - - -
MovieQA [13] ✓ ✓ - ✓ - -
TGIF-QA [14] ✓ ✓ - ✓ - -
TVQA+ [15] ✓ ✓ - ✓ - -
AGQA [16] ✓ ✓ - ✓ - -
IntPhys [17] ✓ - ✓ - ✓ -
Galileo [18] ✓ - ✓ - ✓ -
PHYRE [19] ✓ - ✓ - ✓ -
CATER [8] ✓ ✓ ✓ - ✓ -
CoPhy [20] ✓ - ✓ - ✓ -
CRAFT [10] ✓ ✓ ✓ - ✓ -
CLEVRER [7] ✓ ✓ ✓ - ✓ -
ComPhy [21] ✓ ✓ ✓ - ✓ -
CLEVRER-Humans ✓ ✓ ✓ ✓ ✓ ✓
Table 1: Comparison between CLEVRER-Humans with other visual reasoning benchmarks. Our dataset is the only dataset of natural language descriptions of physical events and human judgments.

3 CLEVRER-Humans

Videos contain at most five objects moving and colliding on a plane, drawn from three shapes, eight colors, and two materials. The benchmark focuses on explanatory questions: which listed event is responsible for a target event. It expands language and event categories while retaining the synthetic visual domain.

3.1 Delving into Causal Relation Annotations

On 50 videos, original heuristic relations and an intervention-based labeling method are compared with human judgments. Only 62% of heuristic-positive pairs and 61% of intervention-positive pairs are human-positive; the original heuristic covers 96% of human-positive pairs. Wording may partly explain this asymmetry. The source describes its intervention test using “B happens even if” relevant objects are removed, which is inconsistent with the usual difference-making interpretation; its reported labels and tables are retained without silently reversing that statement.

Refer to caption
Figure 2: This are two examples showing difference of human causal judgment and CLEVRER’s heuristic causal relation. The arrows in the image represent the moving direction of objects of interest.
P(X|Y) Y = CLEVRER Y = Counterfactual Y = Human
X = CLEVRER 1.00 0.74 0.96
X = Counterfactual 0.89 1.00 0.54
X = Human 0.62 0.61 1.00
Table 2: Comparison between different heuristics-generated causal labels and human labels, on a sampled subset of CLEVRER [7]. The entry P(X|Y) denotes the fraction of event relations that are annotated as causal by protocol X given that the relations are annotated as causal by protocol Y.

3.2 Augment CLEVRER with Human Annotations

A causal event graph stores positive, negative, or unlabelled directed relations between event descriptions. These human-labeled graphs need not obey the inference rules of formal causal graphical models. Collection has three stages: seed human descriptions on a small video subset, generate candidate descriptions for more videos, and have annotators verify nodes and rate edges. The final human checks are essential because the candidate events are partly machine-generated.

Refer to caption
Figure 3: The overall labeling pipeline of CLEVRER-Humans. (a) Starting from input videos, (b) we use a event cloze task to collect a small number of human-written event descriptions about videos (Stage I). (c) Next, we train neural event description generators to augment all videos with a collection of events (Stage II). (d) Finally, human annotators label the correctness of all generated events (in this case, event E is incorrect and thus the node is dropped) as well as their causal relations (Stage III).

3.3 Stage I: Event Cloze

Starting from a CLEVRER event, workers complete either an effect-before-“because” or a cause-after-“because” sentence. Each new response can seed a later prompt. The 100 training videos with the most collisions yield 1,724 descriptions: 740 single-object and 984 pairwise examples. Selecting collision-rich videos improves annotation density but also shapes the seed distribution.

Object State Relative Position Collision
Object A comes/moves/is hurled from some direction. Object A is in the path of object B. Object A is hit/struck by object B
Object A is thrown/pushed from some direction. Object A is aimed at object B. Object A hits/collides with/bumps into/runs into object B
Object A changes direction (towards some direction) Object A moves together with object B Object A is avoided hitting object B.
Object A stops moving Object A pushes object B into object C.
Table 3: Event examples collected in the first stage, categorized into single-object states, pairwise relative positions, and pairwise collision events.
Original figure
Figure 4: We use a novel iterative data collection procedure to collect CEGs on MTurk. Starting from a single node (iteration 0, blue), we iteratively sample nodes in the current CEG and collect either a cause or an effect event, and add this new node to the CEG. Red: nodes added in the first iteration; yellow: nodes added in the second iteration.

3.4 Stage II: Trajectory-based Event Description Generation

Two attention-based GRU encoder–decoders generate single-object and pairwise descriptions from ground-truth simulator trajectories, including attributes, position, velocity, and angular velocity. A manually defined concept vocabulary links noun phrases to objects. Pairwise generation first uses rules for approaching, colliding, or moving together, since the seed data lacks negative interaction descriptions. On a manually labeled 100-video subset, this filter has 99.2% recall and raises human-judged caption accuracy from 80.6% to 87.2%. Generation considers ten likely sequences per object or pair, then removes grammar errors and nonexistent-object references, rebalances verbs to match the seed distribution, and samples at most ten descriptions per video. Thus event diversity remains influenced by the rule filter, vocabulary, and seed captions.

Refer to caption
Figure 5: The neural event description generation pipeline (Stage II). (a) Given the input video, (b) we first extract per-object trajectories, composed of their attributes, positions, velocities, and angular velocities. For each object, we use a single-object event generator to sample event descriptions. For each pair of objects, we use a cascaded generator composed of a rule-based event detector and a neural pairwise generator. All generated events will pass a post-processing unit composed of three stages: grammar check, object existence check, and verb re-balancing. (c) The final product of the pipeline is a candidate node set of the CEG, which will be further annotated by humans in Stage III, the CEG condensation stage.
Refer to caption
Figure 6: The MTurk interface illustration for the second stage. The MTurker sees the input video with objects mentioned in the events highlighted with colored bounding boxes. The MTurker answers two questions about the video: 1) whether event descriptions are correct and 2) a graded judgement of whether event A is responsible for event B. We use the same wording choice “be responsible for” to be consistent with the original CLEVRER dataset [7].

3.5 Stage III: CEG Condensation

Workers first decide whether both descriptions are understandable and occur in the video. If so, they rate whether the first event is responsible for the second on a five-point scale. Majority-rejected descriptions are removed; scores of at least four define positive edges. The authors also manually remove ambiguous descriptions and enrich negative edges. Raw graded judgments are retained alongside the binary task.

3.6 QA conversion

Eligible graph nodes become “Which event is responsible for…” questions. Positive and negative options are sampled to balance candidate labels, with a randomly chosen number of correct options. The split follows the original CLEVRER video split.

3.7 Dataset Statistics

The statistics section reports 1,099 videos, 5,538 descriptions, 23,032 relation annotations, and 1,413 question-answer pairs. It reports average graph sizes of 4.88 nodes and 20.68 labeled edges, description vocabulary 157 versus CLEVRER’s 82, and about 30 listed verbs. The average causal score is 1.70, reflecting many unrelated event pairs. Elsewhere the paper reports 219 vocabulary items and 1,076 question pairs without fully reconciling the scopes, so these should not be treated as one consistent release-size specification.

4 Experiments

Baselines test language-only cues, symbolic-program transfer, and end-to-end video-language learning.

4.1 Methods

The best-guess baseline always chooses the majority “No” label. An LSTM language-only classifier tests textual biases. NS-DR and VRDP parse text into programs using original CLEVRER supervision; there are no program labels for the new dataset. CNN+LSTM and CNN+BERT combine frame features with text encoders. ALOE uses object-centric MONet features and a multimodal transformer.

4.2 Results

On CLEVRER-Humans, per-option model accuracies range around 50–54%, versus 50.7% for majority guessing and 84.5% for people. Per-question human accuracy is 71.4%; the best listed model result is 34.2%. Strong original-CLEVRER performance does not transfer: pretrained ALOE drops from 98.5% to 54.0% per option. Training or fine-tuning on the small dataset yields little improvement and can overfit. The authors attribute the difficulty to unfamiliar language/events, different causal labels, and sample scarcity; these factors are not cleanly isolated by the comparisons. Confidence intervals cover five runs where available, while released pretrained checkpoints lack such intervals. Human evaluation uses 50 test videos and reports 90% of descriptions as natural. Question-level comparisons across datasets also differ in option count, about four versus two.

CLEVRER CLEVRER-Humans
Model Training Per-Option Per-Ques. Per-Option Per-Ques.
Best Guess N/A 50.2 16.5 50.7 30.1
Lang-Only Scratch 59.7 13.6 51.9 (±\pm 1.09) 30.4 (±\pm 1.90)
NS-DR [7] Pretrain 87.6 79.6 51.0 32.0
VRDP [47] Pretrain 96.3 91.9 50.9 31.6
CNN+LSTM Pretrain 62.0 17.5 50.3 30.0
CNN+LSTM Scratch N/A† N/A† 51.7 (±\pm 0.64) 34.2 (±\pm 1.69)
CNN+LSTM Pretrain+Finetune 62.0 17.5 51.5 (±\pm 2.35) 30.8 (±\pm 0.69)
CNN+BERT Pretrain 55.1 11.5 52.9 32.0
CNN+BERT Scratch N/A† N/A† 52.0 (±\pm 2.34) 30.2 (±\pm 2.41)
CNN+BERT Pretrain+Finetune 55.1 11.5 50.1 (±\pm 0.68) 30.4 (±\pm 3.09)
ALOE [43] Pretrain 98.5 96.0 54.0 26.9
ALOE [43] Scratch N/A† N/A† 51.8 (±\pm 1.00) 31.7 (±\pm 0.79)
ALOE [43] Pretrain+Finetune 98.5 96.0 52.7 (±\pm 1.36) 32.1 (±\pm 1.36)
Human N/A N/A N/A 84.5 71.4
Table 4: Test performance of different models on both the original CLEVRER dataset and our new CLEVRER-Humans dataset. The training column denotes the training schema for different models (in percentage, and the ±\pm sign shows the 95% confidence interval computed across 5 different runs). We compare both per-option accuracy and per-question accuracy, following the original paper [7]. Note that the number of options per question is ∼\sim4 for CLEVRER and 2 for CLEVRER-Humans. N/A marker†: models trained only on CLEVRER-Humans.

5 Conclusion

The dataset is intended chiefly for zero-shot evaluation or few-shot learning because of its size. Its main methodological contribution is economical annotation through iterative cloze prompts and checked neural caption generation. Extending the method to natural video, more complex events, and graded causal reasoning remains future work. The authors acknowledge the listed research support.

Appendix A Dataset Visualization and Analysis

The appendix shows the growth of cloze graphs and examples of final human-labeled graphs. The reported 29.4 descriptions per illustrated stage-I video concerns the iterative collection analysis, while the main seed totals and final condensed graph counts describe different reported quantities.

A.1 Dataset Statistics

Descriptions average 6.99 words versus 8.93 in original CLEVRER. Color and shape distributions account for synonyms such as ball/sphere and gray/silver. Verb families distinguish motion and collision nuances, and tense can change the event being attributed as a cause. The source lists 30 verb entries but repeats some words, so the displayed list alone does not establish 30 unique lemmas. The seven statistics panels retain graph sizes, sentence length, object properties, verb families, and graded edge scores.

Original figure
Figure 7: Statistics on the CLEVRER-Humans dataset. From left to right, the first row figures are distributions of (a) the number of nodes per CEG, (b) the number of edges per CEG, and (c) sentence lengths excluding the "which of the following is responsible for" prefix. The second row figures are distributions of (d) object shapes, (e) colors, (f) event type attributions based on verbs, and (g) CEG edge scores labelled by MTurkers, respectively.
Refer to caption
Figure 8: Visualization of two samples of annotations collected in Stage I causal cloze tasks. They are collected progressively by feeding the response of a user as the input of another one. The black arrows indicate the annotation orders.
Refer to caption
Figure 9: Visualization of two samples of CEGs in CLEVRER-Humans. The green arrows represent causal relations and the red edges represent bidirectional irrelevance. We can see the rich causal relations among physical events presented in the CEGs.
Original figure
Figure 10: Effect of different causality thresholds on the binarized human causal relation. The x-axis is the ablation threshold (i.e., 4 means a score ≥\geq 4 represents a causal relation). The y-axis is the conditional probability.
Y = y1y_{1} Y = y2y_{2} Y = y1∧y2y_{1}\wedge y_{2} Y = y1∨y2y_{1}\vee y_{2} Y = y1⊕y2y_{1}\oplus y_{2}
P(X = Human | Y) 0.62 0.61 0.23 0.34 0.34
P(Y | X = Human) 0.96 0.54 0.29 0.62 0.33
Table 5: Comparison between different combinations of heuristics-generated causal labels and human labels, on a sampled subset of CLEVRER [7]. The entry P(X|Y) denotes the fraction of event relations that are annotated as causal by protocol X given that the relations are annotated as causal by protocol Y. y1,y2y_{1},y_{2} denote the existence of causal relations defined CLEVRER’s heuristic and Counterfactual causal relation, respectively.

A.2 Comparison between Heuristic and Human Causal Judgments

Participants suggest an average binary cutoff of 3.6; the authors choose four to favor stronger causal judgments. Figure 10 varies that cutoff, and Table 5 compares Boolean combinations of the two heuristic labelers. The authors conclude that these combinations remain poor substitutes for human labels. Some conditional-probability entries in Table 5 do not respect the expected monotonicity of the stated union operation, so the original table is retained as reported rather than used to derive new quantitative claims.

Appendix B Implementation Details

Implementation details specify the privileged trajectory inputs used for annotation generation and the video/text inputs used by evaluation baselines.

B.1 Stage II Implementation

Each object contributes a 24-dimensional state over 128 frames. Pairwise rules inspect segments where every coordinate displacement is under 0.5 m; sustained proximity and velocity changes distinguish moving together, approach, and collision. Pairs without detected events are discarded. Separate attention-GRU models use hidden widths 256 for single objects and 128 for pairs, decoder dropout 0.3/0.1, and learning rate 0.001. Post-processing rejects repeated words and incomplete verb arguments.

B.2 Baseline Implementation

Language-Only models.

The language-only model uses GloVe embeddings, a 512-wide LSTM, dropout 0.2, Adam at 0.0004, weight decay 0.00001, and batch size four. Model selection holds out 20% of training pairs.

CNN+LSTM.

CNN+LSTM samples 25 frames, extracts pretrained ResNet-50 features, and uses separate 512-wide video and question LSTMs with GloVe initialization. Batch size is 128; Adam learning rates are 0.00001 from scratch and 0.001 for fine-tuning. Twenty percent of training pairs supplies validation.

BERT+LSTM

CNN+BERT keeps the same visual encoder and freezes pretrained uncased BERT-base text weights. Inputs are padded or truncated to 32 tokens. Batch size, learning-rate settings, and validation fraction follow the preceding baseline. The appendix heading says BERT+LSTM, while its prose describes CNN+BERT.

ALOE.

ALOE uses released pretrained MONet object embeddings. Because the release lacks training code, the authors implement training themselves, use Adam with weight decay 0.001 after comparison with nearby values, and select models on a 5% training holdout.

B.3 Error analysis

Transferred program parsers map unfamiliar events such as slowing or stopping to incorrect collision operations. Learned models struggle with temporal and linguistic distinctions, such as what part of a trajectory “comes up” denotes. Pretraining may also preserve heuristic causal associations that people reject. These qualitative examples support several plausible failure sources, while the small dataset limits the strength of causal attribution among them.

Refer to caption
Figure 11: Examples of common prediction errors. The arrows in the image represent the moving direction of objects of interest in the video. Left: failure caused by nuances in human language. While the purple ball is constantly moving upwards coordinate-wise, humans understand the phrase "comes up" as more of the later part of the trajectory (after the purple ball collides with the brown ball). Therefore, machines cannot give a correct prediction. Right: failure in bridging the domain shift. Humans may consider the change in the trajectory to be minor and appears not to be a deciding factor of the outcome event, but the model predicts it as a cause following similar heuristics in CLEVRER.
Event Parsed program
The purple sphere slows down from the right. ["events", "objects", "purple", "filter_color", "sphere", "filter_shape", "unique", "filter_collision", "objects", "unique", "filter_color", "sphere", "filter_shape", "unique", "filter_collision", "unique"]
The red ball comes to a stop. ["events", "objects", "red", "filter_color", "unique", "filter_collision", "objects", "red", "filter_color", "unique", "filter_collision", "unique"]
The yellow cube comes from the right side at a fast speed. ["events", "objects", "yellow", "filter_color", "cube", "filter_shape", "unique", "filter_out", "unique"]
Table 6: Examples of errors produced by program parser. In the first row, the model cannot identify the event "slow down from the right" and gives incorrect parsing to find another object involved in a collision ("filter_collision"). In the second row, the model cannot represent the event "come to a stop" due to the expansion in vocabulary and gives an incorrect output ("filter_collision"). In the third row, the model mistakenly represents the enter event as the exit event ("filter_out") because the description is more complicated in CLEVRER-Humans. We follow the notation of programs as in NSDR and VRDP.

Appendix C Labeling Interface

MTurk interfaces provide replayable videos, highlighted objects, examples, and explicit response instructions. Cloze workers complete a seven-question comprehension quiz with two attempts. The authors report about $3,500 total compensation: U.S.-restricted tasks paid an estimated $7.7/hour, while later recruitment outside the U.S. brought the overall estimated average to $6.1/hour. Figures 12 and 13 preserve both complete example interfaces.

Refer to caption
Figure 12: Example of a causal cloze trial. The participants are asked to fill in the blank after watching a video. They can also select the checkboxes if they do not understand the prompt.
Refer to caption
Figure 13: Example of a CEG trial. The participants are first asked to select if the event descriptions are correct. If both correct, they are asked to label the level of causal relations between the descriptions. For each event pair, we provide bounding boxes for objects involved in the events for better annotation efficiency.

Appendix D Dataset Release

The dataset is released under CC0 with a structured data statement.

Short form data statement

The texts combine English crowd-written seeds and machine-generated descriptions; correctness and causal labels come from workers watching videos and reading event pairs.

Curation Nationale

Curation aims for interpretable, correct, diverse descriptions through cloze collection, trajectory-based generation, grammar/object filtering, verb balancing, and human event and relation judgments.

Language Variety

The source includes mainstream U.S. English, but detailed language-variety information is unavailable.

Speaker Demographic

Seed cloze writers were restricted to U.S. workers. The authors expect many native English speakers but do not provide measured participant demographics.

Annotator Demographic

Annotators must have at least 1,000 approved tasks. Batches contain 30 question pairs, usually from one or two videos; English comprehension and annotation quality are checked. Broader demographic estimates are referenced rather than measured for this sample.

Speech Situation

Descriptions are written for a general audience. Workers may replay the roughly five-second videos while completing prompts or judging event pairs.

Text Characteristics

Sentences describe one or more objects and their motion or interactions. Generated language is intended to resemble the seed-caption vocabulary and structure.

Recording Quality

The source marks recording quality as not applicable.

Other

No additional information is supplied under this field.

Provenance Appendix

All source videos come from CLEVRER.

D.1 Intended Use

The intended use is evaluating physical-scene and causal-language reasoning in short, restricted synthetic videos; it does not establish competence on real-world causal inference.

D.2 Maintenance Plan

The authors state plans for persistent project-site hosting, release versioning, and regular archival backups.

D.3 Quality Check

Three annotators provide overlapping event-correctness judgments for majority voting. Relation-label batches include an author-curated quality-control video selected from a 30-video set; submissions are accepted only when control answers sufficiently agree with the authors. This checks consistency with the chosen annotation standard rather than making subjective causal labels uniquely correct.

Concept demonstration

Open image ↗