At a glance
This study tests whether MiniMax-H3 can infer missing event information from complementary images, audio, and video. Human evaluation of 517 instances gives 41.97% overall success. Video-based decisions perform best (56.00%); audio-based disambiguation performs worst (27.40%). The benchmark tests the combined understanding-and-generation pipeline, so failures do not isolate reasoning from rendering errors.
1 Introduction
The benchmark deliberately withholds the answer from the text prompt. A correct generated video must recover it from the supplied observations. This probes cross-modal evidence use beyond reproducing an explicitly described scene. The dataset spans four reasoning scenarios and 29 subcategories; the results concern MiniMax-H3 rather than a comparison of competing models.
Related references: [21], [15], [37], [22], [26], [23], [12], [14].
| Evaluation Sets | # Examples | Input Modalities | Prompt Type | Multi Scene | Phys. Aaware | Reasoning |
|---|---|---|---|---|---|---|
| VBench (Huang et al., 2024) | 800 | T + I | Explicit | ✗ | ✗ | ✗ |
| TC-Bench (Feng et al., 2024) | 150 | T + I | Explicit | ✗ | ✗ | ✗ |
| VideoPhy (Bansal et al., 2025) | 688 | T + I | Explicit | ✗ | ✓ | ✗ |
| WorldModelBench (Li et al., 2026b) | 350 | T + I | Explicit | ✗ | ✓ | ✗ |
| Ours | 517 | T + I + A + V | Implicit | ✓ | ✓ | ✓ |
2 Related Work
Prior benchmarks cover generation quality, physical consistency, and reasoning expressed through video.
Video generation evaluation.
Video benchmarks measure perceptual quality, temporal structure, and physical consistency. Here the task is to infer the event that should occur from incomplete multimodal evidence.
Related references: [12], [45], [48], [8], [46], [44], [47], [2], [3], [17], [43], [42], [24], [31], [5], [16], [13].
Reasoning through video generation.
Earlier work uses generated sequences as solutions to visual, symbolic, and planning tasks. This evaluation distributes the required evidence across views and modalities instead of specifying the solution in text.
World-model evaluation.
World-model benchmarks test scene consistency, control, and state transitions. This study adds evidence-dependent generation: the observations determine the appropriate event or response.
3 Reasoning Evaluation with MiniMax-H3
The evaluation specifies four input configurations, constructs implicit prompts, and judges whether generated outcomes satisfy the corresponding semantic constraints.
3.1 What Do Omni-Models Enable?
Images supply layout, video supplies motion, and audio supplies event cues or spoken constraints. Their combination can make an otherwise ambiguous target inferable. Generated motion and state changes expose whether the model used that evidence.
3.2 Evaluation on Physical-World Reasoning Tasks
The four tasks assess cross-view geometry, audio-dependent interpretation, temporal decisions, and joint audiovisual reasoning. Text requests an operation while leaving the intended inference unspecified.
Equation 1
| (1) |
Multi-view Spatial Reasoning (MSR).
MSR combines multiple views of one scene. Success requires consistent entity correspondence, occlusion handling, and spatial relationships that no single view fully reveals.
Equation 2
| (2) |
Audio-based Disambiguation Reasoning (ADR).
ADR pairs an ambiguous image with a discriminating sound. The model must connect the sound to the correct object or event, such as identifying which differently made cup fell.
Equation 3
| (3) |
Video-based Decision Reasoning (VDR).
VDR asks for a continuation responding to the dynamics in a video prefix. For example, a ball entering a road can justify stopping before a following child is visible.
Equation 4
| (4) |
Audiovisual Integrated Reasoning (AVIR).
AVIR combines video with environmental audio or spoken constraints. The output must continue or correct the sequence using both sources; the audio need not be temporally aligned with the video.
Equation 5
| (5) |
3.3 Evaluation Data Construction and Human Annotation
Each example contains observations, an implicit prompt, and an annotated semantic target. Reviewers check that the target is inferable and can be assessed in a generated video.
Source Collection.
The authors combine real recordings and synthetic visual/audio sources, screening for perceptual quality and coherent task evidence.
Task-Specific Instance Construction.
Inputs are arranged so the required spatial relation, sound source, anticipated response, or audiovisual correction depends on the observations. Success allows multiple valid appearances and motions.
Implicit Prompt Construction.
Language models propose prompt variants; experts edit them to avoid revealing the target or introducing unsupported assumptions.
Iterative Expert Review.
Ten review rounds examine scene complexity, required inference, alignment of the observations, and prompt implicitness. Accepted examples cover 517 cases in four domains and 29 subdomains.
4 Experiments
Experiments measure overall success, differences across tasks, and recurring failures.
4.1 Evaluation Data and Protocol
The protocol uses task-specific human judgments rather than a single reference video.
Data sources and coverage.
The dataset has 200 MSR, 100 VDR, 146 ADR, and 71 AVIR instances. Sources include robot-manipulation views, general videos, sound datasets, and reviewed synthetic material. Scenario composition differs, which limits direct modality comparisons.
Related references: [1], [38], [25], [39], [10], [41], [9], [28].
Human evaluation.
Three experts independently assess outputs and cross-check their judgments. Realistic appearance alone is insufficient: the generated event must satisfy the evidence-dependent task requirement.
Metric and scope.
SR is the percentage of examples meeting the binary task criterion. Overall results are weighted by example count.
Equation 6
| (6) |
4.2 Quantitative Results
The original table gives every subcategory count and success rate.
| MSR | VDR | ADR | AVIR | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Subcategory | SR (%) | Subcategory | SR (%) | Subcategory | SR (%) | Subcategory | SR (%) | ||||
| Deformation | 34 | 41.20 | Humans | 34 | 55.88 | Vocalizations | 16 | 31.30 | Activities | 14 | 57.14 |
| Cleaning | 27 | 33.30 | Cartoons | 5 | 20.00 | Alerts | 26 | 23.10 | Animation | 6 | 33.33 |
| Articulation | 26 | 34.60 | Animals | 11 | 63.64 | Music | 16 | 25.00 | Making | 21 | 52.38 |
| Arrangement | 23 | 47.80 | Traffic | 9 | 100.00 | Machinery | 34 | 17.60 | Animals | 10 | 30.00 |
| Transport | 23 | 34.80 | Physics | 10 | 50.00 | Contact | 32 | 21.90 | Cues | 20 | 50.00 |
| Loading | 20 | 50.00 | Memory | 6 | 66.67 | Nature | 22 | 54.50 | |||
| Assembly | 20 | 50.00 | Puzzles | 6 | 16.67 | ||||||
| Pouring | 12 | 58.30 | Synthetic | 19 | 52.63 | ||||||
| Threading | 8 | 75.00 | |||||||||
| Control | 7 | 42.90 | |||||||||
| Overall | 200 | 43.50 | Overall | 100 | 56.00 | Overall | 146 | 27.40 | Overall | 71 | 47.89 |
Overall success rate across all 517 samples: 41.97%.
Overall reliability remains limited.
Overall success is 41.97%. Scenario scores are 56.00% for VDR, 47.89% for AVIR, 43.50% for MSR, and 27.40% for ADR. The 28.60-point VDR–ADR gap reflects different tasks and datasets; it does not establish an isolated advantage of video over audio.
Results across scenarios.
Performance varies within scenarios: spatial threading reaches 75.00%, while several manipulation tasks remain near one third. Video puzzles reach 16.67%. Audio grounding remains difficult, especially for machinery, contact, and alerts.
4.3 Qualitative Analysis
Successful examples include coordinated manipulation stages, plausible short-term motion, and visible events guided by sounds. The figures show both inputs and generated frames.
4.4 Failure Cases
Failures include binding evidence to the wrong object, omitting the required interaction, violating geometry or contact dynamics, and losing previously established state. Plausible-looking videos can therefore fail the actual reasoning task.
4.5 Discussion
The evaluation measures understanding, cross-modal integration, and generation together. Controlled removal or replacement of individual modalities would help distinguish their contributions and failure causes.
5 Conclusion
Implicit multimodal prompts expose a substantial gap between accepting different input types and using them reliably. Broader scenarios and reliable automated evaluation are proposed next steps.