Literature
Display
← All papers

Towards Fine-Grained Video Question Answering

Wei Dai · Alan Luo · Zane Durante · Debadutta Dash · Arnold Milstein · Kevin Schulman · Ehsan Adeli · Li Fei-Fei

Preprint posted · arXiv (preprint) ↗

Computer vision9 min text · CondensedOriginal paper ↗
Contents

Overview

MOMA-QA pairs entity-specific video questions with scene graphs and time intervals. This edition includes the main dataset/model study and all four supplementary sections on graph prediction, answer scoring, training, and attention-mask ablation.

At a glance

MOMA-QA tests video question answering with explicit time intervals, object relationships, and queries about a highlighted person. It derives 300,791 question–answer pairs from MOMA’s human scene graphs, including videos augmented with actor outlines.

SGVLM predicts scene relationships, retrieves relevant frames, and passes separate scene and visual representations to a language model. Fine-tuning improves overall MOMA-QA accuracy from SeViLA’s 77.19% to 79.66%. Results support the value of structured supervision, while zero-shot relationship accuracy remains only 13.03% and gains are not uniform across external benchmarks.

1 Introduction

A correct video answer can require identifying both when an event occurs and which of several similar people or objects the question concerns. Many earlier datasets emphasize short clips, coarse actions, or temporal reasoning without human scene-relationship annotations.

MOMA-QA builds on MOMA to supply question-specific time intervals and frame-level scene graphs. Relationship questions comprise 71.6% of the dataset. Visual outlines identify the target actor in crowded scenes without giving the answering model numerical box coordinates.

The accompanying SGVLM architecture makes spatial relationships and frame selection explicit intermediate representations. These can help diagnose mistakes; the examples do not establish that a predicted graph is a complete causal explanation of a language model’s answer.

Refer to caption
Figure 1: Visualizations of Sample Questions from MOMA-QA. We illustrate the three distinct types of questions in our dataset, each representing a different category for video question answering. All questions in our dataset are generated from a human-annotated spatio-temporal scene graph (shown on the right). The node of interest for the relationship and motion questions is colored red in the scene graph and outlined in the video.
Category Question Format Answer Format Example Question Example Answer
Relationship When [Ci][C_{i}], what is [Vs][V_{s}] [Ei][E_{i}]? [Vt][V_{t}] When the players are passing the basketball for the 3rd time, Who is the outlined person looking at? Basketball Player
Motion When [Ci][C_{i}], is [Vt][V_{t}] standing, walking or running? [Att(VtV_{t})] When the players are passing the basketball for the 3rd time, is the outlined person standing, walking, or running? Running
Description When [Ci][C_{i}], how many [Identity] are in the scene? [Id. Count] When the players are passing the basketball for the 3rd time, how many basketball players are in the scene? 7
Table 1: General Structure of Generated Questions. CiC_{i} denotes the description of a particular sub-activity. Additionally, VsV_{s} and VtV_{t} denotes the name of a source and a target node from the sub-activity. EiE_{i} represents the description of a relationship connecting VsV_{s} and VtV_{t}.

2 Related Works

Earlier benchmarks span short-clip QA, temporal reasoning, long-video understanding, and entity grounding. The retained comparison table distinguishes video length, answer format, boxes, and scene annotations. Its temporal-localization checkmarks should not be read as equivalent annotation quality across datasets.

Graph-based models encode relations among objects, while language-model approaches such as SeViLA select a small number of informative frames to meet input limits. SGVLM combines both approaches, using Neural Motifs for scene structure and UniVTG for temporal grounding.

Dataset Video Source #Videos
#QA
Pairs
Average
Length (s)
Open
Ended
Temporal
Localization
Bounding Box
Augmentation
Scene Graph
Annotation
MSVD-QA [52] MSVD 1,9701,970 50,50550,505 1010 ✓ ✗ ✗ ✗
MSRVTT-QA [52] MSRVTT 10,00010,000 243,690243,690 1515 ✓ ✗ ✗ ✗
TGIF-QA [11] TGIF 71,74171,741 165,165165,165 33 ✓ ✓ ✗ ✗
TVQA [16] TV Show 21,79321,793 152,545152,545 7676 ✗ ✓ ✗ ✗
ActivityNet-QA [57] ActivityNet 5,8005,800 58,00058,000 180180 ✓ ✓ ✗ ✗
Social-IQ [58] YouTube 1,2501,250 7,5007,500 6060 ✗ ✗ ✗ ✗
EgoSchema [27] Ego4D 5,0635,063 5,0635,063 180180 ✗ ✗ ✗ ✗
NExT-QA [51] YFCC-100M 5,4405,440 52,04452,044 4444 ✓ ✓ ✗ ✗
STAR [49] Charades 23,01323,013 60,20660,206 1111 ✗ ✓ ✗ ○∗\bigcirc^{*}
TVQA+ [17] TV Show 4,1984,198 29,38329,383 6161 ✓ ✓ ✓ ✗
MOMA-QA (Raw only) MOMA 1,4121,412 83,22383,223 376376 ✓ ✓ ✗ ✓
MOMA-QA (Aug.) MOMA 27,58627,586 300,791300,791 144144 ✓ ✓ ✓ ✓
Table 2: Dataset Comparisons. Our proposed dataset sets a new benchmark for open-ended, long VideoQA by providing extensive human annotations and a large number of QA pairs. Raw only: Includes raw videos only. Aug.: Includes both raw videos and box-augmented videos. * The graph annotations provided by the STAR dataset are automatically generated, while MOMA-QA’s are human annotated.

3 MOMA-QA Dataset

Dataset construction uses MOMA’s existing video-wise training, validation, and test split. Questions and outlined variants derive from the same source videos; the augmented-video count represents edited versions rather than that many independent recordings.

3.1 Annotations

Each annotated frame has a graph whose nodes are entities with boxes and attributes, and whose edges describe relationships. Consecutive frames belong to labeled sub-activities with start and end times. These annotations support both question generation and intermediate scene/temporal supervision.

3.2 Bounding Box Augmentations

Bounding-box augmentation edits a video to outline the actor named by a question. Replacing an identity such as “basketball player” with “outlined person” reduces linguistic hints and resolves which actor is meant. The models see the modified pixels, not the ground-truth box coordinates as an additional QA input.

3.3 Questions

Templates generate relationship, motion, and descriptive questions tied to a particular sub-activity, as retained in Table 1. The paper reports manual verification of each question and wording adjustments. Every question has temporal-interval and scene-graph annotations; the paper does not quantify annotator agreement or template-related answer biases.

3.4 Dataset Statistics

The dataset contains 300,791 QA pairs from 147 hours of original video plus 956 hours of outlined variants. Table 2 lists 1,412 raw videos and 27,586 videos including augmentation, with mean durations of 376 and 144 seconds respectively.

Questions are 71.6% relationship, 24.2% motion, and 4.21% description, with a median length of 20 words. The text reports 4,045 scenes with more than ten actors and 72.3% of questions using outlined videos. These are separate scene, video, and question counts; the dataset is strongly weighted toward relationships.

Refer to caption
Figure 2: Statistics of MOMA-QA. (a) The distribution of the number of actors. (b) The percentage of each question type in MOMA-QA. (c) The distribution of question lengths in MOMA-QA in words. (d) The percentage of box-augmented questions.

4 Method

The model has five components. An EVA-02 vision transformer produces image patches and object proposals. A Neural Motifs-style predictor uses bidirectional LSTMs to build object and edge contexts, retaining the highest-scoring relation features. The vision and graph components receive Visual Genome pretraining; dataset-specific tuning is described in the supplement.

For retrieval, mean-pooled frame and scene features are separately projected, combined with question tokens, and given positional and type embeddings. A transformer alignment branch predicts whether each frame belongs to the relevant interval. Figure 4 masks interactions between frame tokens and scene tokens while leaving within-modality and question interactions unmasked. The accompanying prose instead describes visual tokens interacting exclusively with questions; the diagram and that wording are not fully consistent.

A second branch sums frame–question and scene–question cosine similarities. Intra-video contrastive training favors a positive frame over lower-saliency negatives, while inter-video training uses other batch videos as negatives. The combined objective adds these terms to foreground binary cross-entropy. Learned weights combine foreground probability and saliency to rank frames; only the top-ranked frames proceed.

When interval annotations exist, they supervise the retriever directly. Otherwise, Equation 8 builds pseudo-labels from answer correctness and the frame-selection threshold: retain the selection for a correct answer and reverse it for a wrong one. This is heuristic supervision from QA outcomes, not independently verified temporal grounding.

Separate BLIP-2 Q-Formers process scene and frame features, project them into language-model space, and concatenate them with the question. The paper does not specify all critical choices, including the exact language-model checkpoint, selected-frame count, loss weights, and threshold settings.

The original equations are preserved. Some notation is inconsistent: the feature-pooling sums omit their summation index on the feature term; the contrastive paragraph swaps the frame/scene variable names; the stated intra-video negative set restricts indices to those before the positive frame. These printed details should not be silently replaced by assumed implementation behavior.

Refer to caption
Figure 3: Model Architecture of SGVLM. The model employs a frame encoder to extract frame embeddings from the input video, which are subsequently used by a Scene Graph (SG) Predictor to generate scene graph embeddings. These embeddings are then concatenated with the frame features. The combination, along with question embeddings, is processed by a transformer encoder in the Frame Localizer to produce similarity scores for identifying relevant frames. Key frame features are then processed by Frame Q-Former and SG Q-Former to align with the language query and scene graph features. An LLM finally generates answers using a structured representation of scene graph and frame data, merged with the natural language question.
Refer to caption
Figure 4: Self-Attention Mask of the Transformer Encoder in Frame Localizer. To separate frame and scene graph tokens, we mask out portions of the input with −∞-\infty.
Equations 1–8
𝐱i=1|𝐗𝐢|​∑j=1|𝐗𝐢|𝐗𝐢​𝐖x​s,𝐬i=1|𝐒𝐢|​∑j=1|𝐒𝐢|𝐒𝐢​𝐖s​s,\displaystyle\mathbf{x}_{i}=\frac{1}{|\mathbf{X_{i}}|}\sum_{j=1}^{|\mathbf{X_{i}}|}\mathbf{X_{i}}\mathbf{W}_{xs},\quad\mathbf{s}_{i}=\frac{1}{|\mathbf{S_{i}}|}\sum_{j=1}^{|\mathbf{S_{i}}|}\mathbf{S_{i}}\mathbf{W}_{ss},
𝐤i,m\displaystyle\mathbf{k}_{i,m} =softmax​(𝐖Qi,m​𝐙i−1​(𝐖Ki,m​𝐙i−1)Tdki+𝐌),\displaystyle=\text{softmax}\left(\frac{\mathbf{W}_{Q}^{i,m}\mathbf{Z}_{i-1}(\mathbf{W}_{K}^{i,m}\mathbf{Z}_{i-1})^{T}}{\sqrt{d_{k}^{i}}}+\mathbf{M}\right),
𝐡i,m\displaystyle\mathbf{h}_{i,m} =𝐤i,m​𝐖Vi,m​𝐙i−1,\displaystyle=\mathbf{k}_{i,m}\mathbf{W}_{V}^{i,m}\mathbf{Z}_{i-1},
𝐙i\displaystyle\mathbf{Z}_{i} =(||m=1M𝐡i,m)𝐖Oi,\displaystyle=(||_{m=1}^{M}\mathbf{h}_{i,m})\mathbf{W}^{i}_{O},
𝐟^=σ⁡(Conv​(𝐙k′)),\mathbf{\hat{f}}=\sigma(\text{Conv}(\mathbf{Z}^{\prime}_{k})), (1)
ℒa=∑i=1n−(filogf^i+(1−fi)log(1−f^i)),\mathcal{L}_{a}=\sum_{i=1}^{n}-\left(f_{i}\text{log}~\hat{f}_{i}+(1-f_{i})\text{log}(1-\hat{f}_{i})\right), (2)
s^c,i=𝐱iT​𝐐′‖𝐱i‖2​‖𝐐′‖2+𝐬iT​𝐐′‖𝐬i‖2​‖𝐐′‖2.\hat{s}_{c,i}=\frac{\mathbf{x}_{i}^{T}\mathbf{Q}^{\prime}}{||\mathbf{x}_{i}||_{2}||\mathbf{Q}^{\prime}||_{2}}+\frac{\mathbf{s}_{i}^{T}\mathbf{Q}^{\prime}}{||\mathbf{s}_{i}||_{2}||\mathbf{Q}^{\prime}||_{2}}. (3)
ℒsintra=−log⁡exp⁡(s^p/τ)exp⁡(s^p/τ)+∑j∈Nexp⁡(s^j/τ),\mathcal{L}_{s}^{\text{intra}}=-\log\frac{\exp(\hat{s}_{p}/\tau)}{\exp(\hat{s}_{p}/\tau)+\sum_{j\in N}\exp(\hat{s}_{j}/\tau)}, (4)
ℒsinter=−log⁡exp⁡(s^p/τ)∑k∈Bexp⁡(s^pk/τ).\mathcal{L}_{s}^{\text{inter}}=-\log\frac{\exp(\hat{s}_{p}/\tau)}{\sum_{k\in B}\exp(\hat{s}_{p}^{k}/\tau)}. (5)
ℒ=λa​ℒa+λi​n​t​r​a​ℒsintra+λi​n​t​e​r​ℒsinter,\mathcal{L}=\lambda_{a}\mathcal{L}_{a}+\lambda_{intra}\mathcal{L}_{s}^{\text{intra}}+\lambda_{inter}\mathcal{L}_{s}^{\text{inter}}, (6)
r^i=wf​f^i+ws​s^i,\hat{r}_{i}=w_{f}\hat{f}_{i}+w_{s}\hat{s}_{i}, (7)
fi′,si′={1,1if ​(y=y^∧r^i>rθ) ∨(y≠y^∧r^i<rθ)0,−1Otherwise.f^{\prime}_{i},s^{\prime}_{i}=\begin{cases}1,1&\text{if }(y=\hat{y}\wedge\hat{r}_{i}>r_{\theta})\\ &\text{~~~}\vee(y\neq\hat{y}\wedge\hat{r}_{i}<r_{\theta})\\ 0,-1&\text{Otherwise}\end{cases}. (8)
Model Description Relationship Action Total
Accuracy WUPS@0.9 Accuracy WUPS@0.9 Accuracy WUPS@0.9 Accuracy WUPS@0.9
InternVideo [47] 0.000.00 0.00000.0000 0.070.07 0.00180.0018 0.000.00 0.00000.0000 0.050.05 0.00130.0013
mPLUG-2 [53] 0.000.00 0.50840.5084 9.839.83 0.28910.2891 0.000.00 0.00000.0000 6.906.90 0.22220.2222
BLIP-2 [21] 55.19¯\underline{55.19} 0.55190.5519 12.3012.30 0.19570.1957 62.1062.10 0.62100.6210 26.4526.45 0.31610.3161
SeViLa [56] 53.2253.22 0.6027 12.99¯\underline{12.99} 0.5045 64.66¯\underline{64.66} 0.8977¯\underline{0.8977} 27.44¯\underline{27.44} 0.6023
SGVLM 58.69 0.5869¯\underline{0.5869} 13.03 0.4663¯\underline{0.4663} 65.43 0.9174 27.94 0.5828¯\underline{0.5828}
Table 3: Zero-Shot Performance Comparison of SGVLM with Baselines on MOMA-QA Dataset. Our method outperforms, or performs on-par with existing methods in the zero-shot setting.
Model Description Relationship Action Total
Accuracy WUPS@0.9 Accuracy WUPS@0.9 Accuracy WUPS@0.9 Accuracy WUPS@0.9
InternVideo [47] 42.1542.15 0.42150.4215 36.7736.77 0.39800.3980 71.1271.12 0.71120.7112 45.9145.91 0.48040.4804
mPLUG-2 [53] 0.940.94 0.00940.0094 47.0047.00 0.70840.7084 0.390.39 0.00560.0056 33.1233.12 0.49900.4990
BLIP-2 [21] 62.9062.90 0.62900.6290 77.3477.34 0.78640.7864 72.5572.55 0.92860.9286 75.7375.73 0.82780.8278
Sevila [56] 63.6063.60 0.63600.6360 78.9278.92 0.82180.8218 74.6074.60 0.94530.9453 77.1977.19 0.84420.8442
SGVLMN​o​L​o​c\text{SGVLM}_{NoLoc} 67.01 0.6701 79.25¯\underline{79.25} 0.82160.8216 74.7174.71 0.95600.9560 77.60¯\underline{77.60} 0.84820.8482
SGVLMN​o​S​G\text{SGVLM}_{NoSG} 65.3365.33 0.65330.6533 78.7378.73 0.8263¯\underline{0.8263} 76.33¯\underline{76.33} 0.9763¯\underline{0.9763} 77.5577.55 0.8558¯\underline{0.8558}
SGVLM 66.64¯\underline{66.64} 0.6664¯\underline{0.6664} 81.36 0.8435 77.06 0.9771 79.66 0.8688
Table 4: Fine-tuned Performance Comparison of SGVLM with Baselines on MOMA-QA Dataset. SGVLMN​o​L​o​c\text{SGVLM}_{NoLoc}: An ablation of SGVLM where the frame localizer is removed and replaced with uniform frame sampling. SGVLMN​o​S​G\text{SGVLM}_{NoSG}: An ablation of SGVLM where the scene graph predictor is removed, and the model inferences solely on the frame embeddings.
Model Causal Temporal Descriptive Average
HGA [12] 46.846.8 52.152.1 59.359.3 50.450.4
All-in-One [46] 48.048.0 48.648.6 63.263.2 50.650.6
Just Ask [54] 49.649.6 51.451.4 63.163.1 52.352.3
MIST [7] 54.654.6 56.656.6 66.966.9 57.257.2
HiTeA [55] 62.462.4 58.358.3 75.675.6 63.163.1
InternVideo [47] 62.562.5 58.558.5 75.875.8 63.263.2
BLIP-2 [21] 72.972.9 68.1¯\underline{68.1} 81.281.2 72.672.6
SeViLA [56] 74.2¯\underline{74.2} 69.4 81.3¯\underline{81.3} 73.8¯\underline{73.8}
SGVLM 75.2 66.366.3 83.4 74.3
Table 5: Comparison of SGVLM with SoTA on NExT-QA. We achieve comparable or slightly superior performance to existing methods on the NExT-QA dataset. This is noteworthy considering NExT-QA lacks explicit relationship and scene-graph oriented questions, underscoring the versatility of our approach.

5 Experiments

Experiments test MOMA-QA answering, multiple-choice NExT-QA, and QVHighlights retrieval/highlight detection. The last evaluation trains and measures the retriever alone, so its scores do not describe end-to-end answer quality.

Moment Retrieval HD
R11 mAP ≥\geq Very Good
Model @0.50.5 @0.70.7 @0.50.5 @0.750.75 Avg. mAP HIT@11
BeautyThumb [39] −- −- −- −- −- 14.3614.36 20.8820.88
DVSE [24] −- −- −- −- −- 18.7518.75 21.7921.79
MCN [2] 11.4111.41 2.722.72 24.9424.94 8.228.22 10.6710.67 −- −-
CAL [4] 25.4925.49 11.5411.54 23.4023.40 7.657.65 9.899.89 −- −-
CLIP [31] 16.8816.88 5.195.19 18.1118.11 7.07.0 7.677.67 31.3031.30 61.0461.04
XML [18] 41.8341.83 30.3530.35 44.6344.63 31.7331.73 32.1432.14 34.4934.49 55.2555.25
XML+ [19] 46.6946.69 33.4633.46 47.8947.89 34.6734.67 34.9034.90 35.3835.38 55.0655.06
MDETR [19] 52.8952.89 33.0233.02 54.8254.82 29.4029.40 30.7330.73 35.6935.69 55.6055.60
UniVTG [22] 58.8658.86 40.8640.86 57.6057.60 35.5935.59 35.4735.47 38.2038.20 60.9660.96
UMT [25] 56.2356.23 41.1841.18 53.8353.83 37.0137.01 36.1236.12 38.1838.18 59.9959.99
QD-DETR [28] 62.40¯\underline{62.40} 44.98¯\underline{44.98} 62.52 39.88¯\underline{39.88} 39.86 38.94¯\underline{38.94} 62.40
SeViLA [56] 54.5054.50 36.5036.50 −- −- 32.3032.30 −- −-
SGVLM 63.36 46.30 62.47¯\underline{62.47} 42.00 39.82¯\underline{39.82} 39.17 62.26¯\underline{62.26}
Table 6: Moment Retrieval and Highlight Detection Results on QVHighlights Test Split. We only include models not trained on additional video retrieval datasets (no extra training data). SGVLM (ours) and SeViLA are the only two VideoQA models.

5.1 Dataset & Metrics

MOMA-QA reports accuracy and WUPS@0.9. The printed accuracy is the average fraction of ground-truth answer tokens matching predicted tokens at the same position, not strict whole-answer exact match; the summation stops at the shorter answer length. Additional predicted tokens beyond that length are not explicitly penalized in Equation 9.

WUPS gives graded semantic credit and is defined in Appendix B. NExT-QA reports categorical and overall multiple-choice accuracy. QVHighlights reports moment Recall@1 at overlap thresholds, mean average precision, and highlight mAP/HIT@1. These different metrics should not be compared as one common accuracy scale.

Equation 9
acc=1|𝐐|∑𝐐1|q|∑i=1m​i​n​(|q^|,|q|)𝐈[q^i=qi].acc=\frac{1}{|\mathbf{Q}|}\sum_{\mathbf{Q}}\frac{1}{|q|}\sum_{i=1}^{min(|\hat{q}|,|q|)}\mathbf{I}[\hat{q}_{i}=q_{i}]. (9)

5.2 Experimental Setup

MOMA-QA comparisons include InternVideo, mPLUG-2, BLIP-2, and SeViLA in both zero-shot and fine-tuned settings. The supplement records pretraining and training hardware, but not a complete common compute budget, hyperparameter search, or repeated-run uncertainty.

6 Results & Discussions

The original result tables and qualitative examples retain both improvements and failures, with separate evidence for QA, temporal retrieval, and component ablations.

Refer to caption
Figure 5: Visualization Results of SGVLM with Previous SoTA (SeViLA) on MOMA-QA. Left: An example where SGVLM makes the correct prediction while SeViLA fails. Right: An example where both our model and SeViLA produce incorrect answers. We magnify the part from the frame that is relevant to the question for better readability.

6.1 Zero Shot Results

SGVLM has the highest reported zero-shot total accuracy, 27.94%, narrowly above SeViLA’s 27.44%. SeViLA has the higher total WUPS, 0.6023 versus 0.5828. Both struggle on relationship questions: SGVLM reaches 13.03% accuracy, while SeViLA’s relationship WUPS is higher at 0.5045 versus 0.4663.

The source prose’s 13.03%/0.5045 maxima refer to the relationship category, not all questions. Here zero-shot means no MOMA-QA fine-tuning: SGVLM already trains its graph predictor on Visual Genome, retriever on QVHighlights, and QA model on NExT-QA.

6.2 Fine-tuned VideoQA Results

After MOMA-QA fine-tuning, SGVLM reaches 79.66% total accuracy and 0.8688 WUPS, versus SeViLA’s 77.19% and 0.8442. Relationship accuracy is 81.36% versus 78.92%; description accuracy is 66.64% versus 63.60%, a 3.04 percentage-point gain.

Removing the localizer reduces total accuracy to 77.60%; removing scene graphs gives 77.55%. The no-localizer variant has slightly better description accuracy than the full model, 67.01% versus 66.64%, so each component is not uniformly beneficial.

On NExT-QA, SGVLM averages 74.3% versus SeViLA’s 73.8%. Causal and descriptive scores improve to 75.2% and 83.4%, but temporal accuracy falls to 66.3% from 69.4%. No confidence intervals or repeated-seed variation are reported for these modest aggregate gains.

6.3 Frame Localization Results

On the QVHighlights hidden test set, SGVLM leads the listed models on moment R1@0.5 and R1@0.7, at 63.36 and 46.30, and mAP@0.75, at 42.00. QD-DETR remains slightly higher on mAP@0.5, 62.52 versus 62.47, and average mAP, 39.86 versus 39.82.

For highlight detection, SGVLM has the highest listed mAP at 39.17, but HIT@1 is 62.26 versus QD-DETR’s 62.40. This reverses the rankings described in the source prose. The table uses mAP@0.75 rather than the prose’s @0.7. Its “no extra training data” qualifier concerns additional video-retrieval datasets, not the absence of all visual pretraining.

6.4 Qualitative Analysis

In the table-tennis example, SGVLM identifies a score table through the relation “match official behind score table,” while SeViLA answers table-tennis table. In the security example, both methods miss a cabinet; the cabinet’s absence from SGVLM’s predicted graph suggests a recognition or occlusion failure.

These intermediate graphs make concrete errors inspectable. The two selected examples do not isolate scene graphs as the cause of success, quantify interpretability, or establish robustness to all occlusions.

7 Conclusion

MOMA-QA provides unusually detailed supervision for entity-specific video questions, and SGVLM uses scene representations and temporal retrieval to improve aggregate benchmark results. Remaining limitations include low zero-shot relationship accuracy, dependence on predicted objects and relationships, template/augmentation effects, and incomplete reproduction settings. The paper does not demonstrate deployment performance, inference latency, or broad real-world transfer.

Appendix A Details of Scene Graph Predictor.

The scene predictor encodes each object’s visual feature and class probabilities with a bidirectional LSTM. An autoregressive object decoder and another bidirectional LSTM/MLP produce edge contexts. Pairwise projected object contexts form relation features, with softmax scores used to keep the top-ranked relationships.

The supplement says the predictor is pretrained and frozen during QA training, while its fine-tuning subsection says the predictor and backbone are tuned on MOMA-QA. A staged tuning-then-freezing interpretation is plausible, but the exact schedule is not specified consistently. The original formulas remain available below.

Equations 10
𝐂=biLSTM​([𝐟i;𝐖c​t​x​𝐩i]i=1,…,n)\mathbf{C}=\text{biLSTM}([\mathbf{f}_{i};\mathbf{W}_{ctx}\mathbf{p}_{i}]_{i=1,\dots,n}) (10)
𝐨^i\displaystyle\mathbf{\hat{o}}_{i} =argmax​(𝐖o​LSTM​([𝐜i;𝐨^i−1]))\displaystyle=\text{argmax}(\mathbf{W}_{o}\text{LSTM}([\mathbf{c}_{i};\mathbf{\hat{o}}_{i-1}]))
𝐃\displaystyle\mathbf{D} =MLP​(biLSTM​([𝐜i;𝐖d​𝐨^i−1]))\displaystyle=\text{MLP}(\text{biLSTM}([\mathbf{c}_{i};\mathbf{W}_{d}\mathbf{\hat{o}}_{i-1}]))
𝐬i,j=(𝐖h​𝐝i)​(𝐖t​𝐝j)\displaystyle\mathbf{s}_{i,j}=(\mathbf{W}_{h}\mathbf{d}_{i})(\mathbf{W}_{t}\mathbf{d}_{j})
Pr​(xi→j)=softmax​(𝐖r​𝐬i,j)\displaystyle\text{Pr}(x_{i\rightarrow j})=\text{softmax}(\mathbf{W}_{r}\mathbf{s}_{i,j})

Appendix B Calculation of WUPS@0.9

WUPS compares words in both directions, taking each word’s best semantic match, multiplying within each answer, and retaining the smaller directional score. Similarities below 0.9 are downweighted by a factor of ten. The appendix preserves this expression but does not explicitly define the underlying similarity function W or the symbol A appearing in one product; tokenization and semantic-resource details are also absent.

Equations
WUPS=1|Q|∑q∈Qmin{∏qi∈Amaxq^jWγ(qi,q^j),\displaystyle WUPS=\frac{1}{|Q|}\sum_{q\in Q}\min\{\prod_{q_{i}\in A}\max_{\hat{q}_{j}}W_{\gamma}(q_{i},\hat{q}_{j}),
∏q^imaxqjWγ(q^i,qj)}\displaystyle\prod_{\hat{q}_{i}}\max_{q_{j}}W_{\gamma}(\hat{q}_{i},q_{j})\}
Wγ​(qi,q^j)={W⁡(qi,q^j)if ​W​(qi,q^j)≥γ0.1​W​(qi,q^j)if ​W​(qi,q^j)<γ\displaystyle W_{\gamma}(q_{i},\hat{q}_{j})=\begin{cases}W(q_{i},\hat{q}_{j})&\text{if }W(q_{i},\hat{q}_{j})\geq\gamma\\ 0.1W(q_{i},\hat{q}_{j})&\text{if }W(q_{i},\hat{q}_{j})<\gamma\end{cases}

Appendix C Experimental Setup

The supplement distinguishes transfer before MOMA-QA evaluation, direct dataset fine-tuning, and separate public-benchmark training.

C.1 Zero Shot

Baselines use their selected pretrained weights. Closed-vocabulary models map unavailable answers to the nearest vocabulary word. SGVLM trains its scene predictor on Visual Genome, localizer on QVHighlights, and full QA model on NExT-QA before MOMA-QA testing.

The appendix cites reference 50 for Visual Genome, although the dataset’s actual bibliography entry is reference 15. The intended dataset is clear from the main text; both original references are retained.

C.2 Fine-tuned

Each model trains for at most five epochs on one node with four NVIDIA A6000 GPUs, using the model’s default hyperparameters. SGVLM starts from the zero-shot configuration, tunes its visual/graph components on MOMA-QA, and tunes the localizer during full QA training. A shared epoch limit does not establish equal compute or tuning effort.

C.3 Experiments on Public Datasets

For NExT-QA, the paper uses multiple-choice evaluation and describes choosing among the highest-probability answer letters A–D. The exact stated candidate set is retained; the source does not further reconcile it with alternative benchmark configurations.

On QVHighlights, only the retriever is trained and evaluated. Hyperparameters are selected using validation data, then training and validation are combined for the reported hidden-test evaluation. Appendix D’s validation ablation is a different comparison from Table 6’s test results.

Appendix D Effect of Attention Masks

On QVHighlights validation, adding the attention mask improves all seven reported metrics: R1@0.5 rises 63.81→64.65, R1@0.7 47.35→48.06, and mAP@0.5 62.21→63.12. The largest listed gain is 0.91 percentage points. This ablation supports the mask in the tested setup; no run-to-run uncertainty is provided.

Moment Retrieval HD
R11 mAP ≥\geq Very Good
Model @0.50.5 @0.70.7 @0.50.5 @0.750.75 Avg. mAP HIT@11
Without Mask 63.8163.81 47.3547.35 62.2162.21 42.3242.32 40.6040.60 39.6939.69 63.5563.55
With Mask 64.65 48.06 63.12 43.19 41.13 40.17 64.19
Table 7: Ablation Results on QVHighlights Validation Split. The best in each column is bolded.
Concept demonstration

Open image ↗