Literature
Display
← All papers

VideoJev: System One Model for Video Understanding and Interaction

Yuhan Zhu · Xiangyu Zeng · Qingyi Si · Chenxu Yang · Yifei Li · Luozheng Qin · Junhao Zhou · Dingyu Yao · Jun Zhang · Chuanyu Qin · Zhiqiu Zhang · Yuandong Yang · Yuqi Xu · Tianyu Zhang · Hongcheng Guo · Jiaqi Wang · Nan Duan · Limin Wang

Preprint posted · arXiv (preprint) ↗

Contents

Overview

VideoJev turns a pretrained video-language model into a shared scorer for temporal intervals and answer options. The overview compares benchmark quality with measured forward latency.

Figure 1 : VideoJev: structured visual decisions with shared computation. Top: temporal intervals and textual options share a candidate-scoring interface; Cached inference reuses the video prefix for arriving queries, while Joint inference scores each query group in one forward pass. Bottom: prediction quality versus model forward latency at both model scales, using the results in Tables 1 and 2 . Left: mean grounding mIoU across three benchmarks and per-query latency on full Charades. Right: mean Video QA score across four benchmarks and per-question latency on full LVBench.
Figure 1: VideoJev: structured visual decisions with shared computation. Top: temporal intervals and textual options share a candidate-scoring interface; Cached inference reuses the video prefix for arriving queries, while Joint inference scores each query group in one forward pass. Bottom: prediction quality versus model forward latency at both model scales, using the results in Tables 1 and 2. Left: mean grounding mIoU across three benchmarks and per-query latency on full Charades. Right: mean Video QA score across four benchmarks and per-question latency on full LVBench.

At a glance

VideoJev replaces autoregressive answers with direct candidate scores for temporal grounding and multiple-choice video QA. A shared video prefix supports independent arriving queries (Cached inference) or several queries in one forward pass (Joint inference).

Across three grounding benchmarks, its 4B and 35B-A3B models improve mean mIoU over their Instruct backbones by 6.7 and 10.8 points. On four QA benchmarks they approach Thinking-mode quality at approximately 100–129× lower measured forward time. These measurements exclude video preparation and transfers; open-ended planning and streaming interaction remain proposed applications.

1 Introduction

Many visual-agent steps require a compact judgment—whether an event occurred, when it occurred, or which candidate action fits the evidence. VideoJev extends System One decision models [2] to video, retaining the language flexibility of MLLMs while avoiding answer-token generation.

One candidate-scoring interface serves both temporal intervals and textual options. Shared-prefix computation lowers the marginal cost of another question about the same video. A matched-data autoregressive control isolates gains beyond added training data, while seven benchmarks test accuracy and efficiency.

2 VideoJev

2.1 A Unified Video Decision Framework

The same normalized similarity head scores both kinds of candidates. Task instructions, candidate construction, supervision, and decoding determine whether the output is an event span or an answer.

Figure 2 : VideoJev: direct candidate scoring with shared visual computation. (a) Temporal intervals, represented by timestamps, and textual options share an input format. (b) Cached inference prefills the video once and processes each query suffix separately; Joint inference processes each query group in one causal language-model forward pass. Both modes share the backbone and scorer. (c) Temporal scores yield an event span; option scores select an answer. Scores and the reference interval are illustrative.
Figure 2: VideoJev: direct candidate scoring with shared visual computation. (a) Temporal intervals, represented by timestamps, and textual options share an input format. (b) Cached inference prefills the video once and processes each query suffix separately; Joint inference processes each query group in one causal language-model forward pass. Both modes share the backbone and scorer. (c) Temporal scores yield an event span; option scores select an answer. Scores and the reference interval are illustrative.

Architecture.

Remove the final MLP and replace the language-model output head with a scorer. New <|option_end|> and <|decide|> tokens expose candidate and query states, initialized from the pretrained end-of-turn embedding. Separate 512-dimensional projections are unit-normalized; their dot product receives fixed scale 10 and a learned bias.

Equation 1
u=Wq​hd+bq‖Wq​hd+bq‖2,vi=Wk​hi+bk‖Wk​hi+bk‖2,si=10​u⊤​vi+b.u=\frac{W_{q}h_{d}+b_{q}}{\|W_{q}h_{d}+b_{q}\|_{2}},\qquad v_{i}=\frac{W_{k}h_{i}+b_{k}}{\|W_{k}h_{i}+b_{k}\|_{2}},\qquad s_{i}=10\,u^{\top}v_{i}+b. (1)

Unifying temporal and option decisions.

Place video tokens before the task, query, and candidate list. Each candidate ends with its readout marker; a final decision marker sees the full list. Timestamp candidates use sigmoid membership scores to recover a span; textual options use softmax probabilities to select one answer. Causal attention lets each candidate see earlier candidates.

Reusable video computation.

Cached inference encodes and prefills the video once, then branches each query independently from that cache. Joint inference concatenates queries after the prefix and reads all scores in one forward pass. Later queries see earlier query text and options, but no predicted or reference answers. Training uses this joint layout.

2.2 Temporal Grounding through Interval Decisions

Grounding combines local interval membership with event-wide coverage supervision, then decodes a single contiguous event.

Learning temporal decisions.

The backbone groups frames in pairs. Each pair defines an interval whose midpoint determines its binary event label; adjacent boundaries lie halfway between frame pairs. BCE trains independent membership probabilities, and duration-weighted soft IoU encourages coverage without excess background. Omit soft IoU when no midpoint is positive.

Equations 2–3
ℒBCE=−1K∑i=1K[yilogpi+(1−yi)log(1−pi)].\mathcal{L}_{\mathrm{BCE}}=-\frac{1}{K}\sum_{i=1}^{K}\left[y_{i}\log p_{i}+(1-y_{i})\log(1-p_{i})\right]. (2)
ℒIoU=1−∑idi​pi​yi∑idi​[yi+pi​(1−yi)].\mathcal{L}_{\mathrm{IoU}}=1-\frac{\sum_{i}d_{i}p_{i}y_{i}}{\sum_{i}d_{i}\left[y_{i}+p_{i}(1-y_{i})\right]}. (3)

Decoding a temporal interval.

Subtract the log-odds of the fixed threshold 0.25 from interval logits and use Kadane’s algorithm to find the highest-sum nonempty contiguous range in linear time. If every adjusted score is negative, return the strongest interval. Convert endpoints to continuous time and apply boundary postprocessing; smoothing window 1 is fixed throughout.

Equation 4
(a⋆,b⋆)=arg⁡max⁡∑i=ab1≤a≤b≤K⁡[si−log⁡τ1−τ].(a^{\star},b^{\star})=\arg\max_{1\leq a\leq b\leq K}\sum_{i=a}^{b}\left[s_{i}-\log\frac{\tau}{1-\tau}\right]. (4)

2.3 Video QA through Option Selection

Learning candidate preferences.

Normalize option logits with softmax and minimize cross-entropy for the single correct answer.

Equation 5
pi=exp⁡(si)∑j=1Kexp⁡(sj),ℒCE=−log⁡py,p_{i}=\frac{\exp(s_{i})}{\sum_{j=1}^{K}\exp(s_{j})},\qquad\mathcal{L}_{\mathrm{CE}}=-\log p_{y}, (5)

Reducing position bias.

Apply reproducible, answer-independent option permutations during training, except when positions or earlier turns affect meaning. Preserve supplied option order at inference.

2.4 Training Recipe

The dense backbone is Qwen3.5-4B; the MoE backbone is Qwen3.6-35B-A3B. New projection weights use Gaussian initialization with standard deviation 0.005 and zero biases.

Training data.

Grounding uses TimeLens-100K and TimeLens2-93K [55] [61]. QA uses an internally curated corpus of roughly 240,000 questions over 19,000 videos, spanning recognition, retrieval, temporal reasoning, counting, OCR, and summarization. Group questions by video to share its prefix.

Two-stage training.

Stage one trains only the scorer on 30% of videos, freezing backbone and marker embeddings. Stage two adapts the language backbone, markers, and head on the remaining 70%, adding twice the soft-IoU loss for grounding. Both stages run one epoch; the visual encoder and merger stay frozen. Questions receive equal loss weight.

Equations 6–8
ℒTG=ℒBCE,ℒQA=ℒCE.\mathcal{L}_{\mathrm{TG}}=\mathcal{L}_{\mathrm{BCE}},\qquad\mathcal{L}_{\mathrm{QA}}=\mathcal{L}_{\mathrm{CE}}. (6)
ℒTG\displaystyle\mathcal{L}_{\mathrm{TG}} =ℒBCE+2​ℒIoU,\displaystyle=\mathcal{L}_{\mathrm{BCE}}+2\,\mathcal{L}_{\mathrm{IoU}}, (7)
ℒQA\displaystyle\mathcal{L}_{\mathrm{QA}} =ℒCE.\displaystyle=\mathcal{L}_{\mathrm{CE}}. (8)

3 Experiments

Experiments use Joint inference unless specified otherwise; timing and prediction-quality protocols differ and are detailed in Appendix A.

3.1 Experimental Setup

Training configuration.

Training uses 64 H200 GPUs, BF16, 64 packed sequences per batch, 2 fps videos, and 144K-token sequences with an 80% visual budget. AdamW uses weight decay 0.01, clipping 1.0, 5% warm-up, and cosine decay. Head learning rates are 3×10⁻⁴ then 10⁻⁴; stage-two backbone rates are 5×10⁻⁶ (4B) or 2×10⁻⁶ (MoE). The stages take 160 and 375 updates.

Temporal grounding benchmarks and metrics.

Grounding uses TimeLens-Bench refined Charades-STA, ActivityNet Captions, and QVHighlights annotations, reporting mIoU and rank-one recall at overlap thresholds 0.3, 0.5, and 0.7.

Video QA benchmarks and metrics.

QA covers LVBench (1,549 questions), Video-MME (2,700), LongVideoBench (1,337), and seven MLVU tasks (2,174). Inputs contain frames only, without subtitles or audio. Report accuracy except for MLVU’s task-averaged score.

Inference latency.

Measure batch-one H200 forward time on all Charades and LVBench test videos, charging video encoding, prefix prefill, and any autoregressive decoding once per video before dividing by query count. Generative baselines also reuse the prefix. Loading, preprocessing, transfers, and postprocessing are excluded; truncated generations remain timed.

Table 1: Video temporal grounding. Generative models use Instruct mode by default; VideoJev uses Joint inference. The superscript TL marks TimeLens-Bench refined annotations. Bold marks the best value and underline the second best. Appendix A details the evaluation settings.
   
Model seconds     R1@.3 R1@.5 R1@.7 mIoU R1@.3 R1@.5 R1@.7 mIoU R1@.3 R1@.5 R1@.7 mIoU     Avg.
Proprietary models
GPT-5 [31] –     59.3 42.0 22.0 40.5 57.4 44.9 30.4 42.9 72.4 60.4 46.4 56.8     46.7
Gemini-2.5-Pro [13] –     74.1 61.1 34.0 52.8 72.3 64.2 47.1 58.1 84.1 75.9 61.1 70.4     60.4
Open-source dense models (4B language backbone)
Molmo2-4B [8] 0.636     44.4 31.1 18.5 34.7 50.8 40.3 29.9 40.9 73.7 62.6 51.8 60.8     45.5
Qwen3-VL-4B [3] 0.703     71.8 53.8 28.7 49.4 67.4 54.8 37.6 50.7 78.4 66.4 51.1 62.3     54.1
Mage-VL [45] 1.437     75.5 63.1 36.1 55.0 65.1 57.0 39.7 51.6 76.4 68.1 53.6 63.4     56.7
Qwen3.5-4B [33] 0.763     75.2 56.6 31.5 52.3 72.6 61.6 42.6 55.6 82.9 73.1 58.1 67.9     58.6
VideoChat3-4B [25] 1.067     78.4 64.9 35.9 56.1 70.1 60.5 42.2 54.6 81.0 72.5 56.8 67.0     59.2
Video-ORA-4B [26] 2.347     77.1 64.9 37.9 56.8 69.3 60.1 43.8 57.3 81.4 72.6 58.6 67.5     60.5
TimeLens2-4B [61] 0.646     80.3 66.6 37.5 57.7 74.6 65.5 47.2 59.0 83.3 75.2 60.0 69.3     62.0
VideoJev-4B 0.276     82.8 69.4 40.3 60.1 78.8 70.5 51.4 62.8 87.3 79.4 64.8 73.2     65.3
Open-source MoE models (30-35B total, 3B active)
Nemotron-3-Nano-Omni-30B-A3B [30] 3.882     65.2 49.9 26.6 45.3 54.7 43.5 27.9 40.6 66.7 55.7 41.2 52.6     46.2
Qwen3-VL-30B-A3B [3] 1.055     70.3 46.5 25.1 48.1 65.7 51.1 36.5 49.4 79.3 67.6 52.6 63.2     53.6
Qwen3.5-35B-A3B [33] 1.021     71.5 44.2 21.1 46.0 72.1 57.3 40.4 54.4 80.3 67.3 53.2 64.3     54.9
Qwen3.6-35B-A3B [34] 1.054     72.6 47.0 23.2 47.3 73.9 59.8 42.4 56.0 83.3 72.5 57.6 67.5     56.9
VideoJev-35B-A3B 0.336     86.1 74.8 44.5 63.7 81.0 72.5 54.4 65.5 88.0 79.9 65.5 74.0     67.7

3.2 Video Temporal Grounding

Mean grounding mIoU reaches 65.3 (4B) and 67.7 (MoE), improving on their backbones by 6.7 and 10.8 points. The 4B model exceeds TimeLens2-4B by 3.3 points and improves stricter-overlap recall. Measured grounding speedups are 2.8–3.1× against prefix-sharing generative backbones; the larger VideoJev adds 2.4 mIoU points with 22% higher latency.

Table 2: Multiple-choice Video QA. Molmo2 uses a 32K context; Nemotron uses 128K. Bold marks the best value and underline the second best. Appendix A details the evaluation settings.
Latency     LVBench Video-MME LongVideoBench MLVU     Avg.
Model seconds     test; w/o sub. test; w/o sub. val; w/o sub. dev MCQ; M-Avg     4-bench mean
Proprietary models
GPT-5.2 [33] –     73.7 85.8 – 85.6     –
Gemini-3-Pro [33] –     76.2 87.7 – 83.0     –
Claude-Opus-4.5 [33] –     57.3 81.4 – 81.7     –
Seed2.0-Pro [4] –     76.4 – 80.3 –     –
Open-source dense models (4B language backbone)
Molmo2-4B [8] 0.748     53.9 69.6 68.0 74.3     66.5
VideoChat3-4B [25] 1.210     56.7 70.1 64.1 73.8     66.2
Qwen3-VL-4BInstruct [3] 3.012     56.2 69.3 66.5 75.3     66.8
Qwen3-VL-4BThinking [3] 19.713     53.5 68.9 63.4 75.7     65.4
Qwen3.5-4BInstruct [33] 1.355     60.0 70.6 61.2 73.3     66.3
Qwen3.5-4BThinking [33] 144.929     66.4 76.9 73.6 82.8     74.9
VideoJev-4B 1.123     67.9 75.0 73.6 80.6     74.3
Open-source MoE models (30-35B total, 3B active)
Nemotron-3-Nano-Omni-30B-A3BInstruct [30] 1.106     54.9 70.8 67.5 77.2     67.6
Nemotron-3-Nano-Omni-30B-A3BThinking [30] 229.095     52.2 72.2 69.6 79.6     68.4
Qwen3-VL-30B-A3BInstruct [3] 4.817     62.5 74.5 70.9 81.3     72.3
Qwen3-VL-30B-A3BThinking [3] 9.637     59.2 73.3 69.1 78.9     70.1
Qwen3.5-35B-A3BInstruct [33] 1.683     58.8 74.4 65.7 77.4     69.1
Qwen3.5-35B-A3BThinking [33] 164.730     71.4 82.5 74.5 85.6     78.5
Qwen3.6-35B-A3BInstruct [34] 2.379     66.0 77.1 72.4 79.5     73.8
Qwen3.6-35B-A3BThinking [34] 149.777     71.4 82.5 74.9 86.2     78.8
VideoJev-35B-A3B 1.493     73.4 81.8 77.1 85.9     79.5

3.3 Option Selection on Video QA Benchmarks

All VideoJev QA runs use a 256K-token context, requested 2 fps sampling capped at 2,048 frames, and an 80% visual-token budget. Mean QA gains over Instruct backbones are 8.0 and 5.7 points. Against Thinking counterparts, 4B trails by 0.6 points and MoE leads by 0.7; Thinking still leads on Video-MME and MLVU. Measured speedups are 129× and 100×, respectively.

3.4 Ablation Study

Matched-data and architectural ablations use complete benchmark splits, the same decoder, and 256K-token QA contexts.

Isolating VideoJev’s architectural gains.

Single-stage VideoJev-4B beats autoregressive Qwen3.5-4B trained on the identical data and schedule by 1.3 grounding and 1.9 QA points. This supports a benefit from decision-oriented architecture and supervision beyond additional data.

Table 3: Isolating architectural gains with an autoregressive control. VideoJev-4B and an autoregressive Qwen3.5-4B are both fine-tuned in a single stage on exactly the same post-training data. † denotes the VideoJev-4B Single-stage (100%) variant from Table 8. The two models share the same training and evaluation settings.
   
Model     CharadesTL ActivityNetTL QVHighlightsTL Avg.     LVBench Video-MME LongVideoBench MLVU Avg.
VideoJev-4B†\dagger (baseline)     60.6 62.8 72.9 65.4     67.7 74.9 73.5 82.6 74.7
Qwen3.5-4B AR SFT     61.0 [1pt](+0.4) 61.3 [1pt](-1.5) 70.0 [1pt](-2.9) 64.1 [1pt](-1.3)     66.9 [1pt](-0.8) 74.5 [1pt](-0.4) 71.4 [1pt](-2.1) 78.6 [1pt](-4.0) 72.8 [1pt](-1.9)

Cached inference and Joint inference.

With unchanged checkpoints, Joint inference improves 13 of 14 model–benchmark comparisons. Mean grounding/QA gains over independent cached queries are 0.8/0.5 points at 4B and 0.3/0.7 at MoE. Standard causal attention allows query interactions; alignment with joint training is a proposed explanation, not an isolated causal result.

Table 4: Cached inference versus Joint inference.
   
Setting     CharadesTL ActivityNetTL QVHighlightsTL Avg.     LVBench Video-MME LongVideoBench MLVU Avg.
VideoJev-4B
Joint inference (baseline)     60.1 62.8 73.2 65.3     67.9 75.0 73.6 80.6 74.3
Cached inference     58.6 [1pt](-1.5) 61.8 [1pt](-1.0) 73.1 [1pt](-0.1) 64.5 [1pt](-0.8)     67.1 [1pt](-0.8) 74.9 [1pt](-0.1) 72.9 [1pt](-0.7) 80.1 [1pt](-0.5) 73.8 [1pt](-0.5)
VideoJev-35B-A3B
Joint inference (baseline)     63.7 65.5 74.0 67.7     73.4 81.8 77.1 85.9 79.5
Cached inference     63.1 [1pt](-0.6) 65.0 [1pt](-0.5) 74.1 [1pt](+0.1) 67.4 [1pt](-0.3)     72.2 [1pt](-1.2) 81.3 [1pt](-0.5) 76.7 [1pt](-0.4) 85.0 [1pt](-0.9) 78.8 [1pt](-0.7)

Joint training for grounding and QA.

Joint grounding and QA training improves all seven benchmarks relative to corresponding single-task models: +0.7 mean grounding and +1.3 mean QA points.

Table 5: Joint versus single-task training. VideoJev-4B is trained on temporal grounding (TG), video question answering (QA), or both.
   
Setting     CharadesTL ActivityNetTL QVHighlightsTL Avg.     LVBench Video-MME LongVideoBench MLVU Avg.
TG+QA (baseline)     60.1 62.8 73.2 65.3     67.9 75.0 73.6 80.6 74.3
TG-only     59.6 [1pt](-0.5) 61.8 [1pt](-1.0) 72.5 [1pt](-0.7) 64.6 [1pt](-0.7)     – – – – –
QA-only     – – – –     66.3 [1pt](-1.6) 74.9 [1pt](-0.1) 71.7 [1pt](-1.9) 79.2 [1pt](-1.4) 73.0 [1pt](-1.3)

Auxiliary supervision.

Adding soft IoU raises mean grounding from 60.7 to 65.3 and QA from 73.9 to 74.3, supporting event-coverage supervision.

Table 6: Effect of Soft IoU supervision in VideoJev-4B. Soft IoU augments the grounding BCE objective; QA retains cross-entropy. Deltas are relative to BCE/CE only (baseline). Both variants use two-stage training.
   
Setting     CharadesTL ActivityNetTL QVHighlightsTL Avg.     LVBench Video-MME LongVideoBench MLVU Avg.
BCE/CE only (baseline)     53.3 58.8 70.0 60.7     67.4 74.6 72.9 80.5 73.9
+ Soft IoU     60.1 [1pt](+6.8) 62.8 [1pt](+4.0) 73.2 [1pt](+3.2) 65.3 [1pt](+4.6)     67.9 [1pt](+0.5) 75.0 [1pt](+0.4) 73.6 [1pt](+0.7) 80.6 [1pt](+0.1) 74.3 [1pt](+0.4)

Final-layer MLP removal.

Removing the final MLP changes mean grounding and QA by only +0.2 points each while simplifying computation.

Table 7: Effect of the final-layer MLP in VideoJev-4B.
   
Setting     CharadesTL ActivityNetTL QVHighlightsTL Avg.     LVBench Video-MME LongVideoBench MLVU Avg.
Remove final MLP (baseline)     60.1 62.8 73.2 65.3     67.9 75.0 73.6 80.6 74.3
Retain final MLP     60.5 [1pt](+0.4) 62.1 [1pt](-0.7) 72.7 [1pt](-0.5) 65.1 [1pt](-0.2)     67.9 [1pt](0.0) 74.7 [1pt](-0.3) 73.0 [1pt](-0.6) 80.8 [1pt](+0.2) 74.1 [1pt](-0.2)

Two-stage adaptation.

Two-stage adaptation remains close to single-stage training, which leads by 0.1 grounding and 0.4 QA points. It updates the expensive backbone on only 70% of videos; possible stabilization is conjectured.

Table 8: Two-stage versus single-stage adaptation of VideoJev-4B. The baseline warms up the head on 30% of videos and jointly adapts on the remaining 70%; single-stage training directly adapts on all videos with the full objectives.
   
Setting     CharadesTL ActivityNetTL QVHighlightsTL Avg.     LVBench Video-MME LongVideoBench MLVU Avg.
Two-stage (baseline)     60.1 62.8 73.2 65.3     67.9 75.0 73.6 80.6 74.3
Single-stage (100%)     60.6 [1pt](+0.5) 62.8 [1pt](0.0) 72.9 [1pt](-0.3) 65.4 [1pt](+0.1)     67.7 [1pt](-0.2) 74.9 [1pt](-0.1) 73.5 [1pt](-0.1) 82.6 [1pt](+2.0) 74.7 [1pt](+0.4)

3.5 Analysis

Query scaling and inference efficiency.

Timing uses 40 duration-stratified videos per task and 1–10 queries. Accuracy labels come from separate full-benchmark evaluations. Joint VideoJev adds only 55 ms for grounding and 31 ms for QA between one and ten queries. At ten queries, times are 0.39/1.64 seconds versus Cached inference’s 0.85/2.17 seconds.

A one-token Direct QA baseline supports joint readouts, but its LVBench accuracy falls from 60.0% to 58.7%; VideoJev reaches 67.9% at similar latency. Multi-token intervals and reasoning require sequential decoding even when all queries are available together.

Figure 3 : Multi-query efficiency at 4B. Mean forward time for 1–10 queries per video on Charades (a) and Video-Holmes [ 6 ] (b); shading shows 95% confidence intervals. Solid/dotted lines use Cached inference (dotted: multi-token generation); dashed lines use Joint inference . End labels pair ten-query latency with separately evaluated full-benchmark Charades mIoU or LVBench accuracy; quality settings follow the source evaluations in Appendix A.2 .
Figure 3: Multi-query efficiency at 4B. Mean forward time for 1–10 queries per video on Charades (a) and Video-Holmes [6] (b); shading shows 95% confidence intervals. Solid/dotted lines use Cached inference (dotted: multi-token generation); dashed lines use Joint inference. End labels pair ten-query latency with separately evaluated full-benchmark Charades mIoU or LVBench accuracy; quality settings follow the source evaluations in Appendix A.2.
Figure 4 : Confidence calibration on Video QA. Points show mean confidence and observed accuracy in equal-width bins containing at least 20 questions; all bins contribute to ECE. The dashed diagonal denotes perfect calibration. MLVU uses seven-task-balanced weights; the other benchmarks weight questions equally. Curves use the reported per-benchmark predictions.
Figure 4: Confidence calibration on Video QA. Points show mean confidence and observed accuracy in equal-width bins containing at least 20 questions; all bins contribute to ECE. The dashed diagonal denotes perfect calibration. MLVU uses seven-task-balanced weights; the other benchmarks weight questions equally. Curves use the reported per-benchmark predictions.

Confidence and calibration.

Higher confidence generally predicts greater accuracy, but probabilities overestimate correctness. At confidence ≥99%, the MoE model gets 95.8% of 591 LVBench predictions right. Four-benchmark mean ECE is 8.65/8.13 percentage points for 4B/MoE; no calibration-specific training was used. Selective prediction would need improved calibration.

4 Related Work

MLLM video understanding.

Prior video-language models expand instruction following, long-video reasoning, streaming memory, and proactive responses. VideoJev instead emphasizes repeated direct decisions over an encoded video context.

Temporal grounding.

Temporal grounding has used candidate matching, span prediction, explicit time tokens, and boundary distributions. GroundVQA and SEAL already combine grounding with QA [9] [40]; VideoJev shares a scorer between interval membership and textual options.

System One models.

Jev and image-based System One models motivate typed probabilistic judgments. VideoJev adds temporal decisions and shared-prefix multi-query training and inference.

5 Future Work

Toward general-purpose visual decisions.

Action recognition, scene-state checks, and event detection are possible extensions of the candidate interface; these tasks were not evaluated here.

From closed-set decisions to open-ended agent workflows.

A higher-level planner could propose candidate events or actions, call VideoJev to evaluate them, and revise ambiguous candidates. This proposed System Two/System One workflow extends beyond the closed-set experiments.

Toward streaming and reliable visual interaction.

Streaming would require incremental encoding and memory management. Reliability also needs calibration, uncertainty estimation, and decisions about when to defer to further reasoning.

6 Conclusion

Direct candidate scoring and shared visual computation provide efficient structured video judgments. The results support the interface on seven benchmarks, while broader interactive workflows remain future work.

Supplementary Material

Contents

Appendix A Detailed Evaluation Protocol

Inputs contain frames without subtitles or audio. Grounding generally samples at 4 fps; QA requests up to 2 fps and caps frames at 2,048. Maximum frame area is 640×640 pixels, reduced to satisfy context budgets.

A.1 Latency

Latency includes synchronized encoding, prefill, and decoding through EOS or the output limit on initially idle H200 GPUs, with batch size one. It excludes data preparation, transfers, parsing, and metrics. Invalid and truncated outputs remain included; generative query branches do not receive earlier answers.

Temporal grounding latency.

Grounding timing covers 1,313 videos and 3,363 queries with per-video warm-up. Most models use a 262,144-token context and 4,096 output-token limit; Nemotron allows 1,024 outputs and Molmo2 uses 110,592 context tokens. Native visual merging/pruning remains.

Video QA latency.

QA timing covers 103 videos and 1,549 questions. Qwen and VideoJev share fixed sampling geometry and a 256K context, while other architectures keep native visual settings. MoE routers and decision heads retain FP32 precision.

Multi-query scaling.

Query-scaling timing samples ten videos per Charades duration quartile and five per Video-Holmes duration stratum, using the same ordered queries and 2 fps inputs. After warm-up, retain the median ten-query-time trace from three fixed-seed runs. Equal-video means have stratified paired-bootstrap 95% confidence intervals from 10,000 resamples.

A.2 Performance

Temporal grounding performance.

Grounding quality evaluates all 3,363/4,500/1,541 Charades/ActivityNet/QVHighlights queries. Full-video 4 fps inputs fit a 147,456-token context and 114,688-token input budget. Invalid intervals fail; some comparison scores remain their published values and settings.

Video QA performance.

QA uses complete splits and fixed 256K VideoJev contexts; MLVU averages seven tasks. Generative quality evaluations answer independently, unlike shared-prefix timing. Published scores retain their original protocols. Deterministic option parsing can fall back to a text-only extractor without access to ground truth; invalid answers are incorrect.

Multi-query prediction quality.

Figure 3’s quality labels use full Charades and LVBench, independently of its sampled timing videos. Qwen Instruct uses temperature/top-p 0.7/0.8 and Thinking 1.0/0.95, top-k 20, presence penalty 1.5, and at most 32,768 output tokens. Direct masks invalid options before sampling; invalid or truncated answers score zero.

A.3 Autoregressive Control with the Same Training Data

The matched autoregressive control and single-stage VideoJev share 404,037 questions from 58,424 videos. The control retains the native final MLP and tied output head, trains one epoch for 546 updates with frozen visual modules, and supervises answer tokens including EOS.

Targets are interval lists or option letters without Thinking. Evaluation shares VideoJev’s visual budgets and independently branches queries from an immutable prefix. Greedy generation allows 256 tokens for intervals and 16 for QA; grounding scores the first interval. Thus Table 3 compares both architecture and the corresponding task supervision under matched data.

Figure 5 : Qwen3.5-4B Direct inference. (a) Queries branch independently from a shared video cache. (b) All option predictions are read in one causal forward. Both modes use batch size 1, restrict logits to valid option tokens, and leave answer slots empty; [readout] marks a logit readout, not inserted text.
Figure 5: Qwen3.5-4B Direct inference. (a) Queries branch independently from a shared video cache. (b) All option predictions are read in one causal forward. Both modes use batch size 1, restrict logits to valid option tokens, and leave answer slots empty; [readout] marks a logit readout, not inserted text.

A.4 Inference Modes and Direct Baseline

Shared inference modes.

Cached inference runs independent suffix branches. Joint inference uses one causal pass with no inserted answers, so later queries see earlier questions. Autoregressive joint readouts require single-token outputs; temporal lists and reasoning traces require cached sequential decoding.

Direct option prediction.

Direct reads valid option-token logits at empty answer positions, allowing plain or leading-space option tokens and applying official sampling filters after masking.

Empty answer histories.

Without masking, empty answer histories produced end-of-turn tokens for 1,418 of 1,446 later LVBench questions. Earlier answers—even wrong ones—restored options. Reported Direct results avoid inserting answers and explicitly mask valid options, exposing a limitation of unconstrained joint autoregressive readouts.

Appendix B Qualitative Benchmark Cases

Eight complete qualitative cases span all seven benchmarks. They include precise adjacent actions, long-video retrieval, successful counting, disagreement about intent, and a shared failure to count four small books. Scores are uncalibrated; enlarged details are for display only. These examples illustrate behavior rather than establish aggregate reliability.

Figure 6 : Joint inference grounds six Charades queries in one forward; three readouts are shown in order. Both models match the one-second transition, while VideoJev-35B-A3B locates adjacent actions more precisely.
Figure 6: Joint inference grounds six Charades queries in one forward; three readouts are shown in order. Both models match the one-second transition, while VideoJev-35B-A3B locates adjacent actions more precisely.
Figure 7 : Localizing a compound jump-rope event on ActivityNet Captions. VideoJev-35B-A3B more closely matches the annotated interval.
Figure 7: Localizing a compound jump-rope event on ActivityNet Captions. VideoJev-35B-A3B more closely matches the annotated interval.
Figure 8 : Retrieving a preparation sequence amid vlog distractors on QVHighlights. Both models match the annotated interval.
Figure 8: Retrieving a preparation sequence amid vlog distractors on QVHighlights. Both models match the annotated interval.
Figure 9 : Joint inference answers 18 questions over a 3,665.5-second LVBench video in one forward. Two readouts test caption retrieval and weather recognition; the caption detail is enlarged for display.
Figure 9: Joint inference answers 18 questions over a 3,665.5-second LVBench video in one forward. Two readouts test caption retrieval and weather recognition; the caption detail is enlarged for display.
Figure 10 : Recognizing an event relation on LongVideoBench. Both models select the annotated answer from six options.
Figure 10: Recognizing an event relation on LongVideoBench. Both models select the annotated answer from six options.
Figure 11 : Counting five recurring scenes amid unrelated footage on MLVU. Each frame pair shows one occurrence. Both models select the annotated answer, with different option scores.
Figure 11: Counting five recurring scenes amid unrelated footage on MLVU. Each frame pair shows one occurrence. Both models select the annotated answer, with different option scores.
Figure 12 : An intent-inference disagreement on LVBench: VideoJev-4B interprets the interaction as courtesy, while VideoJev-35B-A3B selects the annotated arrest intent.
Figure 12: An intent-inference disagreement on LVBench: VideoJev-4B interprets the interaction as courtesy, while VideoJev-35B-A3B selects the annotated arrest intent.
Figure 13 : A shared failure on small-object counting in Video-MME. Both models miss the annotated answer of four books; the book detail is enlarged for display.
Figure 13: A shared failure on small-object counting in Video-MME. Both models miss the annotated answer of four books; the book detail is enlarged for display.
Concept demonstration

Open image ↗