Literature
Display
← All papers

VideoWeave: A Data-Centric Approach for Efficient Video Understanding

Zane Durante · Silky Singh · Arpandeep Khatua · Shobhit Agarwal · Reuben Tan · Yong Jae Lee · Jianfeng Gao · Ehsan Adeli · Li Fei-Fei

Preprint posted · arXiv (preprint) ↗

Computer vision4 min text · CondensedOriginal paper ↗
Contents

At a glance

VideoWeave packs frames and captions from several short videos into one training example. This exposes a video-language model to more source clips while fixing the number of input frames and training examples. With the tested architecture and budget, random pairs of clips improve VideoMME accuracy over single-video fine-tuning. Visually clustering clips and rewriting captions are less effective. The result supports a simple data-packing strategy; it does not establish general long-horizon temporal reasoning.

1 Introduction

Long videos are expensive to annotate and process, while many short captioned clips already exist. VideoWeave constructs synthetic sequences from those clips without adding human annotations. The intended benefit is greater visual diversity per training step. Concatenated clips can contain unrelated scenes, so their usefulness should be assessed through downstream evaluation rather than assumed to supply realistic temporal continuity.

2 Methodology

The model receives a fixed total of 16 frames per example. Increasing the number of source videos reduces frames per video and packs more captions into the same example. The architecture and optimization recipe are otherwise held broadly constant.

Refer to caption
Figure 1: Our method VideoWeave combines several videos together to form a single input sample. Our method is architecture and task agnostic, and is parametrized by LL, the number of videos to use for each input pair. In this figure, we visualize L=2L=2, representing two distinct video/caption pairs.

2.1 Video Training Data Curation

Nested WebVid-10M subsets contain 10,000 to 160,000 videos. Each one-epoch run processes 10,000 packed examples, with captions corresponding to the sampled clips. Thus all strategies see 160,000 frames, but they differ in distinct videos, frames per clip, and caption targets. Equal frame counts and steps are the concrete budget control, rather than proof of identical token-level computation.

2.2 Video-Language Model Architecture

A Prismatic-based model [27] independently encodes frames with CLIP-ViT-L/14, projects them through a GELU multilayer perceptron, and supplies them to LLaMA-2. Frame embeddings are arranged into video sequences after encoding. Rotary position scaling of 3.0 expands the context for 16 frames. The caption prompt is fixed, and frames are uniformly sampled. This is a straightforward video adaptation of an image-language architecture.

2.3 Random Selection

Random splicing tests one, two, four, eight, or sixteen videos per example. Original captions are joined with spaces. Two clips with eight frames each give the best reported overall score, indicating that diversity helps but excessive packing can discard useful within-video information.

2.4 Video Clustering

A fixed-size variant of K-means groups averaged CLIP video features before splicing. Algorithm 1 preserves that clustering procedure. Visually similar groups might improve coherence, but can also reduce diversity and let a model describe many frames from a small initial subset. In the reported experiments, random grouping performs better.

Refer to caption
Figure 2: We visualize clusters generated by our modified K-means algorithm. For the experiments described in Section 2.4, we construct input samples by using video clips within a single cluster.
Algorithm 1 Modified K-means clustering algorithm
1: Dataset XX with nn points, number of clusters KK, dd data points per cluster
2: Cluster assignments and final centroids
3: Initialize KK centroids μ1,μ2,…,μK\mu_{1},\mu_{2},\dots,\mu_{K}
4: repeat
5:   for each point xi∈Xx_{i}\in X do
6:    Assign xix_{i} to the nearest cluster kk such that ∑j=1n𝟏[cj=k]<d\sum_{j=1}^{n}\mathbf{1}[c_{j}=k]<d:
Equation
ci←arg⁡mink⁡‖xi−μk‖2c_{i}\leftarrow\arg\min_{k}\|x_{i}-\mu_{k}\|^{2}
7:   end for
8:   for each cluster k=1,…,Kk=1,\dots,K do
9:    Update μk←1|Ck|​∑xi∈Ckxi\mu_{k}\leftarrow\frac{1}{|C_{k}|}\sum_{x_{i}\in C_{k}}x_{i}
10:   end for
11: until cluster assignments do not change

2.5 Caption Generation

GPT-4o-mini rewrites the joined captions into a cohesive description without seeing the videos. This enrichment substantially lowers downstream accuracy. The main method therefore keeps original captions. More fluent synthetic text need not provide more faithful or useful supervision.

Refer to caption
Figure 3: We show an example caption-caption generation pair where GPT-4o-mini modifies our naive caption into a unified, cohesive video caption.

3 Experiments and Results

Training uses a LLaMA-2 7B backbone, CLIP frames resized to 336×336, one epoch, and two L40-family GPUs. Evaluation uses VideoMME’s [19] 900 videos and 2,700 multiple-choice questions spanning short, medium, and long videos. All models receive 16 sampled frames at test time. Reported run times range from five to 100 hours, so the study’s fixed-budget description refers primarily to its controlled frame/sample counts.

Experimental Results

The main table reports 36.6% overall accuracy for multi-video fine-tuning versus 34.5% for single-video and 31.7% for the image baseline. Short-video accuracy rises from 41.7% to 44.8%, a 3.1-point gain. Table 2 favors two clips; sixteen clips fall to 33.3%. Table 3 lists random-selection values that do not match the corresponding rows of Table 2, so its exact clustering deltas should be read cautiously. Caption rewriting reaches only 21.6%, 15 points below the best original-caption result.

Table 1: Our method compared to the baseline VideoMME under fixed compute setting (1 epoch, 10,000 training iterations). For all methods, we start with a LLaMA-2 backbone with a frozen CLIP-ViT/L-14 visual encoder finetuned on the LLaVA-1.5 training set.
Method VideoMME (short) (↑\uparrow) VideoMME (all) (↑\uparrow)
Image Baseline (No FT) 34.2 31.7
Single-Video FT 41.7 34.5
Multi-Video FT (ours) 44.8 36.6
Table 2: Model performance under a fixed compute budget (1 epoch, 10,000 training iterations) with varying video-frame sampling strategies. L=16 samples 1 frame each from 16 videos per iteration (160,000 videos total). L=1 samples all 16 frames from a single video per iteration (10,000 videos total). All strategies process 160,000 total frames. Bold indicates best performance; underline indicates second best.
Videos (LL) Frames per video VideoMME (↑\uparrow)
16 1 33.3
8 2 34.3
4 4 35.4
2 8 36.6
1 16 34.5
Table 3: Model performance across across videos and frames using our modified K-means clustering method described in Section 2.4 under a fixed compute budget (1 epoch, 10,000 training iterations). We show that random selection outperforms K-means clustering for grouping input videos VtV_{t}.
Videos Frames per video VideoMME (random) VideoMME (clustered)
4 4 34.3 14.8
2 8 35.4 31.6
1 16 34.5 N/A

Qualitative Results

The retained example shows a VideoMME question answered correctly after multi-video training. Both compared models receive the same 16 frames. It illustrates a successful case rather than estimating the frequency of that behavior.

Refer to caption
Figure 4: We show an example question from VideoMME where our model correctly identifies the correct answer. As is true across all our results, both models are evaluated using an identical set of 16 frames uniformly sampled across the video.

Comparison to Single Video (Standard) Finetuning

Per-category plots show the largest improvement on short videos. Reasoning categories improve little, which the authors hypothesize may reflect the shared language backbone. Sparse frame sampling also limits what the system can observe in long videos. These results favor improved data exposure more strongly than a claim of learned long-range causal reasoning.

Refer to caption
(a) VideoWeave vs Image Baseline
Refer to caption
(b) VideoWeave vs Single Video Finetune
Figure 5: Per-category performance improvements of multi-video finetuning over (a) single-video finetuning and (b) image baseline on VideoMME.

4 Related Works

Related work separates model architecture, temporal representation, evaluation, synthetic supervision, and training efficiency.

Vision-Language Modeling Architectures

Modular vision-language systems connect pretrained visual encoders and language models through adapters, projection layers, or query modules. VideoWeave builds on that modular pattern and changes the organization of training examples.

Video-Language Understanding and Temporal Modeling

Video models aggregate temporal features or directly process frame embeddings with a language model. VideoWeave uses the latter simple strategy, without introducing a new temporal module.

Benchmarks for Long Video Understanding

LVBench, VideoMME, HourVideo, and EgoSchema probe increasingly extended video understanding. This paper evaluates VideoMME only, so coverage of other temporal benchmarks remains untested.

Synthetic Video-Text Datasets

Synthetic captions and instruction datasets increase video supervision without extensive manual annotation. VideoWeave instead reuses existing captions; its unsuccessful caption-rewriting experiment cautions against treating text fluency as training quality.

Efficient Training of Video-Language Models

The efficiency contribution is packing diverse short clips into fixed-frame examples. It can complement architectural efficiency methods, but the demonstrated evidence is limited to the chosen backbone, dataset subsets, and training schedule.

5 Conclusion

Randomly combining a small number of captioned clips improves the tested model under controlled frame and sample budgets. Benefits are clearest on short videos with fairly uniform content. Rich temporal dynamics, larger training corpora, different architectures, and additional evaluation datasets remain open tests.

6 Acknowledgments

The authors acknowledge support from Schmidt Futures, the NSF Graduate Research Fellowship Program, and Google Cloud credits through Stanford’s human-centered AI institute.

Concept demonstration

Open image ↗