At a glance
VideoWeave packs frames and captions from several short videos into one training example. This exposes a video-language model to more source clips while fixing the number of input frames and training examples. With the tested architecture and budget, random pairs of clips improve VideoMME accuracy over single-video fine-tuning. Visually clustering clips and rewriting captions are less effective. The result supports a simple data-packing strategy; it does not establish general long-horizon temporal reasoning.
1 Introduction
Long videos are expensive to annotate and process, while many short captioned clips already exist. VideoWeave constructs synthetic sequences from those clips without adding human annotations. The intended benefit is greater visual diversity per training step. Concatenated clips can contain unrelated scenes, so their usefulness should be assessed through downstream evaluation rather than assumed to supply realistic temporal continuity.
2 Methodology
The model receives a fixed total of 16 frames per example. Increasing the number of source videos reduces frames per video and packs more captions into the same example. The architecture and optimization recipe are otherwise held broadly constant.
2.1 Video Training Data Curation
Nested WebVid-10M subsets contain 10,000 to 160,000 videos. Each one-epoch run processes 10,000 packed examples, with captions corresponding to the sampled clips. Thus all strategies see 160,000 frames, but they differ in distinct videos, frames per clip, and caption targets. Equal frame counts and steps are the concrete budget control, rather than proof of identical token-level computation.
2.2 Video-Language Model Architecture
A Prismatic-based model [27] independently encodes frames with CLIP-ViT-L/14, projects them through a GELU multilayer perceptron, and supplies them to LLaMA-2. Frame embeddings are arranged into video sequences after encoding. Rotary position scaling of 3.0 expands the context for 16 frames. The caption prompt is fixed, and frames are uniformly sampled. This is a straightforward video adaptation of an image-language architecture.
2.3 Random Selection
Random splicing tests one, two, four, eight, or sixteen videos per example. Original captions are joined with spaces. Two clips with eight frames each give the best reported overall score, indicating that diversity helps but excessive packing can discard useful within-video information.
2.4 Video Clustering
A fixed-size variant of K-means groups averaged CLIP video features before splicing. Algorithm 1 preserves that clustering procedure. Visually similar groups might improve coherence, but can also reduce diversity and let a model describe many frames from a small initial subset. In the reported experiments, random grouping performs better.
Equation
2.5 Caption Generation
GPT-4o-mini rewrites the joined captions into a cohesive description without seeing the videos. This enrichment substantially lowers downstream accuracy. The main method therefore keeps original captions. More fluent synthetic text need not provide more faithful or useful supervision.
3 Experiments and Results
Training uses a LLaMA-2 7B backbone, CLIP frames resized to 336×336, one epoch, and two L40-family GPUs. Evaluation uses VideoMME’s [19] 900 videos and 2,700 multiple-choice questions spanning short, medium, and long videos. All models receive 16 sampled frames at test time. Reported run times range from five to 100 hours, so the study’s fixed-budget description refers primarily to its controlled frame/sample counts.
Experimental Results
The main table reports 36.6% overall accuracy for multi-video fine-tuning versus 34.5% for single-video and 31.7% for the image baseline. Short-video accuracy rises from 41.7% to 44.8%, a 3.1-point gain. Table 2 favors two clips; sixteen clips fall to 33.3%. Table 3 lists random-selection values that do not match the corresponding rows of Table 2, so its exact clustering deltas should be read cautiously. Caption rewriting reaches only 21.6%, 15 points below the best original-caption result.
| Method | VideoMME (short) () | VideoMME (all) () |
|---|---|---|
| Image Baseline (No FT) | 34.2 | 31.7 |
| Single-Video FT | 41.7 | 34.5 |
| Multi-Video FT (ours) | 44.8 | 36.6 |
| Videos () | Frames per video | VideoMME () |
| 16 | 1 | 33.3 |
| 8 | 2 | 34.3 |
| 4 | 4 | 35.4 |
| 2 | 8 | 36.6 |
| 1 | 16 | 34.5 |
| Videos | Frames per video | VideoMME (random) | VideoMME (clustered) |
|---|---|---|---|
| 4 | 4 | 34.3 | 14.8 |
| 2 | 8 | 35.4 | 31.6 |
| 1 | 16 | 34.5 | N/A |
Qualitative Results
The retained example shows a VideoMME question answered correctly after multi-video training. Both compared models receive the same 16 frames. It illustrates a successful case rather than estimating the frequency of that behavior.
Comparison to Single Video (Standard) Finetuning
Per-category plots show the largest improvement on short videos. Reasoning categories improve little, which the authors hypothesize may reflect the shared language backbone. Sparse frame sampling also limits what the system can observe in long videos. These results favor improved data exposure more strongly than a claim of learned long-range causal reasoning.
4 Related Works
Related work separates model architecture, temporal representation, evaluation, synthetic supervision, and training efficiency.
Vision-Language Modeling Architectures
Modular vision-language systems connect pretrained visual encoders and language models through adapters, projection layers, or query modules. VideoWeave builds on that modular pattern and changes the organization of training examples.
Video-Language Understanding and Temporal Modeling
Video models aggregate temporal features or directly process frame embeddings with a language model. VideoWeave uses the latter simple strategy, without introducing a new temporal module.
Benchmarks for Long Video Understanding
LVBench, VideoMME, HourVideo, and EgoSchema probe increasingly extended video understanding. This paper evaluates VideoMME only, so coverage of other temporal benchmarks remains untested.
Synthetic Video-Text Datasets
Synthetic captions and instruction datasets increase video supervision without extensive manual annotation. VideoWeave instead reuses existing captions; its unsuccessful caption-rewriting experiment cautions against treating text fluency as training quality.
Efficient Training of Video-Language Models
The efficiency contribution is packing diverse short clips into fixed-frame examples. It can complement architectural efficiency methods, but the demonstrated evidence is limited to the chosen backbone, dataset subsets, and training schedule.
5 Conclusion
Randomly combining a small number of captioned clips improves the tested model under controlled frame and sample budgets. Benefits are clearest on short videos with fairly uniform content. Rich temporal dynamics, larger training corpora, different architectures, and additional evaluation datasets remain open tests.
6 Acknowledgments
The authors acknowledge support from Schmidt Futures, the NSF Graduate Research Fellowship Program, and Google Cloud credits through Stanford’s human-centered AI institute.