Literature
Display
← All papers

Cambrian-S: Towards Spatial Supersensing in Video

Shusheng Yang · Jihan Yang · Pinzhi Huang · Ellis Brown · Zihao Yang · Yue Yu · Shengbang Tong · Zihan Zheng · Yifan Xu · Muhan Wang · Daohan Lu · Rob Fergus · Yann LeCun · Li Fei-Fei · Saining Xie

· ICLR 2026 ↗

Robotics18 min text · CondensedOriginal paper ↗
Contents

Overview

This edition follows the 49-page arXiv version 1, dated 6 November 2025. It retains all 25 numbered figures, 17 numbered tables, four algorithms, and the full bibliography, including the appendix’s benchmark examples and training details. The paper separates a spatially trained video model from an additional prediction-guided memory prototype.

Original figure Website https://cambrian-mllm.github.io
Original figure Code https://github.com/cambrian-mllm/cambrian-s
Original figure Cambrian-S Models https://hf.co/collections/nyu-visionx/cambrian-s
Original figure VSI-590K https://hf.co/datasets/nyu-visionx/vsi-590k
Original figure VSI-Super https://hf.co/collections/nyu-visionx/vsi-super

At a glance

Cambrian-S combines a curated spatial question-answer dataset with a four-stage multimodal training pipeline. It reaches 67.5 on VSI-Bench but struggles with long streams. The authors introduce VSI-Super recall and counting tasks, then add next-frame latent prediction to guide memory compression and event segmentation. The resulting improvements support a useful prototype for structured long-video processing; they do not demonstrate human-level spatial cognition or a general predictive world model.

1 Introduction

The proposed hierarchy progresses from semantic perception through continuous-event memory and implicit 3D cognition to predictive world modeling. This is a conceptual research agenda, not a measured developmental law. Three contributions test parts of it: diagnosing existing benchmarks, constructing spatial training data and Cambrian-S, and using prediction error to select and organize memories. The paper explicitly does not claim to realize the complete “supersensing” goal.

Complete original figure 1, preserving native PDF vector labels and embedded image detail.
Figure 1: From pixels to predictive mind. We look beyond linguistic-only understanding to envision multimodal intelligence that sees, remembers, and reasons as part of a continuous, lived world. It begins with semantic perception: naming and describing what is seen. Streaming event cognition goes further, enabling always-on sensing across continuous input streams, integrating memory, and supporting proactive responses. Spatial cognition captures the implicit 3D structure of video, enabling reasoning about objects, configurations, and metrics. Finally, a predictive world model emerges, one that learns passively from experience, updates through prediction and surprise, and retains information for future use. Lower illustration: Video serves as the ideal experimental domain. Models must advance from frame-level Q&A to constructing implicit world models that enable deeper spatial reasoning, scale to unbounded horizons, and achieve supersensing that rivals, and ultimately surpasses, human visual intelligence.

2 Benchmarking Spatial Supersensing

Benchmarking first asks what existing video questions actually require, then introduces long-duration spatial recall and counting tasks. Different benchmarks measure different abilities, so their scores are not interchangeable measures of video understanding.

Complete original figure 2, preserving all native vector labels, values, rules, and grouping.
Figure 2: Benchmark diagnostic results reveal varying dependence on visual input. We evaluate model under distinct input conditions: (a) multiple (32) uniformly sampled frames, (b) a single (middle) frame, and (c) frame captions, benchmarked against chance-level and blind test results (visual input ignored). Panels (a–c) show absolute accuracies; panels (d–j) show performance differences between conditions. Visual inputs are substantially more critical for VSI-Bench [148], Tomato [116], and HourVideo [20], while their impact is less pronounced for VideoMME [42], MVBench [71], and VideoMMMU [53]. VSR and VSC are new supersensing benchmarks introduced in Section 2.2.

2.1 Deconstructing Existing Video Benchmarks

An upgraded image-only Cambrian-1 receives 32 sampled frames, one middle frame, captions of those frames, or no visual information. Captions generated by Gemini-2.0-Flash often match or outperform direct frames on semantic benchmarks; VSI-Bench and Tomato are harder. Caption inputs still contain visual evidence translated by another model, so this is not a pure language-prior control. Results depend on the chosen model, captioner, and temporal sampling, and do not establish that a benchmark universally needs no vision. Near-zero VSC scores prevent meaningful comparisons among these input modes.

Existing benchmarks overwhelmingly focus on linguistic understanding and semantic perception while neglecting the more advanced spatial and temporal reasoning required for supersensing.
Complete original figure 3, preserving native PDF vector labels and embedded image detail.

Figure 3: Illustrations of how spatial sensing is conceptualized in current video benchmarks. The left panel features examples from the ‘‘spatial reasoning’’ subcategory of VideoMME [42], including a question regarding gravity from Shutter Authority’s ‘‘What if the Moon Crashed into the Earth?’’ and a question regarding astronaut gear from NASA’s ‘‘Astronaut Bruce McCandless II Floats Free in Space.’’ In contrast, the right panel shows samples from VSI-Bench [148], which highlight visual-spatial reasoning tasks such as object counting, identifying relative directions, route planning, and more.

2.2 VSI-Super : Towards Benchmarking Spatial Supersensing in Multimodal LLMs

VSI-Super tests ordered recall of unusual objects and cumulative counting across concatenated room tours. Its variable durations deliberately stress fixed context windows. These synthetic constructions expose practical memory weaknesses, while their relevance to natural continuous experience remains an empirical question.

Complete original figure 4, preserving native PDF vector labels and embedded image detail.

Figure 4: Illustration of the VSR benchmark’s construction process and format. We use generative models to edit videos by inserting surprising or out-of-place objects into the space. The core task then challenges models to recall the spatial placements of these objects in the correct order of their appearance across arbitrarily long videos.

VSI-Super Recall: Long-horizon spatial observation and recall.

For VSR, annotators insert the same unusual object into four distinct frames and locations, then concatenate the edited room tour with other tours. A multiple-choice question asks for the locations in appearance order. Five durations run from ten minutes to four hours. This is spatially contextualized needle recall, with temporally sparse edited events rather than naturally persistent objects.

VSI-Super Count: Continual counting under changing viewpoints and scenes.

VSC concatenates tours and asks for the total number of a target object category across rooms, with durations of 10, 30, 60, and 120 minutes. Viewpoint changes and repeated sightings within a tour require avoiding duplicate counts. Evaluation uses the inherited mean relative accuracy metric, which should not be read as the percentage of exactly correct counts. Streaming evaluation repeats the question at intermediate times.

Complete original figure 5, preserving native PDF vector labels and embedded image detail.

Figure 5: Overview of the VSC benchmark. The benchmark evaluates counting capabilities on long-horizon, multi-room videos composed of concatenated scenes. Queries are posed at various time points to simulate a streaming question-answering setting.

State-of-the-art models struggle on VSI-Super .

Gemini-2.5-Flash scores 81.5/79.2 on VideoMME/VideoMMMU but 41.5 on one-hour VSR and 10.9 on one-hour VSC. Two-hour inputs exceed its tested context budget. Its counts saturate at small values as true counts grow, consistent with poor extrapolation under this setup; the plots do not independently identify reliance on training priors as the cause.

Complete original table 1, preserving all native vector labels, values, rules, and grouping.
Table 1: Gemini-2.5-Flash results. As a state-of-the-art video understanding model with long-context capabilities, Gemini demonstrates strong performance on general video benchmarks but shows clear limitations towards spatial supersensing.
Original figure

Figure 6: Visualization of Gemini-2.5-Flash’s predictions v.s. ground truth on VSC. The model’s predicted object counts saturate at small constant values and fail to scale with video length or true object counts, indicating limited generalization in counting and reliance on training distribution priors.

How VSI-Super challenges the current paradigm.

An arbitrarily extensible stream cannot be retained verbatim in a fixed window. The authors therefore motivate selective memory and decomposition into local events. The experiments show limitations of the tested sampling and memory strategies, rather than proving that every possible data-scaled multimodal architecture must fail or that prediction error is the only solution.

VSI-Super tasks challenge the belief that scaling alone guarantees progress. VSI-Super tasks demand generalization to new temporal and spatial scales at test time.

3 Spatial Sensing Under the Current Paradigm

The data-driven baseline first strengthens image understanding, then general video understanding, then spatial skills. This provides a strong short-video model before testing whether a separate memory mechanism improves longer streams.

3.1 Base Model Training: Upgraded Cambrian-1

The base replaces Cambrian-1’s visual encoder with SigLIP2-So400M, uses instruction-tuned Qwen2.5, and connects them with a two-layer MLP. The simpler connector trades the earlier deep visual fusion for computational efficiency.

3.2 Spatial Video Data Curation: VSI-590K

VSI-590K targets spatial relationships absent from much general instruction data. It mixes geometry-derived video questions, simulated scenes, and pseudo-annotated real images; those sources have different supervision quality and modalities.

Complete original figure 7, preserving native PDF vector labels and embedded image detail.

Figure 7: VSI-590K data curation pipeline. We collect data from 3D-annotated real and simulated video sources, as well as from pseudo-annotated frames extracted from web videos. We then use diverse templates to automatically generate question–answer pairs for instruction tuning.

Table 2: Data statistics for VSI-590K. We collect data from 10 sources with different video types and annotations to improve diversity.
Complete original table 2, preserving all native vector labels, values, rules, and grouping.

Data curation and processing.

The final dataset contains 590,667 question-answer pairs from 5,963 videos and 44,858 images across ten sources. Annotated indoor scans and egocentric videos supply instance geometry. ProcTHOR contributes 625 simulated traversals, and Hypersim contributes 5,113 images. Around 19K web room tours and robot datasets supply frames processed with detection, segmentation, and reconstruction. Pseudo-labeling is performed per image because full-video reconstruction is noisier.

Question type definition and template augmentation.

Twelve question types cover size, direction, count, distance, and appearance order, with relative/absolute distinctions and camera/object viewpoints. Templates vary wording and measurement units. The resulting mixture emphasizes configuration questions; appearance order requires video and forms a small portion of the corpus.

Table 3: Contributions of Different Data Sources in the VSI-590K Mixture. This table illustrates the impact of different data sources on VSI-Bench performance. The combined dataset, VSI-590K Full Mix, achieves the best overall results. Among individual sources, annotated real video datasets contribute the most significant improvements, followed by simulated videos, and then pseudo-annotated images.
Complete original table 3, preserving all native vector labels, values, rules, and grouping.

VSI-590K data source ablation.

The full source mixture reaches 63.2 on the VSI-Bench ablation, versus 28.5 for its baseline. Annotated real videos help most among individual sources, followed by simulated data and pseudo-annotated images. This supports the particular data mixture. Since annotation quality, content, quantity, and modality differ together, it does not isolate an inherent superiority of all video data over all image data.

Data effectiveness ranks as: annotated real videos >> simulated data >> pseudo-annotated images.

3.3 Post-Training Recipe for Spatial Sensing

Four initializations progress from alignment-only to image instruction tuning, then 429K or 3M video examples. Stronger video initialization improves subsequent spatial tuning. From the strongest base, spatial-only tuning gives 69.2 VSI-Bench but 54.1 VideoMME; mixed tuning gives 65.1 and 61.9. Thus general-video examples mitigate a real specialization tradeoff rather than preserving every capability automatically.

A stronger base model with greater exposure to general video data leads to improved spatial sensing after SFT. Mixing general video data prevents the generalization loss caused by in-domain SFT.
Table 4: Post-training exploration for spatial sensing. We examine four base models with progressively increasing exposure to visual data, from image-only training to extensive video training, and analyze their distinct trends during spatial sensing tuning under two different data recipes. A1: only the connector is trained for image–language alignment; A2: A1 w/. Cambrian-7M image instruction-tuning data; A3: A2 further finetuned on 429K video instruction-tuning samples; A4: A2 further finetuned on 3M video instruction-tuning samples. From A1 to A4, the models show a monotonic improvement in video understanding ability. I-IT and V-IT denote instruction finetuning on image and video data, respectively. Finally, we show that stronger base models yield better SFT performance on spatial sensing tasks.
Complete original table 4, preserving all native vector labels, values, rules, and grouping.

3.4 Cambrian- S : Spatially-Grounded MLLMs

Cambrian-S uses Qwen2.5 backbones of 0.5B, 1.5B, 3B, and 7B parameters. Alignment and image instruction tuning precede general video tuning on CambrianS-3M. The final stage mixes VSI-590K with general video and image data. These model scales describe the language backbones, with a visual encoder and connector added.

Complete original figure 8, preserving native PDF vector labels and embedded image detail.

Figure 8: Overall Cambrian-S training pipeline. Stages 1 and 2 enhance image understanding, stage 3 improves general video understanding, and stage 4 strengthens spatial sensing capability.

Table 5: Comparison of Cambrian-S with other leading MLLMs. Cambrian-S outperforms both proprietary and open-source models across a range of image and video visual–spatial benchmarks and model sizes. For video evaluation, we uniformly sample 128 frames as input. Detailed evaluation settings are provided in Appendix E.
Complete original table 5, preserving all native vector labels, values, rules, and grouping.

3.5 Empirical Results: Improved Spatial Cognition

Evaluation reports standard spatial tests, debiased tests, general benchmarks, and long-video failures. Good short-video spatial scores coexist with weak continual counting and recall.

Table 6: VSI-Bench sub-task breakdown. Best results are bolded. Notably, even without any route planning data in training, Cambrian-S-7B outperforms Gemini-1.5-Pro on this task.
Complete original table 6, preserving all native vector labels, values, rules, and grouping.

Improved spatial cognition.

The 7B model reaches 67.5 on VSI-Bench, versus Gemini-2.5-Pro’s 51.5 and its own stage-3 score of 35.7. Route planning reaches 41.8 despite that question type being absent from the curated training set; it exceeds Gemini-1.5-Pro’s 36.0 but remains below Gemini-2.5-Pro’s 45.9. The authors report disjoint training versus benchmark splits even when dataset sources overlap. These results support transfer to a held-out task type, not unrestricted spatial generalization.

Cambrian-S achieves state-of-the-art spatial sensing performance with robust generalization to unseen spatial question types, while staying competitive in general video understanding.

Robust spatial reasoning on VSI-Bench-Debiased.

VSI-Bench-Debiased reduces opportunities to answer from familiar object-size priors. Cambrian-S-7B falls from 67.5 to 59.9 but remains above Gemini-2.5-Pro’s 49.1. This is additional evidence of visual grounding under the tested debiasing scheme, rather than proof that language shortcuts are entirely absent.

Results on VSI-Super: limitations in continual spatial sensing.

With one-frame-per-second streaming input, base Cambrian-S-7B VSR scores drop from 38.3 at ten minutes to 35.0 at thirty and 6.0 at sixty, then zero at longer durations. VSC is 0.6 at ten minutes and zero thereafter. Uniformly sampling 128 frames from an entire completed video is reported separately because it uses an offline view of the stream and can miss sparse events. These failures motivate explicit memory mechanisms.

Scaling data and models is essential, but alone it cannot unlock true spatial supersensing.
Table 7: Cambrian-S-7B results on VSI-Super. Despite strong performance on VSI-Bench, accuracy on VSR drops sharply from 38.3% (10 min) to 0.0% (>60 min), and VSC completely fails. Note that VSI-Super focuses on continual, streaming evaluation, where uniform sampling 128 frames across the entire video does not align with the online setting; results shown in gray are provided for reference only.
Complete original table 7, preserving all native vector labels, values, rules, and grouping.

4 Predictive Sensing as a New Paradigm

Predictive sensing adds a lightweight latent-frame prediction objective, then uses prediction error as a control signal. The demonstrated functions are memory allocation and boundary detection. Predicting visual features does not by itself establish a coherent 3D world model, causal dynamics, or action-conditioned planning ability.

4.1 Predictive Sensing via Latent Frame Prediction

A two-layer MLP predicts the next frame’s visual representation from the language model’s contextual state. Training adds mean-squared and cosine losses to instruction tuning, using a 290K video subset sampled at 1 FPS for the prediction objective. The connector, language model, and both heads are jointly updated; SigLIP remains frozen. Later experiments still call this modified checkpoint Cambrian-S, so it should be distinguished from the earlier spatial-only model.

Inference: Estimating surprise via prediction error.

At inference, cosine distance between the predicted next-frame feature and the observed feature supplies a surprise score. High error can signal an unusual object or a scene transition, but can also arise from motion, nuisance changes, or model error; it is a learned feature discrepancy rather than calibrated semantic novelty.

Complete original figure 9, preserving native PDF vector labels and embedded image detail.

Figure 9: Training and inference pipeline for the latent frame prediction (LFP) approach. Our model employs a Latent Frame Prediction (LFP) head to predict the next frame in latent space. During training, the LFP head predicts the latent representation of the subsequent video frame. During inference, the model measures surprise by computing the cosine distance between the LFP head’s prediction and the actual latent features of the subsequent frame. The surprise signal exhibits distinct spikes for events such as the sudden appearance of unusual objects and abrupt scene changes. Our predictive-sensing prototype allows Cambrian-S to generalize to longer videos on VSI-Super, outperforming frontier models (e.g., Gemini-2.5-Flash) that rely solely on context length expansion.

Complete original figure 10, preserving native PDF vector labels and embedded image detail.

Figure 10: Surprise-driven memory management framework design. The proposed memory system (a) encodes incoming visual streams, compressing frames with low surprise; (b) performs consolidation when memory is full by dropping or merging the least surprising frames; and (c) retrieves relevant frames during query answering. Color shading (dark→\rightarrowlight) reflects the degree of surprise, with hatched boxes denoting compressed frames and solid boxes representing uncompressed ones.

4.2 Case Study I: Surprise-driven Memory Management System for VSI-Super Recall.

The recall prototype keeps a bounded long-term store and retrieves relevant memories only when a query arrives. Surprise guides retention before the future question is known.

Surprise-driven memory management system.

Incoming frames are encoded with a fixed sliding context window. Low-surprise frame key/value caches undergo spatial downsampling before entering long-term memory. When the token budget is exceeded, consolidation drops or merges memories using surprise. A query then selects top-K relevant frames through query–key feature similarity. This bounds the stored representation, while total processing work still grows with stream duration.

Original figure

(a) VSR results

Original figure

(b) GPU memory usage

Original figure

(c) Surprise comparison

Figure 11: Performance analysis of surprise-driven memory on VSR. (a) Surprise-driven memory allows Cambrian-S to maintain strong performance as video length increases. (b) Surprise-driven memory maintains a stable GPU memory footprint as video length increases. (c) Ablation: Using LFP prediction error as the surprise signal is more robust and consistently outperforms using adjacent-frame similarity.

Results.

The memory variant scores 45.0, 41.7, 40.0, 40.0, and 40.0 on VSR from ten minutes to four hours. It is stable but far from perfect recall. Gemini-2.5-Flash is much stronger at ten minutes, with 86.7, and remains competitive within its supported context; the proposed method’s clearest advantage is sustaining processing beyond that limit. Memory usage remains roughly 20–25 GB over the plotted durations, unlike the growing uncompressed baseline. Figure 11 labels its secondary baseline Gemini-2.0-Flash, while the accompanying prose calls it Gemini-1.5-Flash; that source discrepancy is retained.

Ablation on surprise measurement.

Prediction-error surprise outperforms adjacent-frame SigLIP feature difference under the same memory design. Thresholds are tuned separately for both methods at each duration. The paper does not detail an independent threshold-validation split, so this is a tuned comparison rather than evidence that one fixed threshold generalizes to arbitrary streams.

Predictive sensing provides a more principled approach to modeling the spatiotemporal dynamics of video data than static similarity measures based on per-frame features.

4.3 Case Study II: Surprise-driven continual video segment for VSI-Super Count.

The counting prototype divides the video into local events, answers the counting query for each segment, and sums the intermediate answers. Its benefit comes from both segmentation and explicit numerical aggregation.

Original figure

Figure 12: Illustration of our surprise-driven event segmentation framework for VSC. The model continuously accumulates frame features in an event buffer. When a high-surprise frame is detected, the buffered features are summarized to produce a segment-level answer, and the buffer is cleared to start a new segment. This process repeats until the end of the video, after which all segment answers are aggregated to form the final output. Color shading (dark→\rightarrowlight) reflects the degree of surprise.

Surprise-driven event segmentation.

A large surprise score triggers decoding from the accumulated event memory and resets that memory. This approximately aligns segments with room-tour transitions. The method assumes that summing segment counts is meaningful; revisiting the same physical objects across arbitrary event boundaries would need additional identity handling not demonstrated by concatenated tours.

Original figure

(a) VSC results

Original figure

(b) Different surprise measurement

Original figure

(c) Streaming evaluation

Figure 13: Performance analysis on VSC. (a) Cambrian-S with surprise-driven event segmentation achieves consistently higher and more stable performance across all video lengths compared to Gemini-2.5-Flash; (b) Ablation: prediction error as surprise outperforms adjacent-frame similarity; (c) Streaming evaluation: Although GPT-Realtime and Gemini-Live are marketed as ‘‘live assistants’’, they achieve less than 15% MRA and their performance drops to near zero on long videos, while our method maintains substantially higher performance.

Original figure

Figure 14: Cambrian-S scales to higher ground truth object counts whereas Gemini saturates. Predicted counts are plotted against ground-truth counts for videos of different lengths (10, 30, 60, and 120 minutes). Using surprise-driven segmentation, Cambrian-S’s predicted counts grow approximately linearly with the ground-truth, tracking the y=xy=x perfect-count line (gray dashed), whereas Gemini-2.5-Flash’s predicted counts remain clustered near small values and fail to increase with ground-truth count, indicating early saturation and poor extrapolation to larger counts.

Results.

Surprise segmentation yields VSC MRA of 40.6, 42.0, 35.0, and 34.0 over 10–120 minutes. Ground-truth boundaries improve results further, suggesting imperfect segmentation remains a bottleneck. Predicted counts scale more closely with true counts than Gemini’s, but scatter and systematic errors remain. These scores do not mean exact counting on 34–42% of samples.

Ablation on surprise measurement.

Using prediction error for boundaries beats adjacent-frame similarity at all tested durations after tuning both methods. This isolates the signal within the implemented pipeline more directly than comparisons against whole commercial systems, but does not show that all simpler or task-specific segmentation methods are inferior.

Evaluation in streaming setup.

Streaming evaluation asks the same cumulative-count question at ten timestamps and averages the scores. Cambrian-S obtains about 38 MRA at ten minutes and 28 at two hours; tested Gemini-Live and GPT-Realtime configurations are below 15 at ten minutes and near zero at two hours. This is a benchmark-specific live-input comparison, not an overall assessment of conversational assistants or deployment latency.

Summary.

Both case studies support selective storage and structured accumulation for these long-video tasks. Claims about unbounded human-level or superhuman spatial intelligence remain the paper’s motivating direction, beyond its measured evidence.

5 Related Work

Related work spans multimodal language models, long-stream processing, spatial intelligence, and predictive representations.

Video Multimodal Large Language Models

Video models typically combine pretrained visual representations with language models and instruction tuning. Cambrian-S studies how their data, evaluation, and objectives can be adapted for spatial tasks rather than introducing a wholly independent perception system.

Streaming Video Understanding

Existing approaches use efficient attention, larger context windows, retrieval, and token compression. The proposed contribution is prediction-error-guided memory organization, situated alongside these engineering methods rather than a proof that sequence modeling cannot support continual perception.

Visual Spatial Intelligence

Spatial benchmarks and training methods study object relations, geometry, viewpoint changes, and embodied grounding. VSI-590K and VSI-Super emphasize video-based spatial reasoning and long-duration information retention.

Predictive Modeling

Predictive coding, self-supervised representations, learned world models, and model predictive control motivate anticipating observations. The current prototype predicts latent visual features; it does not implement the control or multi-step simulation capabilities of those broader approaches.

6 Conclusion

The strongest demonstrated outcomes are improved standard spatial benchmark scores and better sustained recall/counting with prediction-guided memory. The authors acknowledge limitations in benchmark diversity, data quality and scale, generalizability, and model design. Synthetic edits, concatenated rooms, small per-duration test sets, tuned thresholds, and incomplete reproduction details constrain broader conclusions; no human-level evaluation or embodied deployment is shown.

Acknowledgments

The work builds on Cambrian-1, uses Google TPU Research Cloud and cloud credits, and acknowledges infrastructure collaborators, manuscript reviewers, fellowships, and research grants.

Appendix

The appendix provides diagnostic results, benchmark construction, data taxonomy, all training stages, detailed model evaluations, and the predictive-memory algorithms. Its complete example figures are retained alongside the edited explanation.

Appendix A Benchmark Diagnostic Test Results

Table 8 reports the input-ablation diagnostics. For example, VideoMME scores 31.2 blind, 53.7 with 32 frames, and 55.3 with frame captions; VSI-Bench scores 17.4, 25.8, and 21.8 respectively. VSR remains below its 25% chance baseline in these image-model tests, and VSC is essentially zero. This cautions against reading differences near the floor as reliable capability rankings.

Table 8: Detailed results of our improved Cambrian-1-7B on video MLLM benchmarks under different evaluation setups.
Complete original table 8, preserving all native vector labels, values, rules, and grouping.

Appendix B VSI-Super Benchmark

VSI-Super supplies two distinct tasks and duration suites. The construction procedures make the intended challenges reproducible in outline while also defining their synthetic scope.

B.1 VSI-Super Recall

VSR measures ordered spatial recall of an edited object in long concatenated tours.

Implementation details.

Annotators insert unusual objects with Gemini-2.0-Flash into four selected frames and record their locations and order. Each of five durations has 60 videos and one question per video. Videos are downsampled to 1 FPS so the edited frames are guaranteed visible to the evaluated sampling scheme. This alignment favors detection of the deliberately sparse events and differs from arbitrary real-world sampling.

Visualization.

Figure 18 retains all four edited-frame sequences and questions. The objects are visually plausible within each frame, but the figure does not establish temporally consistent insertion across neighboring frames. Sparse appearance is directly relevant to the surprise-based memory mechanism.

B.2 VSI-Super Count

VSC measures accumulated counts over concatenated source environments, with a separate intermediate-query setting.

Implementation Details.

The construction sums annotated scene counts, normalizes object categories across ScanNet, ScanNet++, and ARKitScenes, and rebalances quantities toward a more uniform distribution. Each duration has 50 corresponding questions. Stored VSC videos are 24 FPS, while the proposed sensing prototype typically samples them at 1 FPS; these are different stages of the pipeline.

Streaming setups.

Ground truth at ten query timestamps uses each unique object’s first appearance, following Thinking in Space / VSI-Bench. Thus streaming targets change only as new objects become observable, rather than using the final scene total at every time.

Appendix C VSI-590K Dataset

Dataset documentation distinguishes geometry-backed labels from pseudo-labels and specifies the question taxonomy. Selected examples illustrate the pipeline but are not a comprehensive label-quality audit.

C.1 Details of Question Type Definition

Questions are organized by physical attribute, relative versus absolute measurement, reference viewpoint, and whether an image or video is required.

Taxonomy.

Size, direction, count, distance, and appearance order generate twelve combinations of task type. Configuration accounts for about 72.3% of examples, measurement 24.9%, and spatiotemporal order 2.7%. Relative queries compare objects, absolute queries request quantities, and direction/distance queries vary camera or object reference frames. These are deliberately uneven training proportions.

Table 9: Taxonomy of spatiotemporal question types in VSI-590K. Questions are stratified along five axes: attribute type, relative vs. absolute (Rel./Abs.), perspective, modality (V: video, I: image), and group. An example question template is provided for each type.
Complete original table 9, preserving all native vector labels, values, rules, and grouping.
Original figure

Original figure

Figure 15: VSI-590K dataset statistics. QAs are grouped by: question types (left) and task groups (right).

C.2 Detailed QA-Pair Construction Pipeline

A unified scene representation feeds augmented question templates. Ground-truth geometry and inferred geometry enter that same interface through different collection routes.

3D-annotated real videos.

Real annotated sources provide instance counts, 3D bounding boxes, and room measurements. Their training splits are standardized into scene metadata, then converted to question-answer examples. The approach inherits source annotation accuracy and object-category conventions.

3D-annotated simulated videos and images.

ProcTHOR scenes are traversed from randomized agents, whereas Hypersim supplies annotated individual images. Both use the common template machinery. Simulation supplies controlled geometry but differs from real visual and object distributions.

Unannotated web-crawled real videos.

Algorithm 1 samples and filters frames, rejects invalid categories or insufficient objects, runs Grounding DINO and SAM2, erodes masks, and associates them with VGGT point maps. Sufficient valid points then generate questions. Per-frame processing reduces temporal inconsistency but does not eliminate detector, segmentation, scale, or reconstruction errors. All original pseudocode, thresholds-as-symbols, and comments remain visible.

C.3 Additional Ablation Study

Removing any task group lowers overall VSI-Bench scores from the full mixture’s 63.2. Removing configuration reduces route planning from 36.6 to 27.3; removing measurement gives 35.1, and removing spatiotemporal examples gives 32.5. This supports complementary training groups, without isolating whether differences arise from type, quantity, or difficulty.

Table 10: Ablation study on VSI-590K task groups. We study models’ performance change when one certain task group are omitted from the training data.
Complete original table 10, preserving all native vector labels, values, rules, and grouping.

C.4 Examples of VSI-590K

Figures 19–25 preserve real-video, simulated-video, and pseudo-annotated-image examples in full. Some source headings or templates are imperfect: Figure 19 labels a size-comparison question “Absolute Distance,” and the bathroom example in Figure 23 includes an office-object distance question. These are retained source artifacts and caution against treating selected examples as proof of uniformly clean labels.

Algorithm 1 QA generation pipeline for unannotated web-scrawled videos
Original algorithm 1 body, preserving every line, indentation, symbol, and comment.

Appendix D Cambrian- S Implementation Details

Implementation details specify the multimodal modules, data mixtures, resolution and token budgets, and distributed training infrastructure.

D.1 Model Architecture

A frozen SigLIP2-So400M visual encoder feeds a GELU two-layer connector and instruction-tuned Qwen2.5 decoder. The connector replaces Cambrian-1’s spatial vision aggregator. The original visual encoder’s pretraining objectives are background to the architecture, not objectives retrained from scratch here.

D.2 Training Data Mixture

Stages one and two reuse Cambrian alignment and image instruction datasets. CambrianS-3M combines roughly three million open-source video QA examples, including general captions, fine-grained questions, ego-centric material, temporal localization, and interface interactions. Figure 16 retains the complete source composition. The final stage adds spatial specialization while retaining sampled general examples.

Original figure

Figure 16: General video instruction tuning datasets of CambrianS-3M, used in Cambrian-S stage 3 & 4 training.

D.3 Training Recipe

All four stages keep the visual encoder frozen. The connector alone learns alignment initially; subsequent stages also update the language decoder.

Table 11: Training configuration for stage 1 and stage 2.
Complete original table 11, preserving all native vector labels, values, rules, and grouping.

Stage 1: Vision-language alignment.

Alignment uses Cambrian-Alignment-2.5M, padded 384×384 images, a 2,048-token sequence limit, learning rate 10⁻³, and batch size 512. Only the connector is trained.

Stage 2: Image instruction tuning.

Image instruction tuning uses Cambrian-7M and an AnyRes strategy with up to nine 384×384 crops. The sequence limit expands to 8,192, learning rate drops to 10⁻⁵, and batch size is 256. The connector and language decoder are updated.

Table 12: Training recipe for Cambrian-S stage 3 and stage 4.
Complete original table 12, preserving all native vector labels, values, rules, and grouping.

Stage 3: General video instruction tuning.

General video tuning mixes CambrianS-3M with sampled image examples. Each video supplies 64 uniformly sampled 384×384 frames, spatially reduced to 64 visual tokens per frame. Image handling follows the earlier resolution strategy.

Stage 4: Spatial video instruction tuning.

Spatial tuning mixes VSI-590K with 590K general-video examples and 120K image examples. It uses 128 frames and a 16,384-token sequence limit. The preserved configuration table lists learning rate 10⁻⁵ and global batch size 256. This recipe differs from the 1-FPS additional latent-prediction training subset.

D.4 Infrastructure

TPU v4 Pods, TorchXLA/GSPMD sharding, and Pallas FlashAttention support long-sequence training. The 7B model reaches a 16,384-token training length on a v4-512 Pod. These infrastructure details establish the reported setup, not a comparison of total training cost or energy efficiency.

Appendix E Cambrian- S Additional Results

Additional results expose benchmark-specific changes across stages, model sizes, image/video data proportions, and spatial versus general tuning.

E.1 Detailed Evaluation Setups

Standard image evaluation allows up to nine aspect-preserving 384×384 crops. Standard video evaluation uses 32 frames for stage-1/2 checkpoints, 64 for stage 3, and 128 for stage 4. Consequently, across-stage comparisons change both trained capabilities and available visual input; VSI-Super uses separate streaming protocols.

E.2 Detailed Performance on Image and Video Benchmarks

Tables 13–14 retain every checkpoint and benchmark score. The 7B spatial stage raises VSI-Bench from 35.7 to 67.5, while general results are mixed: VideoMME rises from 62.8 to 63.3, Perception Test falls from 70.8 to 69.8, and Tomato falls from 30.3 to 27.9. Small differences from headline tables are preserved rather than harmonized. “Without sacrificing general capabilities” is therefore an aggregate characterization, not improvement on every test.

Table 13: Detailed results of Cambrian-S checkpoints on image MLLM benchmarks.
Complete original table 13, preserving all native vector labels, values, rules, and grouping.
Table 14: Detailed results of Cambrian-S checkpoints on video MLLM benchmarks.
Complete original table 14, preserving all native vector labels, values, rules, and grouping.

E.3 Contributions from Image-based and Video-based Instruction Tuning

Models trained on 1M, 4M, or 7M images receive 25–100% of a video dataset. Even a quarter of the video data improves the tested image-only baselines, but further scaling is not uniformly beneficial. At full video data, the 7M-image model wins five of nine benchmarks. Data distribution and task alignment matter alongside size.

Table 15: Video MLLM performance trained with different proportions of image and video data.
Complete original table 15, preserving all native vector labels, values, rules, and grouping.

E.4 On the Trade-off between Spatial Sensing and General Video Understanding

Spatial-only tuning produces stronger VSI-Bench scores; mixed tuning better preserves general video performance across the plotted scales. The spatial gap narrows for larger models. The prose mentions a 1B scale here, while the model family and plotted labels use 1.5B; the original source displays are retained.

Original figure

Figure 17: On the trade-off between spatial-sensing and general video understanding.

Appendix F Predictive Sensing

Predictive sensing combines auxiliary next-feature training with two different memory applications. Full original pseudocode is retained because scheduling, budget handling, and aggregation affect interpretation.

F.1 Latent Frame Prediction Implementation Details

The prediction head maps language-model hidden states to visual-feature dimension, with a tunable auxiliary loss contribution.

Latent frame prediction head.

Algorithm 2 gives the 7B head as Linear(3584,3584), GELU, and Linear(3584,1152), with biases. The 1152-dimensional output matches SigLIP2 features. The native code rendering preserves its exact nested structure and indentation.

Algorithm 2 Latent frame prediction (LFP) head architecture (in PyTorch style).
Original algorithm 2 body, preserving every line, indentation, symbol, and comment.

On the balance between LFP and instruction tuning losses.

The default auxiliary coefficient is 0.1. Table 16 shows VSI-Bench changing from 67.5 without prediction loss to 66.1 at 0.1, 60.8 at 0.5, and 56.6 at 1.0. Prediction training therefore has a measurable specialization cost, with stronger weighting generally harming the reported spatial score. The table does not itself quantify long-memory gains at every coefficient.

Table 16: Evaluation results across different benchmarks with varying LFP loss weights. Our default setup (0.1 loss coefficient) is highlighted in gray.
Complete original table 16, preserving all native vector labels, values, rules, and grouping.

F.2 Memory Framework Design for VSI-Super Recall

Recall memory has sensory, long-term, and query-time working stores. Compression, consolidation, and retrieval operate on internal transformer caches rather than an external caption index.

Basic memory units.

Per-layer key/value pairs are the retained memory units. This lets retrieval reuse internal language-model attention representations, while making storage dependent on model layers and token counts.

Streaming sensing.

Each frame is visually encoded and then contextualized by the language model using a bounded recent sensory window. Frames leaving that window are candidates for long-term retention. The appendix denotes the window symbolically rather than supplying a complete fixed hyperparameter table.

Surprise-based memory compression.

A low surprise score triggers factor-two spatial downsampling of the outgoing frame’s cache. This selectively reduces stored detail. The prose assigns a surprise score to each frame, while Algorithm 3 writes the current score alongside a dequeued older frame; their timing correspondence is not fully specified in the paper.

Surprise-based memory consolidation.

An example long-term budget is 32,768 tokens. Consolidation repeatedly removes low-priority memories until below budget. Three strategies are discussed: forgetting the oldest, forgetting the least surprising, or combining least-surprising removal with adjacent surprising-memory merges. Exact default choices and all thresholds are not comprehensively tabulated.

Retrieval.

For each transformer layer, query features are compared by cosine similarity with cached frame keys, treating multihead features as one vector. The top-K frames provide key/value context for answering. Retrieval is query-conditioned after passive storage, so information discarded earlier cannot be recovered simply by selecting a different query.

Algorithm 3 Memory framework design for VSI-Super Recall.
Original algorithm 3 body, preserving every line, indentation, symbol, and comment.

F.3 Agentic Framework Design for VSI-Super Count

Algorithm 4 accumulates older frames in event memory, answers on high surprise, clears that store, and sums intermediate answers. Its pseudocode does not explicitly flush the final event or remaining sensory window at end-of-stream, and the prose does not resolve that edge case. The reported implementation may handle it elsewhere, but the paper alone does not establish how. Summation also assumes event counts can be combined without cross-event duplicate identities.

Algorithm 4 Agentic framework design for VSI-Super Count task.
Original algorithm 4 body, preserving every line, indentation, symbol, and comment.

F.4 Comparisons with Existing Long-video Methods

All compared memory strategies use the same LFP-tuned Cambrian-S-7B. MovieChat uses long-/short-memory budgets of 64/16; Flash-VStream omits its learned abstract-memory module because that would require dedicated training. Their VSC scores are zero, while the proposed agentic method scores 40.6–34.0 across durations. The controlled backbone is useful, but the partial Flash-VStream adaptation is not a complete reproduction of its original trained system. VSR comparisons favor the proposed store across all reported lengths.

Table 17: Compare our framework with existing long-video methods on VSI-Super.
Complete original table 17, preserving all native vector labels, values, rules, and grouping.
Complete original figure 18, including all panels and question boxes.
Figure 18: More examples of our VSI-Super Recall benchmark. Note that only edited frames are visualized.
Complete original figure 19, including all panels and question boxes.
Figure 19: Examples of VSI-590K (Annotated Real Video).
Complete original figure 20, including all panels and question boxes.
Figure 20: Examples of VSI-590K (Annotated Real Video).
Complete original figure 21, including all panels and question boxes.
Figure 21: Examples of VSI-590K (Annotated Real Video).
Complete original figure 22, including all panels and question boxes.
Figure 22: Examples of VSI-590K (Annotated Simulated Video).
Complete original figure 23, including all panels and question boxes.
Figure 23: Examples of VSI-590K (Annotated Simulated Video (Frame)).
Complete original figure 24, including all panels and question boxes.
Figure 24: Examples of VSI-590K (Unannotated Real Video (Frame)).
Complete original figure 25, including all panels and question boxes.
Figure 25: Examples of VSI-590K (Unannotated Real Video (Frame)).
Concept demonstration

Open image ↗