Literature
Display
← All papers

TRACER: A Multimodal Harness for Patient Care in Hospitals

Zane Durante · Paul Tang · Keshigeyan Chandrasegaran · Amit Kaushal · Shrinidhi Kowshika Lakshmikanth · Feolu Kolawole · June (Yu Jun) Zheng · Yifan Geng · Shobhit Agarwal · Kevin Schulman · Juan Carlos Niebles · Ehsan Adeli · Arnold Milstein · Li Fei-Fei

Preprint posted · arXiv (preprint) ↗

Healthcare AI16 min text · CondensedOriginal paper ↗
Contents

Reading guide

This edition covers all 37 pages, including the full appendix. It preserves every figure/table, all ten de-identified example questions and choices/checkmarks, eight complete prompts/tool instructions and the original mathematical definitions. The source’s one-line-high SVG prompt boxes are replaced by their exact full downloadable text, retaining whitespace. The first three authors contributed equally; June (Yu Jun) Zheng’s byline is retained with the imported spelling searchable.

At a glance

TRACER is a training-free harness for retrospective reasoning over ICU video and electronic records. Frozen models build minute-level action/caption records, combine them with medication logs into hourly reports, and let an agent query those summaries, the record and targeted video. Perception reaches 84.8% accuracy/68.6 micro-F1 on 397 selected clips. With the same Gemini 3.1 Flash Lite backbone, QA rises from the strongest baseline’s 49.3% to 61.0%; the best agent reaches 76.9%. Those QA results concern 144 questions from only two stays. The separate 280-patient behavioral study reports associations with ever-positive delirium screens, not a validated diagnosis/prediction system.

1 Introduction

A whole ICU stay exceeds practical dense-video context, while clinically relevant events range from brief movements to multi-day patterns and require record context. TRACER separates reusable background abstraction from per-question evidence gathering. Its taxonomy, tools and prompts can be shared without releasing patient recordings or fitting model parameters to them. The 397-clip, two-stay QA and 280-patient cohort analyses address different questions and should keep their denominators separate.

Complete original Figure 1: Overview of TRACER. TRACER equips an agent model with hierarchical behavioral representations and tools for understanding multi-day ICU patient trajectories. Vision–language models analyze one-minute bedside video segments to detect atomic actions defined by a clinically informed taxonomy and generate descriptive captions. These outputs are combined with concurrent medication records to generate structured hourly behavioral reports. Through report, timeline, video, and EHR tools, the agent iteratively retrieves evidence and examines relevant video segments to answer retrospective clinical questions. Our evaluation suite assesses atomic action detection on 397 annotated one-minute clips and longitudinal clinical reasoning on 144 multiple-choice questions spanning 264 hours of continuous video.
Figure 1: Overview of TRACER. TRACER equips an agent model with hierarchical behavioral representations and tools for understanding multi-day ICU patient trajectories. Vision–language models analyze one-minute bedside video segments to detect atomic actions defined by a clinically informed taxonomy and generate descriptive captions. These outputs are combined with concurrent medication records to generate structured hourly behavioral reports. Through report, timeline, video, and EHR tools, the agent iteratively retrieves evidence and examines relevant video segments to answer retrospective clinical questions. Our evaluation suite assesses atomic action detection on 397 annotated one-minute clips and longitudinal clinical reasoning on 144 multiple-choice questions spanning 264 hours of continuous video.

2 Harness Design

The harness consists of accessible representations, evidence tools and a bounded reasoning loop. It continuously constructs a hierarchy, then retrieves selected evidence per question rather than placing every raw observation in one prompt. The complete visual overview remains.

Complete original Table 1: MCQ performance on TRAJECTORY-144. QA accuracy (%) is shown by question type and is averaged over 5 gold-answer placements. TRACER rows share an identical harness (prompts, taxonomy, and temporal hierarchy) and differ only in the agent model. Best per column is in bold, and second best is underlined. The video tool is served by the agent model, or by Gemini 3.1 Flash Lite where marked † .
Table 1: MCQ performance on TRAJECTORY-144. QA accuracy (%) is shown by question type and is averaged over 5 gold-answer placements. TRACER rows share an identical harness (prompts, taxonomy, and temporal hierarchy) and differ only in the agent model. Best per column is in bold, and second best is underlined. The video tool is served by the agent model, or by Gemini 3.1 Flash Lite where marked †.
Accessible table, original values and citation links
Video EHR Cross-modal Avg.
No video
Random 20.1 20.0 20.0 20.1
Text-only LLM, Gemini 3.1 Flash Lite 32.3 26.9 25.0 30.9
Long-context VLMs
Qwen3-VL 8B, 475 frames 31.6 10.0 47.5 28.6
Qwen3-VL 8B, 950 frames 36.3 13.8 50.0 33.0
Gemini 3.1 Flash Lite, 950 frames 36.5 14.6 25.0 31.9
Gemini 3.1 Flash Lite, 475 frames + EHR 41.8 83.8 40.0 49.3
Socratic models (HourVideo)
Qwen3-VL 8B 33.7 22.3 20.0 30.9
Qwen3.5 9B 30.4 18.5 10.0 27.1
Gemini 3.1 Flash Lite 36.8 16.2 47.5 33.7
Gemini 3.1 Flash Lite + EHR 37.8 76.2 42.5 45.0
TRACER harness (ours)
   w/ gpt-oss-20b† 50.3 93.1 40.0 57.4
   w/ Qwen3.5 9B 49.5 95.4 52.5 57.9
   w/ Gemini 3.1 Flash Lite 53.9 96.2 45.0 61.0
   w/ Gemma-4 31B† 61.8 100.0 47.5 67.9
   w/ Gemini 3.5 Flash 72.0 99.2 72.5 76.9

2.1 Problem Formulation

A stay contains paired one-minute camera clips and timestamped clinical events. The task is evidence-supported answering anywhere in that timeline. Median ICU stay is 3.3 days, with stays approaching two weeks. Token budget and protected-data sharing constraints motivate a frozen-model hierarchy. These are system-design requirements; they do not establish clinical effectiveness outside the reported retrospective evaluation.

Original formal definition and mathematical context

We model an ICU stay as 𝒮=({Vt(1),Vt(2)}t=1T,ℰ)\mathcal{S}=\big(\{V^{(1)}_{t},V^{(2)}_{t}\}_{t=1}^{T},\,\mathcal{E}\big), where Vt(1)V^{(1)}_{t} and Vt(2)V^{(2)}_{t} are the one-minute clips from two wall-mounted cameras at minute tt, TT is the length of the stay in minutes, and ℰ\mathcal{E} is the set of timestamped events in the electronic health record (medication administrations, bedside assessments, vital signs, etc.). In our cohort the median stay is 3.3 days (T≈4,800T\approx 4{,}800) and stays run up to two weeks (T≈20,000T\approx 20{,}000). Given a free-text clinical question qq, the task is to return an answer supported by evidence that may lie anywhere in 𝒮\mathcal{S}, in either modality, and at a temporal scale ranging from a moment in time to the entire stay. There are two primary constraints that motivate our system design. First, a single day of video from one camera sampled at 1 FPS is approximately an order of magnitude past the context length of most existing models, which generally range from 256K to 1M tokens (Bai et al., 2025; Comanici et al., 2025) (see Appendix A.6 for more details). Second, a model whose weights are fitted to protected health information cannot be easily shared with external institutions. Thus, we design our harness so that all problem-specific structure and performance gains live in components that can be publicly released (our activity taxonomy, timeline structure and construction, aggregation tools, and model prompts).

Complete original Figure 2: Per-Category Breakdown of Atomic Action Performance. (a) We plot the per-category F1 scores of the atomic actions used by TRACER. (b) We show the relative frequency of predicted atomic actions with respect to delirium status across a cohort of 280 patients spanning ∼ 60,000 \sim 60,000 hours of video data. Delirium is defined as any positive CAM-ICU assessment during the stay ( n = 53 n=53 delirium, n = 227 n=227 non-delirium). Other torso movements and clinician restraining contained too few positive samples to be evaluated accurately and are therefore excluded from both panels.
Figure 2: Per-Category Breakdown of Atomic Action Performance. (a) We plot the per-category F1 scores of the atomic actions used by TRACER. (b) We show the relative frequency of predicted atomic actions with respect to delirium status across a cohort of 280 patients spanning ∼60,000\sim 60,000 hours of video data. Delirium is defined as any positive CAM-ICU assessment during the stay (n=53n=53 delirium, n=227n=227 non-delirium). Other torso movements and clinician restraining contained too few positive samples to be evaluated accurately and are therefore excluded from both panels.

2.2 Harness Architecture

Three frozen roles perform perception, hourly reporting and agent reasoning. The first two create reusable records; the agent runs once per question. Swapping roles is allowed, but end-to-end accuracy also depends on their error propagation and the model serving the video query tool.

2.3 Minute-Level Clinical Event Records

Each minute becomes a 15-label action vector with per-view captions. The action taxonomy supplies consistent variables for aggregation, while captions retain details outside the categories.

Taxonomy.

Clinicians, nurses and researchers iteratively defined interaction, head, torso, arm and leg categories. Labels can co-occur. They describe visible actions, not inferred pain, intention or diagnosis. The taxonomy and full perception prompt remain.

Minute-Level Video Inference.

Each camera minute is sampled at 1 fps, giving sixty frames. Predictions merge with logical OR across cameras, while both captions remain. An uncertain label becomes negative in the stored timeline, but benchmark abstentions are excluded from metrics. Appendix A.3 quantifies the difference; this merging/abstention policy matters when interpreting aggregate behavior frequencies.

2.4 Hourly Behavioral Reports

An hourly model receives sixty minute records and concurrent medication events, then produces movement, interaction, significant-moment, medication, observational alignment and narrative sections. Timestamp citations aid inspection. The ten-report audit finds 84.3% of claims supported by the input records; these reports remain fallible summaries rather than independent clinical evidence.

2.5 Tool-Augmented Trajectory Reasoning

Report, timeline, video and EHR tools expose different temporal resolutions. An agent can request new VLM analysis of a specific minute from both views, sampled at 5 fps for Gemini or 2 fps for open-weight models. It gets at most 25 tool calls. Forced guessing is an MCQ evaluation rule; the clinical interface can say evidence is insufficient.

3 Data and Evaluation

Evaluation separates local action recognition, longitudinal question answering and cohort-level behavioral associations. The hospital data are private, and the released harness does not make those patient records public.

3.1 ICU Dataset

Stanford collected paired-camera 1080p/15-fps recordings in eight cardiovascular/medical ICU rooms during August 2021–June 2025. Sequential clinical and technical filters select 280 of 2617 patients, spanning about 60,000 camera-hours. This is a single-center, sedative/analgesic-exposed, non-neurological-admission cohort with stringent continuous-camera requirements, rather than a representative sample of all ICU admissions.

3.2 Perception Benchmark

The perception benchmark contains 397 selected one-minute clips from 26 stays, covering fifteen labels. One annotator labels each clip and another verifies/corrects it. A separate blinded twenty-clip reannotation audit gives 92.7% agreement and κ=.81. Reported model variability comes from five runs on this fixed benchmark, not five independent hospitals or patient cohorts.

Complete original Table 2: Average atomic action detection performance. Micro-averaged across all 15 atomic actions on the 397-clip test set. Model rows are mean ± \pm std across 5 runs. Sampling settings for every model are listed in Appendix A.8 . Best result per column in bold .
Table 2: Average atomic action detection performance. Micro-averaged across all 15 atomic actions on the 397-clip test set. Model rows are mean ±\pm std across 5 runs. Sampling settings for every model are listed in Appendix A.8. Best result per column in bold.
Accessible table, original values and citation links
Model Accuracy F1 Precision Recall
Baselines
Random (Match distr.) 69.9±\pm0.4 44.8±\pm0.8 44.8±\pm0.8 44.7±\pm1.0
Majority Class 77.7 45.2 68.9 33.6
Open-source
Qwen3-VL (8B-Instruct) 78.0±\pm0.1 37.6±\pm0.4 84.8±\pm0.3 24.2±\pm0.3
Qwen3.5 (4B) 78.2±\pm0.3 41.0±\pm0.8 78.3±\pm1.7 27.7±\pm0.6
Qwen3.5 (9B) 79.7±\pm0.5 48.4±\pm2.0 79.4±\pm1.0 34.8±\pm1.9
Gemma-4 (E4B-it) 74.9±\pm0.1 22.5±\pm0.3 72.2±\pm1.6 13.4±\pm0.2
Proprietary
Gemini 2.5 Flash Lite 79.0±\pm0.3 53.0±\pm0.7 68.4±\pm0.9 43.3±\pm0.9
Gemini 2.5 Flash 83.2±\pm0.2 66.5±\pm0.5 71.6±\pm0.6 62.0±\pm0.9
Gemini 2.5 Pro 84.4±\pm0.2 68.1±\pm0.5 76.0±\pm0.4 61.7±\pm0.7
Gemini 3.1 Flash Lite 82.3±\pm0.2 62.2±\pm0.3 74.5±\pm0.7 53.3±\pm0.4
Gemini 3.5 Flash 84.8±\pm0.0 68.6±\pm0.1 78.3±\pm0.3 61.0±\pm0.3

3.3 TRAJECTORY-144: Longitudinal Reasoning Benchmark

TRAJECTORY-144 probes two long patient stays with many questions, rather than broad generalization across patients. Its construction, modality mix, temporal anchors and baseline budgets are detailed in the appendix.

Benchmark Construction.

Annotators construct event timelines, consult clinicians, write questions and revise plausible distractors. The final benchmark has 110 video, 26 EHR and eight cross-modal questions over 264 video hours. Answer placements rotate, but two questions have four choices, so there are 718 scored trials rather than 720. Earlier draft questions inform agent-prompt refinement, while final evaluated wording does not; this is not an untouched patient-level held-out prompt-development split.

Baselines.

Text-only answers test linguistic priors. Long-context models receive sparsely sampled frames, while Socratic baselines read subsampled minute captions. EHR-augmented and two-camera variants reduce modality-access gaps, but temporal coverage, prompts and retrieval differ. Baseline prompt tokens exceed per-question harness tokens; reusable background hierarchy construction is a separate substantial cost.

4 Experiments and Results

The results evaluate perception quality, tool-assisted QA, model-role changes and descriptive group differences. Complete tables keep model-specific settings and small cross-modal denominators visible.

4.1 Minute-level Perception

Best perception is Gemini 3.5 Flash at 84.8% accuracy and 68.6 F1. Gemini 3.1 Flash Lite gives 82.3%/62.2 at the reported lower submission-time cost, motivating its cohort use. Accuracy must be read with imbalanced labels: the majority-class baseline already scores 77.7%. Per-category F1 and recall reveal limits that a single accuracy number obscures.

4.2 Longitudinal Reasoning on TRAJECTORY-144

The main same-backbone comparison is 61.0% TRACER versus 49.3% frames+EHR and 45.0% captions+EHR. The broader best-agent result and fixed-perception experiments use different configurations, retained separately.

Comparison with baselines.

For Gemini 3.1 Flash Lite, the 11.7-point advantage over the strongest baseline has a question-paired bootstrap 95% interval [4.4,18.9]. Resampling questions conditions on the same two stays; it does not quantify patient-population generalization. Several questions remain guessable without patient data, and EHR-only performance is much higher than short-video performance.

Complete original Table 3: Harness ablation. Accuracy (%) averaged over three agent models: Gemini 3.5 Flash, Gemini 3.1 Flash Lite, and Qwen3.5 9B. Single-toolset rows give the agent one tool family; cumulative rows add one family at a time up to the full harness, with gains shown in green. Best per column and block in bold, second best underlined. Additional results in Appendix A.12 and A.13 .
Table 3: Harness ablation. Accuracy (%) averaged over three agent models: Gemini 3.5 Flash, Gemini 3.1 Flash Lite, and Qwen3.5 9B. Single-toolset rows give the agent one tool family; cumulative rows add one family at a time up to the full harness, with gains shown in green. Best per column and block in bold, second best underlined. Additional results in Appendix A.12 and A.13.
Accessible table, original values and citation links
Video EHR Cross-modal Avg.
Agent tools <<5 min 5–30 min >>30 min Record only Video + EHR
Single toolset performance
Timeline only 27.5 43.3 60.7 30.3 25.0 43.7
Reports only 33.7 52.5 47.4 33.3 43.3 42.1
EHR only 32.9 31.7 41.2 96.4 60.0 49.3
Video only 39.8 50.8 38.3 25.1 27.5 37.0
Cumulative toolset performance Δ\Delta
w/ timeline 27.5 43.3 60.7 30.3 25.0 43.7 0.0
+ reports 30.5 55.4 63.8 31.3 31.7 47.6 +3.9
+ EHR 31.7 51.6 64.1 97.2 61.7 61.1 +17.4
+ video (TRACER) 39.2 57.1 69.2 96.9 56.7 65.3 +21.6

Harness ablation.

Across three agents, the full harness averages 65.3%, versus 49.3% for the strongest single family. Timeline/report/EHR/video additions average 43.7→47.6→61.1→65.3; the displayed +17.4 and +21.6 gains are relative to the initial timeline setting. Benefits vary by agent and question type: adding video lowers Qwen3.5 9B’s overall 59.2→57.9 in the per-model table, so monotonic gains are not universal.

Agentic performance.

Fixing background records/reports and all video queries to Gemini 3.1 Flash Lite separates agent reasoning from additional perception. Gemma-4 31B reaches 67.9%, Qwen3.5 9B 61.6% and Gemini 3.5 Flash 70.7%. With its own stronger video tool the latter reaches 76.9%. Thus the best-agent table does not vary only text reasoning: the video-tool model changes unless explicitly marked.

Complete original Table 4: Comparison across agent models with fixed perception capabilities. Accuracy (%) on TRAJECTORY-144 with Gemini 3.1 Flash Lite used for all video tool calls, timeline, and report construction. Best per column in bold, second best underlined.
Table 4: Comparison across agent models with fixed perception capabilities. Accuracy (%) on TRAJECTORY-144 with Gemini 3.1 Flash Lite used for all video tool calls, timeline, and report construction. Best per column in bold, second best underlined.
Accessible table, original values and citation links
Video EHR Cross-modal Avg. Calls
Agent model <<5 min 5–30 min >>30 min Record only Video + EHR
Proprietary
Gemini 3.5 Flash 35.8 57.5 80.3 98.5 77.5 70.7 9.5
Gemini 3.1 Flash Lite 36.4 52.5 63.7 96.2 45.0 61.0 8.9
Open-weight
gpt-oss-20b 32.1 41.2 62.5 93.1 40.0 57.4 3.7
Qwen3.5 9B 35.2 45.0 67.3 95.4 50.0 61.6 9.0
Gemma-4 31B 40.0 61.2 73.7 100.0 47.5 67.9 5.4

4.3 Cohort-Level Behavioral Signal

Patients with any positive CAM-ICU assessment (53) are compared with 227 without a documented positive screen. Minute-weighted ratios are .41 for non-medical object use, .46 for daily-living leg movement, .48 for object exchange and .60 for gesticulation; clinician contact is 1.20. Purposeful activity’s per-patient median is .053 versus .146 (Mann–Whitney p=1.3e-8). These unadjusted retrospective predicted-behavior associations pool whole stays and are not delirium-period classification, causation or prospective detection evidence.

5 Related Work

Related work spans ambient hospital sensing, long-video memory/retrieval and clinical agents. TRACER combines paired bedside video with asynchronous structured records through a shareable prompted hierarchy.

Ambient intelligence in hospitals.

Prior inpatient activity systems train task-specific models on annotated patient data. TRACER instead composes frozen models and releases its taxonomy/prompts/tools. That avoids learning new parameters from this cohort; protected inputs still require the study’s authorized processing environment.

Long-video understanding.

Caption composition, retrieval, backtracking and week-scale egocentric memories address long video. The ICU setting additionally links bedside behavior with records. The result concerns this benchmark and implementation, rather than every possible optimized long-context/retrieval baseline.

Clinical agents.

Existing clinical agents query text/structured records or medical images. This harness adds days of bedside video and multi-resolution evidence access. It is evaluated for retrospective answering and descriptive cohort analysis, without a prospective clinical-outcome trial.

6 Discussion

The discussion emphasizes useful but noisy perception, improved evidence retrieval and potentially meaningful behavioral associations. It also identifies report errors, restricted cohort coverage and deployment prerequisites.

Key findings.

Same-backbone QA improves by 11.7 points, and fixed-perception results show that open-weight agents can reason effectively over the harness. This does not remove the perception gap or demonstrate equivalent entirely on-premise end-to-end performance, because several open-weight-agent rows retain proprietary perception.

Behavioral signal and delirium.

Reduced purposeful movement is consistent with the cited delirium phenomenology, but the paper explicitly treats its cohort results as hypothesis-generating associations. Sedation, ventilation, length of stay, label timing and model errors complicate interpretation. No predictive sensitivity/specificity or calibrated delirium diagnostic threshold is established.

User Interface.

A bedside-video viewer displays action timelines, chat, tool traces and clickable timestamp citations. Clinician feedback informs its design. The intended use is retrospective querying; the study does not report a controlled interface-usability trial or an evaluated continuous-alert workflow.

Limitations.

Limits include one hospital, only two QA trajectories, intermittent delirium screens, error propagation and hallucinated/incorrect hourly claims. Consent, staff surveillance and fairness require further work. These are the paper’s stated limits, not evidence that those issues were quantitatively resolved.

Reproducibility.

The released harness repository is the stated source for tools, taxonomy, prompts and inference/evaluation code. Patient recordings and linked EHR remain unavailable publicly. Re-instantiating the hierarchy elsewhere still requires compatible data and local evaluation.

AI use statement

The authors disclose AI assistance with writing, formatting, code, analysis and some question/distractor wording. Humans set the research/taxonomy and produced annotations, with clinician consultation and author verification. This statement remains distinct from the claim that no model role was trained on the protected cohort.

Acknowledgments

The acknowledgment credits Stanford clinical/engineering staff, Schmidt Futures, Stanford health/medicine support, HAI Google Cloud credits, NAIRR and Oak Ridge computing. All forty references remain with their citation links.

Appendix A Appendix

The complete appendix supplies examples, interfaces/prompts, hierarchy details, benchmark construction, cohort selection, computational costs, audits, ablations and statistical sensitivity analyses.

A.1 TRAJECTORY-144 Example Questions

All ten de-identified questions remain in their original colored boxes, with every choice and correct-answer checkmark. They cover two examples each for video moments, video windows, long spans, EHR and cross-modal reasoning. These are examples from the same two stays, not additional patients or an independent test set.

Complete original de-identified example question 1, all answer choices and printed correct-answer checkmark.
Original selectable question and answer choices
Video moment (<<5 min)   stay B
At [time] on day 3 of the stay, what clinical object was removed from the patient? (A) An SCD sleeve (B) An endotracheal tube (C) The ventilator (extubation) (D) A heel protector boot ✓ (E) A central line
Complete original de-identified example question 2, all answer choices and printed correct-answer checkmark.
Original selectable question and answer choices
Video moment (<<5 min)   stay B
From the observable video, around [time] on day 1 of the stay, what kind of bedside imaging was performed? (A) An MRI (B) A PET scan (C) An ultrasound scan ✓ (D) A CT scan (E) An X-ray
Complete original de-identified example question 3, all answer choices and printed correct-answer checkmark.
Original selectable question and answer choices
Video window (5–30 min)   stay B
Over the 10 minutes from about [time] on day 2 of the stay, the patient was predominantly: (A) Being handled by staff at the bedside (repositioning, linen or line adjustments), with only slight movements of her own (B) Completely still, with no head, eye, or limb movement at all ✓ (C) Awake and talking with staff, gesturing with her hands (D) Still except for occasional small hand or finger movements (E) Repeatedly moving her arms and head on her own with no staff at the bedside
Complete original de-identified example question 4, all answer choices and printed correct-answer checkmark.
Original selectable question and answer choices
Video window (5–30 min)   stay A
Between about [time] and [time] on day 2 of the stay, no staff were in the room. Which of these most accurately describes the patient’s behavior? (A) Repeatedly pulling at the tubing at her face until a nurse came in (B) Talking with a visitor seated at the bedside (C) Awake and using a smartphone while lying in bed ✓ (D) Sitting up on the edge of the bed (E) Asleep with her eyes closed and her hands under the blanket
Complete original de-identified example question 5, all answer choices and printed correct-answer checkmark.
Original selectable question and answer choices
Long-form video (>>30 min)   stay B
Which of the following sequences correctly describes the order of events during the ICU recording? (A) Patient taken out for tests and later returned →\rightarrow Ultrasound/scanning performed →\rightarrow Family members enter the room →\rightarrow SCD compression sleeves placed on the legs (B) Ultrasound/scanning performed →\rightarrow SCD compression sleeves placed on the legs →\rightarrow Family members enter the room →\rightarrow Patient taken out for tests and later returned (C) Ultrasound/scanning performed →\rightarrow Family members enter the room →\rightarrow Patient taken out for tests and later returned →\rightarrow SCD compression sleeves placed on the legs ✓ (D) SCD compression sleeves placed on the legs →\rightarrow Ultrasound/scanning performed →\rightarrow Family members enter the room →\rightarrow Patient taken out for tests and later returned (E) Family members enter the room →\rightarrow Ultrasound/scanning performed →\rightarrow Patient taken out for tests and later returned →\rightarrow SCD compression sleeves placed on the legs
Complete original de-identified example question 6, all answer choices and printed correct-answer checkmark.
Original selectable question and answer choices
Long-form video (>>30 min)   stay B
How did the patient’s level of voluntary movement and interaction before being taken out for tests compare to after returning to the room? (A) The patient did not visibly interact before the tests but later spoke with clinicians after returning to the room. (B) The patient appeared active and responsive both before and after the tests. (C) The patient was mostly motionless before the tests but later sat up in a slanted position in the bed after returning to the room. (D) The patient remained largely motionless both before and after the tests, with no clear voluntary movement observed. ✓ (E) The patient spoke with family members before being taken for tests and later spoke with clinicians after returning to the room.
Complete original de-identified example question 7, all answer choices and printed correct-answer checkmark.
Original selectable question and answer choices
EHR   stay A
Besides sedatives and pain medication, the patient received a class of scheduled medications throughout the stay. Which class? (A) Antibiotics (B) Blood thinners (anticoagulants) (C) Corticosteroids (steroids) (D) Insulin (E) Anti-seizure (antiepileptic) medications ✓
Complete original de-identified example question 8, all answer choices and printed correct-answer checkmark.
Original selectable question and answer choices
EHR   stay A
The first CAM-ICU delirium assessment, on the first night of the stay, returned what result? (A) Positive (delirium detected) (B) Negative (no delirium detected) (C) Unable to assess (UTA) ✓ (D) Negative on the initial screen, then re-assessed as positive later the same shift (E) The assessment was attempted but the patient’s RASS was too low (−4-4 or −5-5) to proceed
Complete original de-identified example question 9, all answer choices and printed correct-answer checkmark.
Original selectable question and answer choices
Cross-modal   stay A
During the evening of day 2 of the stay, there was a stretch of about ten minutes with no staff in the room during which the patient used a smartphone while lying in bed. At about what time was that stretch, and which medication was administered within 45 minutes after it ended? (A) [time window]; the norepinephrine infusion was stopped afterwards (B) [time window]; no medication followed within 45 minutes (C) [time window]; an albuterol–ipratropium nebulizer treatment followed ✓ (D) [time window]; a norepinephrine rate change followed (E) [time window]; a chlorhexidine oral rinse followed
Complete original de-identified example question 10, all answer choices and printed correct-answer checkmark.
Original selectable question and answer choices
Cross-modal   stay B
During the morning of day 2 of the stay, when did the patient have her period(s) of wakefulness? (A) Multiple short awakenings starting around [time], each lasting only a few seconds (B) Briefly around [time] and again around [time], with sedation in between ✓ (C) Around [time] only, with no other alert periods that morning (D) A single stretch from about [time] through late morning (E) Briefly around [time] only, then remaining sedated through the afternoon

A.2 Tool Interface and Agent Prompt

The full tool schema, agent prompts, MCQ message, cap-exhaustion instruction and video preamble are retained. They describe the evaluated system; forced answer selection applies to scoring rather than clinical-interface abstention.

Tools.

Eleven tools comprise two scaffold tools, four timeline tools, two report tools, two EHR tools and one video tool. Table 5 retains exact names, optional arguments, returns and inclusive/exclusive time-window conventions. Local time and admission-relative hour indices are deliberately distinguished.

Complete original Table 5: The agent’s tools. Exact names, parameters, and return values of the eleven tools, grouped by family; the row labels of Table 3 name the families each configuration adds to the scaffold pair. A question mark marks an optional parameter.
Table 5: The agent’s tools. Exact names, parameters, and return values of the eleven tools, grouped by family; the row labels of Table 3 name the families each configuration adds to the scaffold pair. A question mark marks an optional parameter.
Accessible table, original values and citation links
Tool Parameters Returns
Scaffold (every configuration)
get_patient_metadata pat_id? demographics, ICU admission time, observation window, encounter history
convert_utc_local dt_str, to_timezone the datetime converted between UTC and local time
Timeline (minute-level records)
get_atomic_actions hour_index per-minute active action labels for one hour from admission
query_actions labels?, start_hour?, end_hour? minutes active per label over the range; per-hour counts; per-label per-hour minutes
find_first_occurrence labels, start_hour?, end_hour? first minute where any listed label is active (hour, minute, timestamp)
find_moments_of_interest target_labels?, start_pt?, end_pt?, min_rise_pct?, drug_window_min? rises and falls of a 15-minute rolling activity signal, with the medications given nearby
Report (hourly reports)
list_hourly_reports pat_id? the (date, hour) pairs with a report
get_hourly_report date, hour the six-section report for that hour
EHR
query_medications start_pt, end_pt, name_filter?, class_filter?, max_rows? medication administration events in the window: time, name, action, dose, route, class
query_flowsheet disp_name_filter?, start_pt?, end_pt?, max_rows? flowsheet rows (time, measurement, value, unit); without a filter, the measurement names available
Video
query_video_minute minute_pt, prompt the perception VLM’s answer to the prompt for each of the two camera views of that minute

Agent prompts.

The full prompt directs evidence gathering, time-window checks, filter discovery, output-cap handling and cross-checking. Reduced configurations remove unavailable-tool lines. The exact plain-text payload retains patient placeholders, punctuation and all working rules; no abbreviated paraphrase substitutes for it.

Agent system prompt and tool descriptions, full harness (placeholders for patient metadata)
You are an analytic assistant embedded in a clinical video-viewer app for ICU patients. Answer the multiple-choice question about the patient currently on screen. Investigate with your tools BEFORE deciding; base your answer on what the tools actually return, not on priors.

Tools available to you:
- get_patient_metadata: Get demographics, ICU admission time, observation window length, and encounter history for a patient. Defaults to the patient currently on screen if pat_id is omitted.
- convert_utc_local: Convert between UTC and local time.
- list_hourly_reports: List all hourly markdown reports available for a patient. Each entry has a date (YYYY-MM-DD) and an hour (0-23). Defaults to the current patient if pat_id is omitted.
- get_hourly_report: Fetch the full markdown content of a single hourly report. Reports follow a fixed 6-section structure: Patient Movement Summary, Patient-Clinician Interaction, Significant Moments, Medication Context, Activity-Medication Alignment, Brief Summary. Defaults pat_id to the current patient.
- get_atomic_actions: Return minute-by-minute atomic action detections for one hour of the patient's stay. hour_index=0 is the first hour from ICU admission. Each minute lists currently-active action labels (from the atomic action taxonomy). Defaults pat_id to the current patient.
- query_actions: Aggregate atomic action counts across a range of hours. Returns: (a) total minutes each label was active over the window, (b) per-hour counts of minutes with any selected label active, and (c) per-label per-hour minutes active (dict[label] -> list over the hour range) -- use (c) to answer questions like 'which hour had the most X'. Filter by specific labels or omit to include all. start_hour/end_hour are zero-indexed from ICU start (end_hour is exclusive). Defaults pat_id to the current patient.
- find_first_occurrence: Find the earliest minute where any of the given action labels was active. Returns hour_index, minute (0-59), and UTC timestamp. Prefer this over reading hourly reports or scanning atomic actions when the user asks 'when did X first happen'. Optionally restrict to a range via start_hour/end_hour (zero-indexed from ICU start, end_hour exclusive). Defaults pat_id to the current patient.
- find_moments_of_interest: Detect 'moments of interest' -- rises and falls in a 15-minute rolling signal built from the atomic-action labels you select -- and cross-reference each detected blurp with medication events given nearby. The signal at minute t is the fraction of the previous 15 minutes in which any selected label fired. **You must choose target_labels to match the question.** Examples:
- query_medications: Return medication-administration-record (MAR) events for the current patient in a local-time window. Each event has taken_time_pt, med_name (display name), mar_action (Given / New Bag / Bolus / Stopped / etc.), dose (rate + unit or bolus + unit), route, and drug class (pharmacologic + therapeutic). Use this to answer questions like 'which meds were administered +/-45 min of X?', 'when was the last dose of oxycodone?', or 'what sedatives were running at 03:00?'. TIME: pass local-time strings 'YYYY-MM-DD HH:MM' (24-hour). Windows are inclusive on both ends. OPTIONAL FILTERS: name_filter (case-insensitive substring on the med name) and class_filter (substring on pharmacologic/therapeutic class, e.g. 'OPIOID', 'SEDATI', 'ANTIBIOTIC'). Results are capped at max_rows (default 50).
- query_flowsheet: Query nursing-flowsheet measurements for the current patient: vitals (Heart Rate, Pulse, SpO2, Resp, Arterial Systolic BP, Mean Arterial Pressure, Arterial Diastolic BP, BP), sedation/agitation (Patient RASS Score, Anxiety Level, Pain Scale Type), and nursing assessments (Activity, ADLs, Communication, Ectopy, Cardiac Rhythm, etc.). Each row has recorded_time_pt, disp_name, value, and unit. DISCOVERY: call without disp_name_filter to see which measurement types exist for this patient (list of disp_name + count). QUERY: pass disp_name_filter (case-insensitive substring, e.g. 'RASS', 'Heart Rate', 'SpO2') to fetch the actual measurements. Optionally scope with start_pt / end_pt (local-time 'YYYY-MM-DD HH:MM'). Use this for questions like 'when was the patient most deeply sedated?' (RASS), 'what was the peak heart rate?', 'when did BP swings occur?', 'what was the anxiety/pain trend?'.
- query_video_minute: Look at the ICU surveillance video for ONE specific minute and answer a free-form question about what is visible -- patient movements/reactions, what a clinician is doing, devices/objects, body position, who is in the room.
  Treat its answer as EVIDENCE TO WEIGH, not ground truth: the video reader can miss brief events or misread motion, and the two camera views may disagree. When the answer matters, re-query (a different prompt or an adjacent minute) and cross-check against your other tools before committing.
  This tool is useful for zooming in on a single minute of interest. You can call it on the SAME minute several times with DIFFERENT prompts to probe different details (e.g. first ask what the clinician is doing, then whether the patient moved). It returns BOTH camera angles (left and right view) answered separately.
  TIME: everything here is local time. Pass the minute exactly as it appears in the question, 'YYYY-MM-DD HH:MM' (24-hour, local time); the tool finds the right clip for you and you never convert timezones. The clip is one minute long, so if an event might straddle a boundary, also query the adjacent minute.

If a tool returns an error or nothing useful, adjust and try another approach. Tools that take a patient id default to the current patient -- omit pat_id. You MUST select exactly one option as your single best guess, even if you are unsure or the data is incomplete. Never refuse and never say you cannot determine the answer.

Patient context (from get_patient_metadata):
- Video and activity data (hourly reports, atomic actions, query_video_minute) cover {start} to {end} local time ({n_hours} hours).
- hour_index counts whole hours from {start}: {hour_examples}. To convert a time to hour_index, subtract {start} and take the whole number of hours; write the subtraction down.
- EHR data (query_medications, query_flowsheet) covers the whole hospital admission, {ehr_start} to {ehr_end} local time, which extends beyond the video window.
Every date or time in a question refers to THIS stay in local time. Never guess a year or a date outside these windows.

Working rules (follow them strictly):
1. For "during the stay", "first", "last", "how many", "which day" questions about medications or flowsheet data, query the ENTIRE admission in ONE call (start_pt = {ehr_start}, end_pt = {ehr_end}); for activity data use start_hour 0 to end_hour {n_hours}. Do not step through the stay day by day.
2. Zero rows or zero events never prove absence. Before concluding "none", re-check the dates against the patient context, remove filters, widen the window, and try a shorter substring. If a result carries a "hint" field, follow it.
3. Medications: use name_filter ALONE with a short substring of the drug name (e.g. 'norepi', 'oxycod', 'hepar'). Do not add class_filter unless the question is about a drug class, and never combine the two.
4. Flowsheet measurement names must be discovered, not guessed: call query_flowsheet with NO disp_name_filter first to see the exact names, then filter with a short substring taken from that list (e.g. 'CAM' rather than 'CAM-ICU', 'Temp' rather than 'Temperature').
5. Outputs are capped. If n_events_returned < n_events_total, n_rows_returned < n_rows_total, or a "_truncated" note appears, narrow the filter or time window (or raise max_rows) before drawing any conclusion from a partial list.
6. Prefer totals the tools already computed (per_label_minutes_active, n_events_total, n_rows_total) over adding long arrays by hand. If you must count or add over a long list, do it in blocks of 10 and keep a running total.
7. For what the patient was doing over a several-minute window, first compute the hour_index, then use get_atomic_actions for that hour (minute-level labels) together with the hourly report; use query_video_minute to check the specific minute (and the adjacent minute) when the question is about a single moment.
8. Investigate with at least two different tools or queries before answering whenever the tools can bear on the question, and cross-check them against each other.
9. Keep the reasoning short and end with the line: Answer: <LETTER>
Agent system prompt, text-only configuration
You are answering a multiple-choice question about an ICU patient. You do NOT have access to the patient's video, activity data, or medical records, and you have no tools. Answer as best you can from general reasoning. You MUST select exactly one option as your single best guess, even if you are unsure or the data is incomplete. Never refuse and never say you cannot determine the answer.
User message for MCQ evaluation
QUESTION: {question}

OPTIONS:
{options, one per line as 'A. ...'}

You must choose exactly one option even if uncertain -- never refuse.
End your reply with a line in exactly this form:
Answer: <LETTER>
Forced-choice message (sent when the 25-call budget is reached)
You have used all available tool calls. Based on what you have found so far, commit to your single best-guess answer now. End with a line: Answer: <LETTER>

Video tool.

The video preamble asks for visible subtle motion with timestamps and rejects unsupported inference. Both camera answers return separately for reconciliation. Re-querying is possible, but all such tool calls consume the shared 25-call budget.

Preamble prepended to the agent’s prompt on every video tool call
You are an expert ICU video analyst. Patients here are often sedated, so subtle patient movement (a brief head turn, eye flutter, blink, cough, swallow, finger/limb twitch) is clinically relevant. Review the one-minute clip carefully and list ONLY movements you can actually see, each with an approximate timestamp (mm:ss). Do NOT infer or assume movements that are not clearly visible -- if the patient appears still, say so explicitly. Then answer the custom prompt below.

A.3 Temporal Hierarchy Construction Details

The minute representation stores fifteen binary labels plus two captions. UNABLE TO IDENTIFY becomes negative for timeline construction, whereas perception evaluation excludes abstained labels. Flash Lite abstains once among 29,775 benchmark labels and zero times among 198,810 sampled cohort labels; some other models abstain about 3%, making cross-model metric interpretation sensitive to that policy.

Perception prompt.

The complete fifteen-category prompt and JSON schema remain exact. Its initial instructions request a short summary and the example includes one, while another output-rule line forbids free-text descriptions. This source-level tension is preserved rather than silently edited. Categories concern visible movement/contact and can co-occur.

Perception prompt (system turn)
MAIN INSTRUCTIONS:
Your objective is to analyze video frames extracted from a 1-minute Intensive Care Unit (ICU) hospital patient surveillance video recorded by a fixed, corner-mounted camera.
This is STRICTLY a FINE-GRAINED video understanding task.
You will receive 60 images sampled every 1 second, totaling 1 minute.
Your ONLY task is to decide, for each predefined category below, whether the corresponding patient action or patient-clinician interaction is present in the video. Then write a short 1-2 sentence summary of this video (STRICTLY keep this short) summary.
You must ONLY output a structured JSON object with categorical labels and Short summary.
KEY VISUAL TASKS (TAXONOMY)
PATIENT MOTION -- ARMS/HANDS
  1) Grabbing medical equipment
     - The patient is actively grasping or holding medical devices
       (e.g., ventilator mask, tubing, IV lines, monitor cables).
  2) Using or grabbing non-medical objects
     - The patient is using or holding non-medical items
       (e.g., cup, towel, tissue, phone, food, remote).
  3) Gesticulating
     - The patient is making expressive arm or hand movements while communicating
       (e.g., waving, pointing, hand gestures while speaking).
  4) Adjusting pillows or bedsheets
     - The patient pulls, pushes, or rearranges pillows, blankets, or sheets.
  5) Other arm movements
     - Any noticeable arm/hand motion such as fidgeting, lifting/dropping arms,
       scratching, or non-specific movements.
     - This category can co-occur with other arm/hand categories above.

PATIENT MOTION -- TORSO
  6) Patient changes body position in bed
     - The patient changes posture in bed
       (e.g., lying flat to leaning on one side, sitting up to lying down).
  7) Other torso movement
     - Other significant torso movements
       (e.g., quick or repeated body movements, jerking, agitation-like movements).

PATIENT MOTION -- HEAD/NECK
  8) Patient is looking toward other people in the room
     - The patient's gaze is clearly directed at another person.
  9) Patient turns or moves head/neck
     - Visible head or neck movement
       (e.g., turning left/right, nodding, tilting up/down).

PATIENT MOTION -- LEGS
  10) Leg movements related to mobility
      - The patient is making leg motions associated with activities of daily living
        (e.g., standing up, sitting down, walking, transferring to or from the bed).
  11) In-bed leg movement
      - The patient is making leg movements while they remain in bed
        (e.g., leg fidgeting, repositioning, leg movements under the covers).

PATIENT-CLINICIAN INTERACTIONS
  12) Clinician touches patient or uses tool/device on patient
      - A clinician or other person makes physical contact with the patient
        OR applies a tool or device directly to the patient's body.
  13) Patient communication
      - The patient appears to be speaking, mouthing words, or nodding in a way that
        responds to another person.
  14) Object exchange
      - An item is being handed from a person to the patient or from the patient
        to another person (e.g., cup, towel, tool, brush, phone).
  15) Patient is being physically restrained or stabilized
      - Another person is clearly holding or stabilizing part of the patient's body
        in a way that limits or controls movement.

DECISION RULES
- Each category is INDEPENDENT. Multiple categories can be YES simultaneously
  for the same video. For example, a patient may be grabbing medical equipment
  (cat 1) while also performing other arm movements (cat 5) at the same time.
  Evaluate each category on its own merits.
- Base ALL judgments only on what is clearly visible in the frames. - Do NOT infer pain, emotions, intentions, diagnoses, or clinical reasoning. - If visibility is poor or you are not confident, use "UNABLE TO IDENTIFY" for that label. - Every label MUST be exactly one of:
    - "YES"
    - "NO"
    - "UNABLE TO IDENTIFY"
- Do NOT add extra fields or narrative text.
OUTPUT RULES
- Return a single JSON object with fixed keys for the video ID and all labels. - Do NOT include explanations, timestamps, or free-text descriptions. - Use the exact key names and allowed values shown below.
EXAMPLE OUTPUT FORMAT

  {
    "video_id": "<date>/<time>_<room>_lt.mp4",
    "labels": {
      "patient_grabbing_medical_equipment": "NO",
      "patient_using_non_medical_objects": "NO",
      "patient_gesticulating": "NO",
      "patient_adjusting_bedding": "NO",
      "patient_other_arm_movements": "NO",
      "patient_position_change": "NO",
      "patient_other_torso_movement": "NO",
      "patient_looking_at_others": "NO",
      "patient_head_turning": "NO",
      "patient_leg_ADL_movement": "NO",
      "patient_leg_in_bed_movement": "NO",
      "clinician_patient_contact": "YES",
      "patient_communication": "NO",
      "clinician_patient_object_exchange": "NO",
      "patient_restrained": "NO"
    },
    "summary": "A clinician is present and makes physical contact with the patient, while the patient remains mostly still with no notable movements."
  }

You must ALWAYS follow this structure exactly.

Hourly report prompt.

Both full hourly-report prompts remain, including six numbered sections, mandatory timing, camera reconciliation, medication event fields and observational-only alignment. Prompt instructions against clinical inference do not guarantee compliance; the independent report audit documents actual errors.

Hourly report prompt (system turn)
You are an ICU behavioral summarization model.
You will receive up to 60 minutes of Task-1 video caption outputs (one JSON entry per minute)
from two camera views (left and right) of an ICU patient room,
plus a list of medication administration events relevant to this hour.

Your job is to generate a structured behavioral report for the
observation period based purely on visible evidence from both camera views.
You must never infer sedation level, intent, pain, emotion, or clinical state.

When both camera views are available for a given minute, reconcile them:
if either view detects an action, treat it as observed. Note discrepancies
only when they are informative (e.g., one view blocked by equipment).

When only one camera view is available, note which view and proceed.

TIMING REQUIREMENT (IMPORTANT):
  - All behavioral findings must cite timing.
  - Use minute timestamps (e.g., min 14, min 27) AND/OR continuous ranges (e.g., min 10-15).
  - No event may be mentioned without a timing reference.

ALLOWED TIMING FORMATS:
  - "arm movement observed at min 12"
  - "clinician contact min 21-24"
  - "no movement detected min 0-18"

Your output must follow the 6-section template provided in the user prompt EXACTLY.
Hourly report prompt (user turn, before the hour’s records)
Generate the behavioral report for this observation hour.

REQUIRED FORMAT:

Observation window: [START_TIME] to [END_TIME] (up to 60 minutes)

1. Patient Movement Summary
    - Overall movement level (high / moderate / low / none)
    - Most common movement types
    - MUST cite minute timestamps or ranges for all movement observed
    - Note any discrepancies between left and right camera views

2. Patient-Clinician Interaction Summary
    - Whether interaction occurred
    - MUST cite timings for clinician contact, communication, restraint, exchanges

3. Significant Moments of Interest
    - Sudden spikes in movement, first movement after stillness, notable behavior
    - MUST include timing (single minute or range)
    - If prolonged stillness >= 10 min, cite the stillness window

4. Medication Context
    - List medications active or administered during this hour with timestamps
    - Include drug name, dose, route, and action (Given/Rate Change/New Bag/etc.)
    - Note any continuous infusions that are running

5. Activity-Medication Alignment (observational only)
    - Timing-based associations ONLY (e.g., "movement at min 22-28 while dexmedetomidine infusing")
    - No clinical cause/effect interpretation ever

6. Brief Summary (<= 5 sentences)
    - Compact overall description of the hour
    - Note uncertainty if many labels = "UNABLE TO IDENTIFY"
    - Note if camera coverage was incomplete

Output MUST contain all 6 sections and MUST cite times when describing events.

A.4 TRAJECTORY-144 Question Creation and Filtering

Humans create timelines/questions, strengthen distractors and review correctness. Blind-LLM accuracy flags questions for revision, but 25 final questions remain answerable at least 80% of the time without patient data. The benchmark therefore retains linguistic priors and earlier draft-question influence despite fixed final prompts.

Evidence types.

The 144 questions split into 33 video moments under five minutes, 16 windows of 5–30 minutes, 61 longer video spans, 26 record-only and eight cross-modal items. Fine-grained cross-modal percentages therefore rest on a small question count. Patient-level generalization is not tested.

Temporal anchoring.

Sixty questions name a clock time, 34 a coarser hour/day reference and fifty none. The agent can anchor through supplied times, EHR events, action queries or reports. Table 6 uses an independent run scoring 61.1%, distinct from the 61.0% main run; its full count/accuracy breakdown remains.

Agent prompt development.

Prompt refinement uses earlier draft benchmark questions. The exact final evaluated forms are excluded from development, and one fixed prompt serves all reported agents. This limits wording-specific tuning but does not supply a separate unseen-stay development/test design.

Complete original Table 6: TRAJECTORY-144 question breakdown by time reference. Accuracy (%) for Gemini 3.1 Flash Lite using TRACER, from an independent run of the Table 1 configuration (achieves 61.1% overall).
Table 6: TRAJECTORY-144 question breakdown by time reference. Accuracy (%) for Gemini 3.1 Flash Lite using TRACER, from an independent run of the Table 1 configuration (achieves 61.1% overall).
Accessible table, original values and citation links
Time reference in the question Questions Accuracy
Clock time, span under 5 minutes 33 30.9
Clock time, span 5–30 minutes 15 57.3
Clock time, span over 30 minutes 12 50.0
Hour 13 50.8
Part of day (e.g., morning, afternoon) 16 66.2
Day (e.g., November 23) 5 56.0
No explicit time, anchored on clinical record 22 96.4
No explicit time, anchored on video data 28 78.6

A.5 Patient Cohort

Figure 3 retains the full 2617→280 selection flow and cohort characteristics, including median age 66, 65.4% male, 56.1% invasive ventilation and 18.9% ever-positive delirium screening. Its embedded characteristics table is preserved along with all sixteen separately numbered tables.

Complete original Figure 3: Patient cohort selection and characteristics. Left , CONSORT-style flow diagram of patient cohort selection. Sequential exclusion criteria were applied to identify patients with sufficient ICU stay duration, sedative exposure, non-neurological admission, and adequate bilateral video coverage. Right , Characteristics of the final study cohort ( N = 280 N=280 ).
Figure 3: Patient cohort selection and characteristics. Left, CONSORT-style flow diagram of patient cohort selection. Sequential exclusion criteria were applied to identify patients with sufficient ICU stay duration, sedative exposure, non-neurological admission, and adequate bilateral video coverage. Right, Characteristics of the final study cohort (N=280N=280).

A.5.1 Patient Cohort Selection

Exclusions sequentially enforce >24-hour/<2-week stay, sedative/analgesic exposure, non-neurological admission, dual cameras, no intra-ICU transfer, full continuous feed and bed-centered views. Most original admissions do not enter the final cohort. The resulting technical completeness supports analysis while restricting coverage of routine ICU conditions.

A.5.2 Electronic Health Records

Demographics/encounter data establish patient context, ADT defines stay windows, medication events feed tools/reports, and nursing flowsheets provide vitals, RASS and CAM-ICU. Questions can access full-admission EHR beyond the video window. The text’s reference to cohort characteristics in Table 15 is retained, although overall characteristics also appear in Figure 3.

A.6 Token Cost of Raw Video

Raw paired-camera video consumes millions of tokens per hour under the paper’s settings. Table 7 retains all resolutions, frame rates and estimates. Gemini’s quoted ~1.05M tokens for 950 stills lies at the stated 1M window limit; these are measured/configuration-specific budgets, not timeless model specifications. Sparse sampling misses short events even if long-duration coverage appears complete.

Complete original Table 7: Token cost of raw video surpasses context lengths. Context windows: 1M tokens for the Gemini models, 256K for Qwen3-VL 8B. Gemini uses the original 1920 × \times 1080 clip at 5 fps. Qwen3-VL uses 512 × \times 288 resolution at 2 fps.
Table 7: Token cost of raw video surpasses context lengths. Context windows: 1M tokens for the Gemini models, 256K for Qwen3-VL 8B. Gemini uses the original 1920×\times1080 clip at 5 fps. Qwen3-VL uses 512×\times288 resolution at 2 fps.
Accessible table, original values and citation links
Input Gemini 3 Family (tokens) Qwen3-VL 8B (tokens)
One minute, one camera view 19,823 8,640
One minute, both views 39,646 17,280
One hour, both views 2.38M 1.04M
One day, both views 57.1M 24.9M
Median stay (3.3 days) 188M 82M
Longest stay in cohort (14 days) 799M 349M
TRAJECTORY-144 span (264 h) 628M 274M
One 512×\times288 frame 1,100 144
950 stills over a stay (long-context baseline) 1.05M 0.14M

A.7 Baseline Implementations

Each configuration is scored on 718 placement trials. The main paired bootstrap averages placements within question and resamples 144 questions 10,000 times. Appendix McNemar tests treat placements as independent and are explicitly approximate. Neither inferential procedure creates additional independent patients.

Text-only.

The no-tool agent tests language priors with the same forced-choice scoring loop. Its above-chance results measure guessability, not recovered patient evidence.

Socratic Models.

The Socratic pipeline performs a separate free-text captioning pass with published HourVideo prompts. Captions exceeding context are uniformly subsampled; Gemini keeps every second/third minute for the two stays, while open-weight contexts keep much less. This differs from TRACER’s structured labels and active evidence access.

Socratic baselines with clinical records and both cameras.

Adding all medication/flowsheet data lifts caption-baseline accuracy to 45.0%. Adding both camera captions under the same context budget increases stride and gives 42.7%. This comparison controls additional modalities while exposing a coverage/detail tradeoff; it does not imply a second camera is intrinsically harmful.

Long-context VLM.

The raw-frame baseline selects chronological midpoint frames without timestamps, usually from the left camera. At 950 frames it samples every 6.0 or 10.7 minutes across the two stays. Many brief events can fall between samples. This is a particular sparse long-context protocol, not exhaustive video inspection.

Long-context baseline with the clinical record.

The strongest baseline prepends the full EHR block and halves video sampling to 475 frames, fitting roughly 654k/943k tokens for the two stays. It reaches 49.3% overall and 83.8% on record-only questions. Table 8 retains all budgets, coverage and category results.

Complete original Table 8: Baselines with clinical records using Gemini 3.1 Flash Lite. Accuracy (%) averaged over gold placements. “Coverage” is the number of captioned minutes or frames that survive subsampling given the fixed 1M context for stay A / stay B; “Tokens” is the prompt length per trial. TRACER is shown on the last row for comparison.
Table 8: Baselines with clinical records using Gemini 3.1 Flash Lite. Accuracy (%) averaged over gold placements. “Coverage” is the number of captioned minutes or frames that survive subsampling given the fixed 1M context for stay A / stay B; “Tokens” is the prompt length per trial. TRACER is shown on the last row for comparison.
Accessible table, original values and citation links
Inputs Coverage Tokens <<5 min 5–30 min >>30 min EHR Cross All
Left camera captions 2.8k / 3.4k 1.0M 34.5 32.5 39.2 16.2 47.5 33.7
Left camera captions + EHR 2.8k / 1.7k 1.0M / 0.9M 30.3 32.5 43.3 76.2 42.5 45.0
Both cameras’ captions + EHR 1.4k / 0.9k 0.7M / 0.7M 32.7 31.2 35.9 76.2 50.0 42.7
950 frames 950 / 950 1.1M 35.8 32.5 38.0 14.6 25.0 31.9
475 frames 475 / 475 0.5M 26.7 35.0 36.6 17.7 22.5 29.9
475 frames + EHR 475 / 475 0.7M / 1.0M 34.5 30.0 48.9 83.8 40.0 49.3
TRACER harness, all tools (Table 1 run) – – 36.4 52.5 63.7 96.2 45.0 61.0

Open-weight question answering for Socratic baselines.

Using fixed Flash Lite captions, open-weight Socratic answerers score 23.9–29.3%, versus 57.4–67.9% with their harness configurations. Context limits reduce caption coverage to roughly 183–391 minutes per stay. The result combines representation/tool benefits with much broader accessible evidence.

Complete original Table 9: Socratic baselines with Gemini 3.1 Flash Lite captions, by open-weight answering model. Accuracy (%) averaged over gold placements; “minutes” is the number of captioned minutes that survive subsampling for stay A / stay B; the last column is the same model as the TRACER agent with Gemini 3.1 Flash Lite serving the video tool. Served context length shown in parentheses.
Table 9: Socratic baselines with Gemini 3.1 Flash Lite captions, by open-weight answering model. Accuracy (%) averaged over gold placements; “minutes” is the number of captioned minutes that survive subsampling for stay A / stay B; the last column is the same model as the TRACER agent with Gemini 3.1 Flash Lite serving the video tool. Served context length shown in parentheses.
Accessible table, original values and citation links
Answering model Minutes <<5 min 5–30 min >>30 min EHR Cross All TRACER
Gemini 3.1 FL (1M) 2.8k / 3.4k 34.5 32.5 39.2 16.2 47.5 33.7 61.0
gpt-oss-20b (131K) 379 / 391 33.3 26.3 22.7 14.6 20.0 23.9 57.4
Gemma-4 31B (131K) 379 / 377 33.9 25.0 31.4 23.1 22.5 29.3 67.9
Qwen3.5 9B (65K) 183 / 185 30.3 28.8 30.8 18.5 10.0 27.1 61.6

A.8 Model Instantiation and Computational Cost

All model roles remain frozen. Flash Lite supplies cohort perception and hourly reports, although hourly reports are generated only for the two QA stays. Authorized proprietary processing is restricted to the Gemini family by the study’s data-use agreement; this is an implementation constraint, not an endorsement of arbitrary data upload.

Computational cost.

Background construction costs two perception calls per patient-minute plus one report call per hour. Per-question Gemini 3.5 Flash use is reported as ~160k agent tokens plus ~132k video-tool tokens, about 292k total, with a heavy tail. Reusable preprocessing is excluded from that per-question comparison. The appendix’s 9.8-turn average and Table 11’s 9.0 mean calls describe the reported accounting without a full reconciliation.

Sampling and serving settings.

Gemini uses API-default generation settings. Open-weight perception uses temperature .6, top-p .95 and top-k 20; baselines use greedy decoding, with thinking disabled in vLLM. Video sampling differs by model family. Exact model/version and sampling choices matter for reproduction and comparisons.

A.9 Perception Benchmark Details

Perception analysis covers submission-time cost, label prevalence and annotation reliability. It does not validate every predicted cohort minute against human ground truth.

Accuracy-cost tradeoff.

Figure 4 preserves the September-2026 API list-price cost axis and separate self-hosted MI250X device-hour axis. Those units should not be combined as a universal monetary ranking. Flash Lite is chosen for this study’s accuracy/cost tradeoff, rather than highest detection F1.

Complete original Figure 4: Perception accuracy versus cost. Left: Gemini models, list-price cost per 1,000 camera-hours of one-minute clips (log scale); the star marks the model used for the cohort run. Right: open-weight models, device-hours per 1,000 camera-hours on one MI250X graphics compute die.
Figure 4: Perception accuracy versus cost. Left: Gemini models, list-price cost per 1,000 camera-hours of one-minute clips (log scale); the star marks the model used for the cohort run. Right: open-weight models, device-hours per 1,000 camera-hours on one MI250X graphics compute die.

Prevalence.

Common benchmark actions occur in about 70% of selected clips, while rare categories are under 5%. Actual cohort prevalence is lower because benchmark sampling intentionally spans activity levels. Aggregate performance on that enriched set may not transfer directly to precision at routine low prevalence.

Annotation reliability.

A blinded author reannotates twenty randomly selected clips with the finalized rubric. Across 300 binary decisions, agreement is 92.7% and κ=.81, with clip-bootstrap 95% interval [.74,.87]. This supports label consistency on a small audit and does not replace full-cohort validation.

A.10 Hourly Report Faithfulness

Ten reports contain 153 verifiable claims: 129 supported and 24 errors. Twelve errors partially mismatch timing; three each misstate movement level, action absence, medication absence and repeated medication absence. Named medications match recorded details, but false absence propagates between sections. The audit checks reports against model minute records/EHR rather than independently verifying every raw video action, and cannot detect omissions. Its 84.3% support rate is not an end-to-end clinical correctness rate.

Complete original Table 10: Claim-level audit of ten hourly reports. Counts are verifiable claims. Partial timing match: under 80% of cited minutes show the action. Carry over: a false medication absence repeated in the alignment section.
Table 10: Claim-level audit of ten hourly reports. Counts are verifiable claims. Partial timing match: under 80% of cited minutes show the action. Carry over: a false medication absence repeated in the alignment section.
Accessible table, original values and citation links
Report section
Movement Interaction Moments Medication Alignment Summary All %
Verifiable claims 41 36 28 16 12 20 153
Supported 31 32 25 13 9 19 129 84.3
Errors 10 4 3 3 3 1 24 15.7
Only partial timing match 7 1 3 0 0 1 12 7.8
Movement summary misleading 3 0 0 0 0 0 3 2.0
Incorrect action absence 0 3 0 0 0 0 3 2.0
Incorrect medication absence 0 0 0 3 0 0 3 2.0
Carry over of incorrect information 0 0 0 0 3 0 3 2.0
Support rate (%) 76 89 89 81 75 95 84

A.11 How does TRACER Perform Across Different Agent Models?

The twelve configurations separate agent and video-tool changes. Gemini 3.5 Flash falls 76.9→70.7 with weaker fixed video perception and reaches the call cap on 10.7% of trials in that condition. Open-weight agents can be competitive when proprietary perception remains. Approximate placement-level McNemar significance is retained with its dependence caveat.

Complete original Table 11: TRACER applied across twelve agent configurations. Full TRACER with the agent model as the only variable; the background records and reports are fixed. Tool calls are means per question. † Video tool served by Gemini 3.1 Flash Lite rather than the agent’s own model.
Table 11: TRACER applied across twelve agent configurations. Full TRACER with the agent model as the only variable; the background records and reports are fixed. Tool calls are means per question. †Video tool served by Gemini 3.1 Flash Lite rather than the agent’s own model.
Accessible table, original values and citation links
Agent model All <<5 min 5–30 min >>30 min EHR Cross Avg. Calls
Gemini 2.5 Flash Lite 49.2 32.7 35.0 53.9 69.2 45.0 3.7
Gemini 2.5 Flash 60.3 33.9 56.2 64.0 90.8 50.0 4.7
Gemini 3 Flash 70.8 44.8 66.3 77.0 91.5 72.5 12.3
Gemini 3.1 Flash Lite 61.0 36.4 52.5 63.7 96.2 45.0 8.9
Gemini 3.5 Flash Lite 61.1 40.0 52.5 59.8 99.2 52.5 6.8
Gemini 3.5 Flash 76.9 54.5 75.0 80.7 99.2 72.5 9.0
Qwen3.5 9B 57.9 26.7 43.8 63.3 95.4 52.5 10.4
Qwen3-VL 8B 42.1 19.4 32.5 45.7 68.5 42.5 4.7
gpt-oss-20b† 57.4 32.1 41.2 62.5 93.1 40.0 3.7
Qwen3.5 9B† 61.6 35.2 45.0 67.3 95.4 50.0 9.0
Gemma-4 31B† 67.9 40.0 61.2 73.7 100.0 47.5 5.4
Gemini 3.5 Flash† 70.7 35.8 57.5 80.3 98.5 77.5 9.5

A.12 Per-Model Cumulative Ablation

All five agents’ cumulative tables retain question-type and stay-A/B results. Performance differs markedly between stays: main Flash Lite is 75.8% on A and 50.8% on B. Reports do not improve every category/agent significantly; the Gemini 3.5 report increment has approximate p=.06. Adding video also reduces some Qwen category/overall scores. Two-stay variation limits broad claims from pooled averages.

Complete original Table 12: Per-model cumulative ablation, by question type and by stay. Cumulative ablation: each row adds one tool family to the row above (Table 5 ). Paired McNemar tests on identical trials for Gemini 3.5 Flash show that every step is significant ( p < 0.001 p<0.001 ) except adding reports ( + 3.5 +3.5 , χ 2 = 3.6 \chi^{2}=3.6 , p = 0.06 p=0.06 ).
Table 12: Per-model cumulative ablation, by question type and by stay. Cumulative ablation: each row adds one tool family to the row above (Table 5). Paired McNemar tests on identical trials for Gemini 3.5 Flash show that every step is significant (p<0.001p<0.001) except adding reports (+3.5+3.5, χ2=3.6\chi^{2}=3.6, p=0.06p=0.06).
Accessible table, original values and citation links
Video EHR Cross-modal Stay All
Agent tools <<5 min 5–30 min >>30 min Record only Video + EHR A B
Gemini 3.5 Flash
Text-only 37.0 32.5 51.4 21.5 27.5 35.9 41.6 39.3
+ timeline 37.6 57.5 67.5 42.3 20.0 62.0 45.6 52.4
+ reports 40.6 72.5 70.5 38.5 30.0 70.2 45.9 55.8
+ EHR 39.4 76.2 71.1 100.0 75.0 84.7 59.5 69.9
+ video (TRACER) 54.5 75.0 80.7 99.2 72.5 88.5 68.9 76.9
Gemini 3.1 Flash Lite
Text-only 24.2 18.8 40.2 26.9 25.0 26.8 33.8 30.9
+ timeline 23.0 32.5 55.3 26.2 22.5 50.5 29.8 38.3
+ reports 24.8 48.8 62.9 31.5 37.5 57.9 36.9 45.5
+ EHR 23.6 36.2 58.6 96.2 47.5 71.2 42.5 54.3
+ video (TRACER) 36.4 52.5 63.7 96.2 45.0 75.8 50.8 61.0
Qwen3.5 9B
Text-only 31.5 27.5 43.9 20.0 40.0 35.3 34.4 34.7
+ timeline 21.8 40.0 59.3 22.3 32.5 52.1 32.2 40.4
+ reports 26.1 45.0 58.0 23.8 27.5 51.9 34.0 41.4
+ EHR 32.1 42.5 62.5 95.4 62.5 74.8 48.4 59.2
+ video (TRACER) 26.7 43.8 63.3 95.4 52.5 74.6 46.4 57.9
Qwen3-VL 8B
Text-only 29.7 28.7 42.6 25.4 32.5 34.9 34.1 34.4
+ timeline 21.2 27.5 39.9 22.3 35.0 39.3 24.9 30.8
+ reports 19.4 48.8 46.7 18.5 22.5 39.5 30.6 34.2
+ EHR 17.0 33.8 43.5 70.0 25.0 53.8 30.6 40.1
+ video (TRACER) 19.4 32.5 45.7 68.5 42.5 55.7 32.7 42.1
gpt-oss-20b (video tool served by Gemini 3.1 Flash Lite)
Text-only 22.4 17.5 36.5 22.3 42.5 27.5 29.9 28.9
+ timeline 22.4 36.2 56.5 16.2 35.0 47.4 31.4 38.0
+ reports 25.5 38.8 62.1 27.7 30.0 55.8 34.3 43.1
+ EHR 26.7 40.0 59.3 90.8 52.5 74.8 41.2 55.0
+ video (TRACER) 32.1 41.2 62.5 93.1 40.0 73.4 46.4 57.4

A.13 Single Tool Family Ablations

Single-family ablations retain all reported model-specific category results. EHR alone excels on record questions; video alone helps short moments but is weaker on long spans. These family effects depend on the shared scaffold and prompts, with incomplete independent factorial control of all interactions.

Complete original Table 13: Single tool families on TRAJECTORY-144. Each row gives the agent one tool family only. Accuracy (%) averaged over gold placements; category columns are question means.
Table 13: Single tool families on TRAJECTORY-144. Each row gives the agent one tool family only. Accuracy (%) averaged over gold placements; category columns are question means.
Accessible table, original values and citation links
Agent tools All <<5 min 5–30 min >>30 min EHR Cross
Gemini 3.5 Flash
Reports only 51.0 44.2 72.5 55.4 36.9 47.5
EHR only 59.6 49.1 47.5 49.5 100.0 72.5
Video only 46.4 54.5 76.2 43.9 30.0 27.5
All tools 76.9 54.5 75.0 80.7 99.2 72.5
Gemini 3.1 Flash Lite
Reports only 37.5 27.9 38.8 44.5 31.5 40.0
EHR only 45.3 26.1 20.0 40.7 95.4 47.5
Video only 34.5 38.2 45.0 36.8 22.3 20.0
All tools 61.0 36.4 52.5 63.7 96.2 45.0
Qwen3.5 9B
Reports only 37.7 29.1 46.2 42.2 31.5 42.5
EHR only 43.0 23.6 27.5 33.5 93.8 60.0
Video only 30.2 26.7 31.2 34.3 23.1 35.0
All tools 57.9 26.7 43.8 63.3 95.4 52.5
gpt-oss-20b (video tool served by Gemini 3.1 Flash Lite)
Reports only 32.8 26.7 43.8 33.9 30.0 37.5
EHR only 37.3 26.1 27.5 23.4 93.1 27.5
Video only 28.4 28.5 43.8 32.3 14.6 12.5
All tools 57.4 32.1 41.2 62.5 93.1 40.0

A.14 Delirium and Atomic Action Frequency Analysis

Equation 1 pools detected action minutes within each ever-positive/never-positive group, weighting longer-observed patients more. Equal-patient weighting preserves the largest purposeful-action reductions but shifts several near-one ratios. Patient bootstrap intervals and Holm/BH-adjusted per-patient tests provide complementary uncertainty measures: head turning’s bootstrap interval excludes no difference, but its Mann–Whitney p=.094 is not significant. Six reduced actions survive Holm at .05; the three strongest survive below 1e-4, while gesticulation’s adjusted p=.0015 remains below .05. Delirium patients have longer stays and more deeply sedated assessments; effects are not multivariable-adjusted or aligned to positive-screen periods. Complete Tables 14–16 and all formulas remain.

Equation 1
FRa=fadelfanon​-​del,fa(g)=∑p∈𝒫gma(p)∑p∈𝒫gM(p),\mathrm{FR}_{a}=\frac{f_{a}^{\mathrm{del}}}{f_{a}^{\mathrm{non\text{-}del}}},\qquad f_{a}^{(g)}=\frac{\sum_{p\in\mathcal{P}_{g}}m_{a}^{(p)}}{\sum_{p\in\mathcal{P}_{g}}M^{(p)}}, (1)
Original formal definition and mathematical context

where 𝒫g\mathcal{P}_{g} denotes the set of patients in group g∈{del,non​-​del}g\in\{\mathrm{del},\mathrm{non\text{-}del}\}, ma(p)m_{a}^{(p)} is the number of patient-minutes in which action aa is detected for patient pp, and M(p)M^{(p)} is the total number of observed minutes for that patient; group frequencies are thus pooled over all observed minutes within each group (minute-weighted).

Complete original Table 14: Significance tests for delirium frequency differences. Per-patient rates (median) by group with two-sided Mann–Whitney U U p p and Holm and Benjamini–Hochberg adjustment over the 13 evaluated actions. Permutation p p for the minute-weighted and equal-weighted log 2 \log_{2} frequency ratio over 2,000,000 label permutations (floor 5 × 10 − 7 5\times 10^{-7} ).
Table 14: Significance tests for delirium frequency differences. Per-patient rates (median) by group with two-sided Mann–Whitney UU pp and Holm and Benjamini–Hochberg adjustment over the 13 evaluated actions. Permutation pp for the minute-weighted and equal-weighted log2\log_{2} frequency ratio over 2,000,000 label permutations (floor 5×10−75\times 10^{-7}).
Accessible table, original values and citation links
Action Med. del. Med. non-del. MWU pp Holm BH Perm. pp (mw / ew)
Using non-medical objects 0.016 0.080 1.8×10−101.8\times 10^{-10} 2.3×10−92.3\times 10^{-9} 2.3×10−92.3\times 10^{-9} 5×10−75\times 10^{-7} / 5×10−75\times 10^{-7}
Object exchange 0.004 0.010 8.9×10−108.9\times 10^{-10} 1.1×10−81.1\times 10^{-8} 5.8×10−95.8\times 10^{-9} 5×10−75\times 10^{-7} / 5×10−75\times 10^{-7}
Leg ADL movement 0.002 0.007 7.3×10−77.3\times 10^{-7} 8.0×10−68.0\times 10^{-6} 3.2×10−63.2\times 10^{-6} 4×10−64\times 10^{-6} / 5×10−75\times 10^{-7}
Clinician–patient contact 0.423 0.349 1.4×10−51.4\times 10^{-5} 1.4×10−41.4\times 10^{-4} 4.7×10−54.7\times 10^{-5} 1.1×10−31.1\times 10^{-3} / 1.0×10−41.0\times 10^{-4}
Other arm movements 0.165 0.243 2.9×10−52.9\times 10^{-5} 2.6×10−42.6\times 10^{-4} 7.4×10−57.4\times 10^{-5} 6.5×10−66.5\times 10^{-6} / 2×10−62\times 10^{-6}
Gesticulating 0.030 0.053 1.9×10−41.9\times 10^{-4} 1.5×10−31.5\times 10^{-3} 4.1×10−44.1\times 10^{-4} 1.4×10−41.4\times 10^{-4} / 3.5×10−43.5\times 10^{-4}
Grabbing medical equipment 0.081 0.102 4.8×10−34.8\times 10^{-3} 0.034 8.9×10−38.9\times 10^{-3} 1.1×10−31.1\times 10^{-3} / 4.1×10−34.1\times 10^{-3}
Head turning 0.363 0.399 0.094 0.56 0.15 4.3×10−34.3\times 10^{-3} / 8.6×10−38.6\times 10^{-3}
Position change 0.027 0.029 0.11 0.56 0.16 0.21 / 0.59
In-bed leg movement 0.022 0.019 0.23 0.93 0.29 0.95 / 0.028
Adjusting bedding 0.037 0.038 0.25 0.93 0.29 0.23 / 0.52
Communication 0.240 0.257 0.27 0.93 0.29 0.080 / 0.20
Looking at others 0.330 0.320 0.74 0.93 0.74 0.54 / 1.0
Purposeful activity (any of four) 0.053 0.146 1.3×10−81.3\times 10^{-8} – – 5×10−75\times 10^{-7}
Complete original Table 15: Cohort characteristics by delirium status. Delirium is defined as any positive CAM-ICU assessment during the ICU stay. RASS percentages are computed over all recorded assessments in each group rather than over patients. Observed minutes run from arrival in the monitored room and can exceed ICU LOS, which counts only time at the critical-care level of care.
Table 15: Cohort characteristics by delirium status. Delirium is defined as any positive CAM-ICU assessment during the ICU stay. RASS percentages are computed over all recorded assessments in each group rather than over patients. Observed minutes run from arrival in the monitored room and can exceed ICU LOS, which counts only time at the critical-care level of care.
Accessible table, original values and citation links
Characteristic Delirium (n=53n=53) Non-delirium (n=227n=227)
ICU LOS, days, median [IQR] 4.62 [3.01, 8.40] 3.09 [1.89, 4.84]
Observed minutes, median [IQR] 7,560 [4,680, 12,840] 4,920 [2,580, 7,440]
Invasive mechanical ventilation, nn (%) 31 (58.5) 126 (55.5)
Per-patient median RASS, median [IQR] 0.00 [−1.00-1.00, 0.00] 0.00 [−0.50-0.50, 0.00]
RASS assessments ≤−3\leq-3, % 5.0 (of 6,979) 2.0 (of 18,402)
Complete original Table 16: Delirium frequency ratios under weighting schemes with bootstrap intervals. Minute-weighted ratios follow Eq. 1 and reproduce Figure 2 (b); equal-weighted ratios average per-patient rates within each group. Intervals are 95% percentile bootstrap intervals over 10,000 patient-level resamples. Values below 1.00 1.00 ( log 2 \log_{2} below zero) indicate actions detected less frequently among patients with documented delirium. The final two actions are excluded from the 13-action evaluation set and from Figure 2 (b) for lack of sufficient positive instances in the annotated benchmark.
Table 16: Delirium frequency ratios under weighting schemes with bootstrap intervals. Minute-weighted ratios follow Eq. 1 and reproduce Figure 2(b); equal-weighted ratios average per-patient rates within each group. Intervals are 95% percentile bootstrap intervals over 10,000 patient-level resamples. Values below 1.001.00 (log2\log_{2} below zero) indicate actions detected less frequently among patients with documented delirium. The final two actions are excluded from the 13-action evaluation set and from Figure 2(b) for lack of sufficient positive instances in the annotated benchmark.
Accessible table, original values and citation links
Minute-weighted Equal patient weight
Action FR log2\log_{2} FR [95% CI] FR log2\log_{2} FR [95% CI]
Using non-medical objects 0.41 −1.28-1.28 [−1.92,−0.75][-1.92,-0.75] 0.39 −1.35-1.35 [−1.94,−0.87][-1.94,-0.87]
Leg ADL movement 0.46 −1.13-1.13 [−1.69,−0.67][-1.69,-0.67] 0.47 −1.07-1.07 [−1.59,−0.64][-1.59,-0.64]
Object exchange 0.48 −1.05-1.05 [−1.57,−0.61][-1.57,-0.61] 0.47 −1.08-1.08 [−1.55,−0.69][-1.55,-0.69]
Gesticulating 0.60 −0.74-0.74 [−1.17,−0.35][-1.17,-0.35] 0.68 −0.56-0.56 [−0.97,−0.21][-0.97,-0.21]
Other arm movements 0.62 −0.70-0.70 [−1.00,−0.42][-1.00,-0.42] 0.69 −0.53-0.53 [−0.79,−0.29][-0.79,-0.29]
Grabbing medical equipment 0.67 −0.58-0.58 [−0.87,−0.30][-0.87,-0.30] 0.80 −0.33-0.33 [−0.58,−0.09][-0.58,-0.09]
Head turning 0.85 −0.24-0.24 [−0.43,−0.07][-0.43,-0.07] 0.90 −0.15-0.15 [−0.30,−0.01][-0.30,-0.01]
Adjusting bedding 0.85 −0.23-0.23 [−0.58,0.14][-0.58,0.14] 1.07 0.100.10 [−0.31,0.49][-0.31,0.49]
Position change 0.85 −0.24-0.24 [−0.61,0.14][-0.61,0.14] 1.06 0.090.09 [−0.37,0.52][-0.37,0.52]
Looking at others 0.96 −0.06-0.06 [−0.25,0.12][-0.25,0.12] 1.00 −0.00-0.00 [−0.17,0.15][-0.17,0.15]
In-bed leg movement 1.01 0.010.01 [−0.38,0.42][-0.38,0.42] 1.30 0.380.38 [−0.06,0.82][-0.06,0.82]
Communication 0.88 −0.18-0.18 [−0.42,0.04][-0.42,0.04] 0.94 −0.10-0.10 [−0.30,0.09][-0.30,0.09]
Clinician–patient contact 1.20 0.260.26 [0.14,0.39][0.14,0.39] 1.18 0.240.24 [0.14,0.35][0.14,0.35]
Not in evaluation set (detection performance not validated)
Other torso movement 0.90 −0.16-0.16 [−0.46,0.15][-0.46,0.15] 1.08 0.100.10 [−0.25,0.45][-0.25,0.45]
Clinician restraining 1.54 0.620.62 [0.15,1.07][0.15,1.07] 1.33 0.410.41 [0.01,0.79][0.01,0.79]

A.15 Ethics, Data, and Code Availability

The appendix states the IRB, access and release boundaries. Publicly shareable code/prompts are distinct from the protected study data.

Human subjects review.

Stanford IRB approval covers the retrospective study, with restricted study-personnel access and secure storage/processing. The paper does not publish raw identifiable bedside recordings.

Data and code availability.

Video and linked records remain unavailable publicly under privacy and IRB restrictions. The stated repository releases taxonomy, prompts and inference/evaluation code. The preserved de-identified question examples are the public manuscript examples, without inventing patient dates or times.

Concept demonstration

Open image ↗