Reading guide
This edition covers all 37 pages, including the full appendix. It preserves every figure/table, all ten de-identified example questions and choices/checkmarks, eight complete prompts/tool instructions and the original mathematical definitions. The source’s one-line-high SVG prompt boxes are replaced by their exact full downloadable text, retaining whitespace. The first three authors contributed equally; June (Yu Jun) Zheng’s byline is retained with the imported spelling searchable.
At a glance
TRACER is a training-free harness for retrospective reasoning over ICU video and electronic records. Frozen models build minute-level action/caption records, combine them with medication logs into hourly reports, and let an agent query those summaries, the record and targeted video. Perception reaches 84.8% accuracy/68.6 micro-F1 on 397 selected clips. With the same Gemini 3.1 Flash Lite backbone, QA rises from the strongest baseline’s 49.3% to 61.0%; the best agent reaches 76.9%. Those QA results concern 144 questions from only two stays. The separate 280-patient behavioral study reports associations with ever-positive delirium screens, not a validated diagnosis/prediction system.
1 Introduction
A whole ICU stay exceeds practical dense-video context, while clinically relevant events range from brief movements to multi-day patterns and require record context. TRACER separates reusable background abstraction from per-question evidence gathering. Its taxonomy, tools and prompts can be shared without releasing patient recordings or fitting model parameters to them. The 397-clip, two-stay QA and 280-patient cohort analyses address different questions and should keep their denominators separate.
2 Harness Design
The harness consists of accessible representations, evidence tools and a bounded reasoning loop. It continuously constructs a hierarchy, then retrieves selected evidence per question rather than placing every raw observation in one prompt. The complete visual overview remains.
Accessible table, original values and citation links
| Video | EHR | Cross-modal | Avg. | |
| No video | ||||
| Random | 20.1 | 20.0 | 20.0 | 20.1 |
| Text-only LLM, Gemini 3.1 Flash Lite | 32.3 | 26.9 | 25.0 | 30.9 |
| Long-context VLMs | ||||
| Qwen3-VL 8B, 475 frames | 31.6 | 10.0 | 47.5 | 28.6 |
| Qwen3-VL 8B, 950 frames | 36.3 | 13.8 | 50.0 | 33.0 |
| Gemini 3.1 Flash Lite, 950 frames | 36.5 | 14.6 | 25.0 | 31.9 |
| Gemini 3.1 Flash Lite, 475 frames + EHR | 41.8 | 83.8 | 40.0 | 49.3 |
| Socratic models (HourVideo) | ||||
| Qwen3-VL 8B | 33.7 | 22.3 | 20.0 | 30.9 |
| Qwen3.5 9B | 30.4 | 18.5 | 10.0 | 27.1 |
| Gemini 3.1 Flash Lite | 36.8 | 16.2 | 47.5 | 33.7 |
| Gemini 3.1 Flash Lite + EHR | 37.8 | 76.2 | 42.5 | 45.0 |
| TRACER harness (ours) | ||||
| w/ gpt-oss-20b† | 50.3 | 93.1 | 40.0 | 57.4 |
| w/ Qwen3.5 9B | 49.5 | 95.4 | 52.5 | 57.9 |
| w/ Gemini 3.1 Flash Lite | 53.9 | 96.2 | 45.0 | 61.0 |
| w/ Gemma-4 31B† | 61.8 | 100.0 | 47.5 | 67.9 |
| w/ Gemini 3.5 Flash | 72.0 | 99.2 | 72.5 | 76.9 |
2.1 Problem Formulation
A stay contains paired one-minute camera clips and timestamped clinical events. The task is evidence-supported answering anywhere in that timeline. Median ICU stay is 3.3 days, with stays approaching two weeks. Token budget and protected-data sharing constraints motivate a frozen-model hierarchy. These are system-design requirements; they do not establish clinical effectiveness outside the reported retrospective evaluation.
Original formal definition and mathematical context
We model an ICU stay as , where and are the one-minute clips from two wall-mounted cameras at minute , is the length of the stay in minutes, and is the set of timestamped events in the electronic health record (medication administrations, bedside assessments, vital signs, etc.). In our cohort the median stay is 3.3 days () and stays run up to two weeks (). Given a free-text clinical question , the task is to return an answer supported by evidence that may lie anywhere in , in either modality, and at a temporal scale ranging from a moment in time to the entire stay. There are two primary constraints that motivate our system design. First, a single day of video from one camera sampled at 1 FPS is approximately an order of magnitude past the context length of most existing models, which generally range from 256K to 1M tokens (Bai et al., 2025; Comanici et al., 2025) (see Appendix A.6 for more details). Second, a model whose weights are fitted to protected health information cannot be easily shared with external institutions. Thus, we design our harness so that all problem-specific structure and performance gains live in components that can be publicly released (our activity taxonomy, timeline structure and construction, aggregation tools, and model prompts).
2.2 Harness Architecture
Three frozen roles perform perception, hourly reporting and agent reasoning. The first two create reusable records; the agent runs once per question. Swapping roles is allowed, but end-to-end accuracy also depends on their error propagation and the model serving the video query tool.
2.3 Minute-Level Clinical Event Records
Each minute becomes a 15-label action vector with per-view captions. The action taxonomy supplies consistent variables for aggregation, while captions retain details outside the categories.
Taxonomy.
Clinicians, nurses and researchers iteratively defined interaction, head, torso, arm and leg categories. Labels can co-occur. They describe visible actions, not inferred pain, intention or diagnosis. The taxonomy and full perception prompt remain.
Minute-Level Video Inference.
Each camera minute is sampled at 1 fps, giving sixty frames. Predictions merge with logical OR across cameras, while both captions remain. An uncertain label becomes negative in the stored timeline, but benchmark abstentions are excluded from metrics. Appendix A.3 quantifies the difference; this merging/abstention policy matters when interpreting aggregate behavior frequencies.
2.4 Hourly Behavioral Reports
An hourly model receives sixty minute records and concurrent medication events, then produces movement, interaction, significant-moment, medication, observational alignment and narrative sections. Timestamp citations aid inspection. The ten-report audit finds 84.3% of claims supported by the input records; these reports remain fallible summaries rather than independent clinical evidence.
2.5 Tool-Augmented Trajectory Reasoning
Report, timeline, video and EHR tools expose different temporal resolutions. An agent can request new VLM analysis of a specific minute from both views, sampled at 5 fps for Gemini or 2 fps for open-weight models. It gets at most 25 tool calls. Forced guessing is an MCQ evaluation rule; the clinical interface can say evidence is insufficient.
3 Data and Evaluation
Evaluation separates local action recognition, longitudinal question answering and cohort-level behavioral associations. The hospital data are private, and the released harness does not make those patient records public.
3.1 ICU Dataset
Stanford collected paired-camera 1080p/15-fps recordings in eight cardiovascular/medical ICU rooms during August 2021–June 2025. Sequential clinical and technical filters select 280 of 2617 patients, spanning about 60,000 camera-hours. This is a single-center, sedative/analgesic-exposed, non-neurological-admission cohort with stringent continuous-camera requirements, rather than a representative sample of all ICU admissions.
3.2 Perception Benchmark
The perception benchmark contains 397 selected one-minute clips from 26 stays, covering fifteen labels. One annotator labels each clip and another verifies/corrects it. A separate blinded twenty-clip reannotation audit gives 92.7% agreement and κ=.81. Reported model variability comes from five runs on this fixed benchmark, not five independent hospitals or patient cohorts.
Accessible table, original values and citation links
| Model | Accuracy | F1 | Precision | Recall |
|---|---|---|---|---|
| Baselines | ||||
| Random (Match distr.) | 69.90.4 | 44.80.8 | 44.80.8 | 44.71.0 |
| Majority Class | 77.7 | 45.2 | 68.9 | 33.6 |
| Open-source | ||||
| Qwen3-VL (8B-Instruct) | 78.00.1 | 37.60.4 | 84.80.3 | 24.20.3 |
| Qwen3.5 (4B) | 78.20.3 | 41.00.8 | 78.31.7 | 27.70.6 |
| Qwen3.5 (9B) | 79.70.5 | 48.42.0 | 79.41.0 | 34.81.9 |
| Gemma-4 (E4B-it) | 74.90.1 | 22.50.3 | 72.21.6 | 13.40.2 |
| Proprietary | ||||
| Gemini 2.5 Flash Lite | 79.00.3 | 53.00.7 | 68.40.9 | 43.30.9 |
| Gemini 2.5 Flash | 83.20.2 | 66.50.5 | 71.60.6 | 62.00.9 |
| Gemini 2.5 Pro | 84.40.2 | 68.10.5 | 76.00.4 | 61.70.7 |
| Gemini 3.1 Flash Lite | 82.30.2 | 62.20.3 | 74.50.7 | 53.30.4 |
| Gemini 3.5 Flash | 84.80.0 | 68.60.1 | 78.30.3 | 61.00.3 |
3.3 TRAJECTORY-144: Longitudinal Reasoning Benchmark
TRAJECTORY-144 probes two long patient stays with many questions, rather than broad generalization across patients. Its construction, modality mix, temporal anchors and baseline budgets are detailed in the appendix.
Benchmark Construction.
Annotators construct event timelines, consult clinicians, write questions and revise plausible distractors. The final benchmark has 110 video, 26 EHR and eight cross-modal questions over 264 video hours. Answer placements rotate, but two questions have four choices, so there are 718 scored trials rather than 720. Earlier draft questions inform agent-prompt refinement, while final evaluated wording does not; this is not an untouched patient-level held-out prompt-development split.
Baselines.
Text-only answers test linguistic priors. Long-context models receive sparsely sampled frames, while Socratic baselines read subsampled minute captions. EHR-augmented and two-camera variants reduce modality-access gaps, but temporal coverage, prompts and retrieval differ. Baseline prompt tokens exceed per-question harness tokens; reusable background hierarchy construction is a separate substantial cost.
4 Experiments and Results
The results evaluate perception quality, tool-assisted QA, model-role changes and descriptive group differences. Complete tables keep model-specific settings and small cross-modal denominators visible.
4.1 Minute-level Perception
Best perception is Gemini 3.5 Flash at 84.8% accuracy and 68.6 F1. Gemini 3.1 Flash Lite gives 82.3%/62.2 at the reported lower submission-time cost, motivating its cohort use. Accuracy must be read with imbalanced labels: the majority-class baseline already scores 77.7%. Per-category F1 and recall reveal limits that a single accuracy number obscures.
4.2 Longitudinal Reasoning on TRAJECTORY-144
The main same-backbone comparison is 61.0% TRACER versus 49.3% frames+EHR and 45.0% captions+EHR. The broader best-agent result and fixed-perception experiments use different configurations, retained separately.
Comparison with baselines.
For Gemini 3.1 Flash Lite, the 11.7-point advantage over the strongest baseline has a question-paired bootstrap 95% interval [4.4,18.9]. Resampling questions conditions on the same two stays; it does not quantify patient-population generalization. Several questions remain guessable without patient data, and EHR-only performance is much higher than short-video performance.
Accessible table, original values and citation links
| Video | EHR | Cross-modal | Avg. | ||||
| Agent tools | 5 min | 5–30 min | 30 min | Record only | Video + EHR | ||
| Single toolset performance | |||||||
| Timeline only | 27.5 | 43.3 | 60.7 | 30.3 | 25.0 | 43.7 | |
| Reports only | 33.7 | 52.5 | 47.4 | 33.3 | 43.3 | 42.1 | |
| EHR only | 32.9 | 31.7 | 41.2 | 96.4 | 60.0 | 49.3 | |
| Video only | 39.8 | 50.8 | 38.3 | 25.1 | 27.5 | 37.0 | |
| Cumulative toolset performance | |||||||
| w/ timeline | 27.5 | 43.3 | 60.7 | 30.3 | 25.0 | 43.7 | 0.0 |
| + reports | 30.5 | 55.4 | 63.8 | 31.3 | 31.7 | 47.6 | +3.9 |
| + EHR | 31.7 | 51.6 | 64.1 | 97.2 | 61.7 | 61.1 | +17.4 |
| + video (TRACER) | 39.2 | 57.1 | 69.2 | 96.9 | 56.7 | 65.3 | +21.6 |
Harness ablation.
Across three agents, the full harness averages 65.3%, versus 49.3% for the strongest single family. Timeline/report/EHR/video additions average 43.7→47.6→61.1→65.3; the displayed +17.4 and +21.6 gains are relative to the initial timeline setting. Benefits vary by agent and question type: adding video lowers Qwen3.5 9B’s overall 59.2→57.9 in the per-model table, so monotonic gains are not universal.
Agentic performance.
Fixing background records/reports and all video queries to Gemini 3.1 Flash Lite separates agent reasoning from additional perception. Gemma-4 31B reaches 67.9%, Qwen3.5 9B 61.6% and Gemini 3.5 Flash 70.7%. With its own stronger video tool the latter reaches 76.9%. Thus the best-agent table does not vary only text reasoning: the video-tool model changes unless explicitly marked.
Accessible table, original values and citation links
| Video | EHR | Cross-modal | Avg. | Calls | |||
| Agent model | 5 min | 5–30 min | 30 min | Record only | Video + EHR | ||
| Proprietary | |||||||
| Gemini 3.5 Flash | 35.8 | 57.5 | 80.3 | 98.5 | 77.5 | 70.7 | 9.5 |
| Gemini 3.1 Flash Lite | 36.4 | 52.5 | 63.7 | 96.2 | 45.0 | 61.0 | 8.9 |
| Open-weight | |||||||
| gpt-oss-20b | 32.1 | 41.2 | 62.5 | 93.1 | 40.0 | 57.4 | 3.7 |
| Qwen3.5 9B | 35.2 | 45.0 | 67.3 | 95.4 | 50.0 | 61.6 | 9.0 |
| Gemma-4 31B | 40.0 | 61.2 | 73.7 | 100.0 | 47.5 | 67.9 | 5.4 |
4.3 Cohort-Level Behavioral Signal
Patients with any positive CAM-ICU assessment (53) are compared with 227 without a documented positive screen. Minute-weighted ratios are .41 for non-medical object use, .46 for daily-living leg movement, .48 for object exchange and .60 for gesticulation; clinician contact is 1.20. Purposeful activity’s per-patient median is .053 versus .146 (Mann–Whitney p=1.3e-8). These unadjusted retrospective predicted-behavior associations pool whole stays and are not delirium-period classification, causation or prospective detection evidence.
5 Related Work
Related work spans ambient hospital sensing, long-video memory/retrieval and clinical agents. TRACER combines paired bedside video with asynchronous structured records through a shareable prompted hierarchy.
Ambient intelligence in hospitals.
Prior inpatient activity systems train task-specific models on annotated patient data. TRACER instead composes frozen models and releases its taxonomy/prompts/tools. That avoids learning new parameters from this cohort; protected inputs still require the study’s authorized processing environment.
Long-video understanding.
Caption composition, retrieval, backtracking and week-scale egocentric memories address long video. The ICU setting additionally links bedside behavior with records. The result concerns this benchmark and implementation, rather than every possible optimized long-context/retrieval baseline.
Clinical agents.
Existing clinical agents query text/structured records or medical images. This harness adds days of bedside video and multi-resolution evidence access. It is evaluated for retrospective answering and descriptive cohort analysis, without a prospective clinical-outcome trial.
6 Discussion
The discussion emphasizes useful but noisy perception, improved evidence retrieval and potentially meaningful behavioral associations. It also identifies report errors, restricted cohort coverage and deployment prerequisites.
Key findings.
Same-backbone QA improves by 11.7 points, and fixed-perception results show that open-weight agents can reason effectively over the harness. This does not remove the perception gap or demonstrate equivalent entirely on-premise end-to-end performance, because several open-weight-agent rows retain proprietary perception.
Behavioral signal and delirium.
Reduced purposeful movement is consistent with the cited delirium phenomenology, but the paper explicitly treats its cohort results as hypothesis-generating associations. Sedation, ventilation, length of stay, label timing and model errors complicate interpretation. No predictive sensitivity/specificity or calibrated delirium diagnostic threshold is established.
User Interface.
A bedside-video viewer displays action timelines, chat, tool traces and clickable timestamp citations. Clinician feedback informs its design. The intended use is retrospective querying; the study does not report a controlled interface-usability trial or an evaluated continuous-alert workflow.
Limitations.
Limits include one hospital, only two QA trajectories, intermittent delirium screens, error propagation and hallucinated/incorrect hourly claims. Consent, staff surveillance and fairness require further work. These are the paper’s stated limits, not evidence that those issues were quantitatively resolved.
Reproducibility.
The released harness repository is the stated source for tools, taxonomy, prompts and inference/evaluation code. Patient recordings and linked EHR remain unavailable publicly. Re-instantiating the hierarchy elsewhere still requires compatible data and local evaluation.
AI use statement
The authors disclose AI assistance with writing, formatting, code, analysis and some question/distractor wording. Humans set the research/taxonomy and produced annotations, with clinician consultation and author verification. This statement remains distinct from the claim that no model role was trained on the protected cohort.
Acknowledgments
The acknowledgment credits Stanford clinical/engineering staff, Schmidt Futures, Stanford health/medicine support, HAI Google Cloud credits, NAIRR and Oak Ridge computing. All forty references remain with their citation links.
Appendix A Appendix
The complete appendix supplies examples, interfaces/prompts, hierarchy details, benchmark construction, cohort selection, computational costs, audits, ablations and statistical sensitivity analyses.
A.1 TRAJECTORY-144 Example Questions
All ten de-identified questions remain in their original colored boxes, with every choice and correct-answer checkmark. They cover two examples each for video moments, video windows, long spans, EHR and cross-modal reasoning. These are examples from the same two stays, not additional patients or an independent test set.
Original selectable question and answer choices
Original selectable question and answer choices
Original selectable question and answer choices
Original selectable question and answer choices
Original selectable question and answer choices
Original selectable question and answer choices
Original selectable question and answer choices
Original selectable question and answer choices
Original selectable question and answer choices
A.2 Tool Interface and Agent Prompt
The full tool schema, agent prompts, MCQ message, cap-exhaustion instruction and video preamble are retained. They describe the evaluated system; forced answer selection applies to scoring rather than clinical-interface abstention.
Tools.
Eleven tools comprise two scaffold tools, four timeline tools, two report tools, two EHR tools and one video tool. Table 5 retains exact names, optional arguments, returns and inclusive/exclusive time-window conventions. Local time and admission-relative hour indices are deliberately distinguished.
Accessible table, original values and citation links
| Tool | Parameters | Returns |
|---|---|---|
| Scaffold (every configuration) | ||
| get_patient_metadata | pat_id? | demographics, ICU admission time, observation window, encounter history |
| convert_utc_local | dt_str, to_timezone | the datetime converted between UTC and local time |
| Timeline (minute-level records) | ||
| get_atomic_actions | hour_index | per-minute active action labels for one hour from admission |
| query_actions | labels?, start_hour?, end_hour? | minutes active per label over the range; per-hour counts; per-label per-hour minutes |
| find_first_occurrence | labels, start_hour?, end_hour? | first minute where any listed label is active (hour, minute, timestamp) |
| find_moments_of_interest | target_labels?, start_pt?, end_pt?, min_rise_pct?, drug_window_min? | rises and falls of a 15-minute rolling activity signal, with the medications given nearby |
| Report (hourly reports) | ||
| list_hourly_reports | pat_id? | the (date, hour) pairs with a report |
| get_hourly_report | date, hour | the six-section report for that hour |
| EHR | ||
| query_medications | start_pt, end_pt, name_filter?, class_filter?, max_rows? | medication administration events in the window: time, name, action, dose, route, class |
| query_flowsheet | disp_name_filter?, start_pt?, end_pt?, max_rows? | flowsheet rows (time, measurement, value, unit); without a filter, the measurement names available |
| Video | ||
| query_video_minute | minute_pt, prompt | the perception VLM’s answer to the prompt for each of the two camera views of that minute |
Agent prompts.
The full prompt directs evidence gathering, time-window checks, filter discovery, output-cap handling and cross-checking. Reduced configurations remove unavailable-tool lines. The exact plain-text payload retains patient placeholders, punctuation and all working rules; no abbreviated paraphrase substitutes for it.
You are an analytic assistant embedded in a clinical video-viewer app for ICU patients. Answer the multiple-choice question about the patient currently on screen. Investigate with your tools BEFORE deciding; base your answer on what the tools actually return, not on priors.
Tools available to you:
- get_patient_metadata: Get demographics, ICU admission time, observation window length, and encounter history for a patient. Defaults to the patient currently on screen if pat_id is omitted.
- convert_utc_local: Convert between UTC and local time.
- list_hourly_reports: List all hourly markdown reports available for a patient. Each entry has a date (YYYY-MM-DD) and an hour (0-23). Defaults to the current patient if pat_id is omitted.
- get_hourly_report: Fetch the full markdown content of a single hourly report. Reports follow a fixed 6-section structure: Patient Movement Summary, Patient-Clinician Interaction, Significant Moments, Medication Context, Activity-Medication Alignment, Brief Summary. Defaults pat_id to the current patient.
- get_atomic_actions: Return minute-by-minute atomic action detections for one hour of the patient's stay. hour_index=0 is the first hour from ICU admission. Each minute lists currently-active action labels (from the atomic action taxonomy). Defaults pat_id to the current patient.
- query_actions: Aggregate atomic action counts across a range of hours. Returns: (a) total minutes each label was active over the window, (b) per-hour counts of minutes with any selected label active, and (c) per-label per-hour minutes active (dict[label] -> list over the hour range) -- use (c) to answer questions like 'which hour had the most X'. Filter by specific labels or omit to include all. start_hour/end_hour are zero-indexed from ICU start (end_hour is exclusive). Defaults pat_id to the current patient.
- find_first_occurrence: Find the earliest minute where any of the given action labels was active. Returns hour_index, minute (0-59), and UTC timestamp. Prefer this over reading hourly reports or scanning atomic actions when the user asks 'when did X first happen'. Optionally restrict to a range via start_hour/end_hour (zero-indexed from ICU start, end_hour exclusive). Defaults pat_id to the current patient.
- find_moments_of_interest: Detect 'moments of interest' -- rises and falls in a 15-minute rolling signal built from the atomic-action labels you select -- and cross-reference each detected blurp with medication events given nearby. The signal at minute t is the fraction of the previous 15 minutes in which any selected label fired. **You must choose target_labels to match the question.** Examples:
- query_medications: Return medication-administration-record (MAR) events for the current patient in a local-time window. Each event has taken_time_pt, med_name (display name), mar_action (Given / New Bag / Bolus / Stopped / etc.), dose (rate + unit or bolus + unit), route, and drug class (pharmacologic + therapeutic). Use this to answer questions like 'which meds were administered +/-45 min of X?', 'when was the last dose of oxycodone?', or 'what sedatives were running at 03:00?'. TIME: pass local-time strings 'YYYY-MM-DD HH:MM' (24-hour). Windows are inclusive on both ends. OPTIONAL FILTERS: name_filter (case-insensitive substring on the med name) and class_filter (substring on pharmacologic/therapeutic class, e.g. 'OPIOID', 'SEDATI', 'ANTIBIOTIC'). Results are capped at max_rows (default 50).
- query_flowsheet: Query nursing-flowsheet measurements for the current patient: vitals (Heart Rate, Pulse, SpO2, Resp, Arterial Systolic BP, Mean Arterial Pressure, Arterial Diastolic BP, BP), sedation/agitation (Patient RASS Score, Anxiety Level, Pain Scale Type), and nursing assessments (Activity, ADLs, Communication, Ectopy, Cardiac Rhythm, etc.). Each row has recorded_time_pt, disp_name, value, and unit. DISCOVERY: call without disp_name_filter to see which measurement types exist for this patient (list of disp_name + count). QUERY: pass disp_name_filter (case-insensitive substring, e.g. 'RASS', 'Heart Rate', 'SpO2') to fetch the actual measurements. Optionally scope with start_pt / end_pt (local-time 'YYYY-MM-DD HH:MM'). Use this for questions like 'when was the patient most deeply sedated?' (RASS), 'what was the peak heart rate?', 'when did BP swings occur?', 'what was the anxiety/pain trend?'.
- query_video_minute: Look at the ICU surveillance video for ONE specific minute and answer a free-form question about what is visible -- patient movements/reactions, what a clinician is doing, devices/objects, body position, who is in the room.
Treat its answer as EVIDENCE TO WEIGH, not ground truth: the video reader can miss brief events or misread motion, and the two camera views may disagree. When the answer matters, re-query (a different prompt or an adjacent minute) and cross-check against your other tools before committing.
This tool is useful for zooming in on a single minute of interest. You can call it on the SAME minute several times with DIFFERENT prompts to probe different details (e.g. first ask what the clinician is doing, then whether the patient moved). It returns BOTH camera angles (left and right view) answered separately.
TIME: everything here is local time. Pass the minute exactly as it appears in the question, 'YYYY-MM-DD HH:MM' (24-hour, local time); the tool finds the right clip for you and you never convert timezones. The clip is one minute long, so if an event might straddle a boundary, also query the adjacent minute.
If a tool returns an error or nothing useful, adjust and try another approach. Tools that take a patient id default to the current patient -- omit pat_id. You MUST select exactly one option as your single best guess, even if you are unsure or the data is incomplete. Never refuse and never say you cannot determine the answer.
Patient context (from get_patient_metadata):
- Video and activity data (hourly reports, atomic actions, query_video_minute) cover {start} to {end} local time ({n_hours} hours).
- hour_index counts whole hours from {start}: {hour_examples}. To convert a time to hour_index, subtract {start} and take the whole number of hours; write the subtraction down.
- EHR data (query_medications, query_flowsheet) covers the whole hospital admission, {ehr_start} to {ehr_end} local time, which extends beyond the video window.
Every date or time in a question refers to THIS stay in local time. Never guess a year or a date outside these windows.
Working rules (follow them strictly):
1. For "during the stay", "first", "last", "how many", "which day" questions about medications or flowsheet data, query the ENTIRE admission in ONE call (start_pt = {ehr_start}, end_pt = {ehr_end}); for activity data use start_hour 0 to end_hour {n_hours}. Do not step through the stay day by day.
2. Zero rows or zero events never prove absence. Before concluding "none", re-check the dates against the patient context, remove filters, widen the window, and try a shorter substring. If a result carries a "hint" field, follow it.
3. Medications: use name_filter ALONE with a short substring of the drug name (e.g. 'norepi', 'oxycod', 'hepar'). Do not add class_filter unless the question is about a drug class, and never combine the two.
4. Flowsheet measurement names must be discovered, not guessed: call query_flowsheet with NO disp_name_filter first to see the exact names, then filter with a short substring taken from that list (e.g. 'CAM' rather than 'CAM-ICU', 'Temp' rather than 'Temperature').
5. Outputs are capped. If n_events_returned < n_events_total, n_rows_returned < n_rows_total, or a "_truncated" note appears, narrow the filter or time window (or raise max_rows) before drawing any conclusion from a partial list.
6. Prefer totals the tools already computed (per_label_minutes_active, n_events_total, n_rows_total) over adding long arrays by hand. If you must count or add over a long list, do it in blocks of 10 and keep a running total.
7. For what the patient was doing over a several-minute window, first compute the hour_index, then use get_atomic_actions for that hour (minute-level labels) together with the hourly report; use query_video_minute to check the specific minute (and the adjacent minute) when the question is about a single moment.
8. Investigate with at least two different tools or queries before answering whenever the tools can bear on the question, and cross-check them against each other.
9. Keep the reasoning short and end with the line: Answer: <LETTER>You are answering a multiple-choice question about an ICU patient. You do NOT have access to the patient's video, activity data, or medical records, and you have no tools. Answer as best you can from general reasoning. You MUST select exactly one option as your single best guess, even if you are unsure or the data is incomplete. Never refuse and never say you cannot determine the answer.
QUESTION: {question}
OPTIONS:
{options, one per line as 'A. ...'}
You must choose exactly one option even if uncertain -- never refuse.
End your reply with a line in exactly this form:
Answer: <LETTER>You have used all available tool calls. Based on what you have found so far, commit to your single best-guess answer now. End with a line: Answer: <LETTER>
Video tool.
The video preamble asks for visible subtle motion with timestamps and rejects unsupported inference. Both camera answers return separately for reconciliation. Re-querying is possible, but all such tool calls consume the shared 25-call budget.
You are an expert ICU video analyst. Patients here are often sedated, so subtle patient movement (a brief head turn, eye flutter, blink, cough, swallow, finger/limb twitch) is clinically relevant. Review the one-minute clip carefully and list ONLY movements you can actually see, each with an approximate timestamp (mm:ss). Do NOT infer or assume movements that are not clearly visible -- if the patient appears still, say so explicitly. Then answer the custom prompt below.
A.3 Temporal Hierarchy Construction Details
The minute representation stores fifteen binary labels plus two captions. UNABLE TO IDENTIFY becomes negative for timeline construction, whereas perception evaluation excludes abstained labels. Flash Lite abstains once among 29,775 benchmark labels and zero times among 198,810 sampled cohort labels; some other models abstain about 3%, making cross-model metric interpretation sensitive to that policy.
Perception prompt.
The complete fifteen-category prompt and JSON schema remain exact. Its initial instructions request a short summary and the example includes one, while another output-rule line forbids free-text descriptions. This source-level tension is preserved rather than silently edited. Categories concern visible movement/contact and can co-occur.
MAIN INSTRUCTIONS:
Your objective is to analyze video frames extracted from a 1-minute Intensive Care Unit (ICU) hospital patient surveillance video recorded by a fixed, corner-mounted camera.
This is STRICTLY a FINE-GRAINED video understanding task.
You will receive 60 images sampled every 1 second, totaling 1 minute.
Your ONLY task is to decide, for each predefined category below, whether the corresponding patient action or patient-clinician interaction is present in the video. Then write a short 1-2 sentence summary of this video (STRICTLY keep this short) summary.
You must ONLY output a structured JSON object with categorical labels and Short summary.
KEY VISUAL TASKS (TAXONOMY)
PATIENT MOTION -- ARMS/HANDS
1) Grabbing medical equipment
- The patient is actively grasping or holding medical devices
(e.g., ventilator mask, tubing, IV lines, monitor cables).
2) Using or grabbing non-medical objects
- The patient is using or holding non-medical items
(e.g., cup, towel, tissue, phone, food, remote).
3) Gesticulating
- The patient is making expressive arm or hand movements while communicating
(e.g., waving, pointing, hand gestures while speaking).
4) Adjusting pillows or bedsheets
- The patient pulls, pushes, or rearranges pillows, blankets, or sheets.
5) Other arm movements
- Any noticeable arm/hand motion such as fidgeting, lifting/dropping arms,
scratching, or non-specific movements.
- This category can co-occur with other arm/hand categories above.
PATIENT MOTION -- TORSO
6) Patient changes body position in bed
- The patient changes posture in bed
(e.g., lying flat to leaning on one side, sitting up to lying down).
7) Other torso movement
- Other significant torso movements
(e.g., quick or repeated body movements, jerking, agitation-like movements).
PATIENT MOTION -- HEAD/NECK
8) Patient is looking toward other people in the room
- The patient's gaze is clearly directed at another person.
9) Patient turns or moves head/neck
- Visible head or neck movement
(e.g., turning left/right, nodding, tilting up/down).
PATIENT MOTION -- LEGS
10) Leg movements related to mobility
- The patient is making leg motions associated with activities of daily living
(e.g., standing up, sitting down, walking, transferring to or from the bed).
11) In-bed leg movement
- The patient is making leg movements while they remain in bed
(e.g., leg fidgeting, repositioning, leg movements under the covers).
PATIENT-CLINICIAN INTERACTIONS
12) Clinician touches patient or uses tool/device on patient
- A clinician or other person makes physical contact with the patient
OR applies a tool or device directly to the patient's body.
13) Patient communication
- The patient appears to be speaking, mouthing words, or nodding in a way that
responds to another person.
14) Object exchange
- An item is being handed from a person to the patient or from the patient
to another person (e.g., cup, towel, tool, brush, phone).
15) Patient is being physically restrained or stabilized
- Another person is clearly holding or stabilizing part of the patient's body
in a way that limits or controls movement.
DECISION RULES
- Each category is INDEPENDENT. Multiple categories can be YES simultaneously
for the same video. For example, a patient may be grabbing medical equipment
(cat 1) while also performing other arm movements (cat 5) at the same time.
Evaluate each category on its own merits.
- Base ALL judgments only on what is clearly visible in the frames. - Do NOT infer pain, emotions, intentions, diagnoses, or clinical reasoning. - If visibility is poor or you are not confident, use "UNABLE TO IDENTIFY" for that label. - Every label MUST be exactly one of:
- "YES"
- "NO"
- "UNABLE TO IDENTIFY"
- Do NOT add extra fields or narrative text.
OUTPUT RULES
- Return a single JSON object with fixed keys for the video ID and all labels. - Do NOT include explanations, timestamps, or free-text descriptions. - Use the exact key names and allowed values shown below.
EXAMPLE OUTPUT FORMAT
{
"video_id": "<date>/<time>_<room>_lt.mp4",
"labels": {
"patient_grabbing_medical_equipment": "NO",
"patient_using_non_medical_objects": "NO",
"patient_gesticulating": "NO",
"patient_adjusting_bedding": "NO",
"patient_other_arm_movements": "NO",
"patient_position_change": "NO",
"patient_other_torso_movement": "NO",
"patient_looking_at_others": "NO",
"patient_head_turning": "NO",
"patient_leg_ADL_movement": "NO",
"patient_leg_in_bed_movement": "NO",
"clinician_patient_contact": "YES",
"patient_communication": "NO",
"clinician_patient_object_exchange": "NO",
"patient_restrained": "NO"
},
"summary": "A clinician is present and makes physical contact with the patient, while the patient remains mostly still with no notable movements."
}
You must ALWAYS follow this structure exactly.Hourly report prompt.
Both full hourly-report prompts remain, including six numbered sections, mandatory timing, camera reconciliation, medication event fields and observational-only alignment. Prompt instructions against clinical inference do not guarantee compliance; the independent report audit documents actual errors.
You are an ICU behavioral summarization model. You will receive up to 60 minutes of Task-1 video caption outputs (one JSON entry per minute) from two camera views (left and right) of an ICU patient room, plus a list of medication administration events relevant to this hour. Your job is to generate a structured behavioral report for the observation period based purely on visible evidence from both camera views. You must never infer sedation level, intent, pain, emotion, or clinical state. When both camera views are available for a given minute, reconcile them: if either view detects an action, treat it as observed. Note discrepancies only when they are informative (e.g., one view blocked by equipment). When only one camera view is available, note which view and proceed. TIMING REQUIREMENT (IMPORTANT): - All behavioral findings must cite timing. - Use minute timestamps (e.g., min 14, min 27) AND/OR continuous ranges (e.g., min 10-15). - No event may be mentioned without a timing reference. ALLOWED TIMING FORMATS: - "arm movement observed at min 12" - "clinician contact min 21-24" - "no movement detected min 0-18" Your output must follow the 6-section template provided in the user prompt EXACTLY.
Generate the behavioral report for this observation hour.
REQUIRED FORMAT:
Observation window: [START_TIME] to [END_TIME] (up to 60 minutes)
1. Patient Movement Summary
- Overall movement level (high / moderate / low / none)
- Most common movement types
- MUST cite minute timestamps or ranges for all movement observed
- Note any discrepancies between left and right camera views
2. Patient-Clinician Interaction Summary
- Whether interaction occurred
- MUST cite timings for clinician contact, communication, restraint, exchanges
3. Significant Moments of Interest
- Sudden spikes in movement, first movement after stillness, notable behavior
- MUST include timing (single minute or range)
- If prolonged stillness >= 10 min, cite the stillness window
4. Medication Context
- List medications active or administered during this hour with timestamps
- Include drug name, dose, route, and action (Given/Rate Change/New Bag/etc.)
- Note any continuous infusions that are running
5. Activity-Medication Alignment (observational only)
- Timing-based associations ONLY (e.g., "movement at min 22-28 while dexmedetomidine infusing")
- No clinical cause/effect interpretation ever
6. Brief Summary (<= 5 sentences)
- Compact overall description of the hour
- Note uncertainty if many labels = "UNABLE TO IDENTIFY"
- Note if camera coverage was incomplete
Output MUST contain all 6 sections and MUST cite times when describing events.A.4 TRAJECTORY-144 Question Creation and Filtering
Humans create timelines/questions, strengthen distractors and review correctness. Blind-LLM accuracy flags questions for revision, but 25 final questions remain answerable at least 80% of the time without patient data. The benchmark therefore retains linguistic priors and earlier draft-question influence despite fixed final prompts.
Evidence types.
The 144 questions split into 33 video moments under five minutes, 16 windows of 5–30 minutes, 61 longer video spans, 26 record-only and eight cross-modal items. Fine-grained cross-modal percentages therefore rest on a small question count. Patient-level generalization is not tested.
Temporal anchoring.
Sixty questions name a clock time, 34 a coarser hour/day reference and fifty none. The agent can anchor through supplied times, EHR events, action queries or reports. Table 6 uses an independent run scoring 61.1%, distinct from the 61.0% main run; its full count/accuracy breakdown remains.
Agent prompt development.
Prompt refinement uses earlier draft benchmark questions. The exact final evaluated forms are excluded from development, and one fixed prompt serves all reported agents. This limits wording-specific tuning but does not supply a separate unseen-stay development/test design.
Accessible table, original values and citation links
| Time reference in the question | Questions | Accuracy |
|---|---|---|
| Clock time, span under 5 minutes | 33 | 30.9 |
| Clock time, span 5–30 minutes | 15 | 57.3 |
| Clock time, span over 30 minutes | 12 | 50.0 |
| Hour | 13 | 50.8 |
| Part of day (e.g., morning, afternoon) | 16 | 66.2 |
| Day (e.g., November 23) | 5 | 56.0 |
| No explicit time, anchored on clinical record | 22 | 96.4 |
| No explicit time, anchored on video data | 28 | 78.6 |
A.5 Patient Cohort
Figure 3 retains the full 2617→280 selection flow and cohort characteristics, including median age 66, 65.4% male, 56.1% invasive ventilation and 18.9% ever-positive delirium screening. Its embedded characteristics table is preserved along with all sixteen separately numbered tables.
A.5.1 Patient Cohort Selection
Exclusions sequentially enforce >24-hour/<2-week stay, sedative/analgesic exposure, non-neurological admission, dual cameras, no intra-ICU transfer, full continuous feed and bed-centered views. Most original admissions do not enter the final cohort. The resulting technical completeness supports analysis while restricting coverage of routine ICU conditions.
A.5.2 Electronic Health Records
Demographics/encounter data establish patient context, ADT defines stay windows, medication events feed tools/reports, and nursing flowsheets provide vitals, RASS and CAM-ICU. Questions can access full-admission EHR beyond the video window. The text’s reference to cohort characteristics in Table 15 is retained, although overall characteristics also appear in Figure 3.
A.6 Token Cost of Raw Video
Raw paired-camera video consumes millions of tokens per hour under the paper’s settings. Table 7 retains all resolutions, frame rates and estimates. Gemini’s quoted ~1.05M tokens for 950 stills lies at the stated 1M window limit; these are measured/configuration-specific budgets, not timeless model specifications. Sparse sampling misses short events even if long-duration coverage appears complete.
Accessible table, original values and citation links
| Input | Gemini 3 Family (tokens) | Qwen3-VL 8B (tokens) |
|---|---|---|
| One minute, one camera view | 19,823 | 8,640 |
| One minute, both views | 39,646 | 17,280 |
| One hour, both views | 2.38M | 1.04M |
| One day, both views | 57.1M | 24.9M |
| Median stay (3.3 days) | 188M | 82M |
| Longest stay in cohort (14 days) | 799M | 349M |
| TRAJECTORY-144 span (264 h) | 628M | 274M |
| One 512288 frame | 1,100 | 144 |
| 950 stills over a stay (long-context baseline) | 1.05M | 0.14M |
A.7 Baseline Implementations
Each configuration is scored on 718 placement trials. The main paired bootstrap averages placements within question and resamples 144 questions 10,000 times. Appendix McNemar tests treat placements as independent and are explicitly approximate. Neither inferential procedure creates additional independent patients.
Text-only.
The no-tool agent tests language priors with the same forced-choice scoring loop. Its above-chance results measure guessability, not recovered patient evidence.
Socratic Models.
The Socratic pipeline performs a separate free-text captioning pass with published HourVideo prompts. Captions exceeding context are uniformly subsampled; Gemini keeps every second/third minute for the two stays, while open-weight contexts keep much less. This differs from TRACER’s structured labels and active evidence access.
Socratic baselines with clinical records and both cameras.
Adding all medication/flowsheet data lifts caption-baseline accuracy to 45.0%. Adding both camera captions under the same context budget increases stride and gives 42.7%. This comparison controls additional modalities while exposing a coverage/detail tradeoff; it does not imply a second camera is intrinsically harmful.
Long-context VLM.
The raw-frame baseline selects chronological midpoint frames without timestamps, usually from the left camera. At 950 frames it samples every 6.0 or 10.7 minutes across the two stays. Many brief events can fall between samples. This is a particular sparse long-context protocol, not exhaustive video inspection.
Long-context baseline with the clinical record.
The strongest baseline prepends the full EHR block and halves video sampling to 475 frames, fitting roughly 654k/943k tokens for the two stays. It reaches 49.3% overall and 83.8% on record-only questions. Table 8 retains all budgets, coverage and category results.
Accessible table, original values and citation links
| Inputs | Coverage | Tokens | 5 min | 5–30 min | 30 min | EHR | Cross | All |
|---|---|---|---|---|---|---|---|---|
| Left camera captions | 2.8k / 3.4k | 1.0M | 34.5 | 32.5 | 39.2 | 16.2 | 47.5 | 33.7 |
| Left camera captions + EHR | 2.8k / 1.7k | 1.0M / 0.9M | 30.3 | 32.5 | 43.3 | 76.2 | 42.5 | 45.0 |
| Both cameras’ captions + EHR | 1.4k / 0.9k | 0.7M / 0.7M | 32.7 | 31.2 | 35.9 | 76.2 | 50.0 | 42.7 |
| 950 frames | 950 / 950 | 1.1M | 35.8 | 32.5 | 38.0 | 14.6 | 25.0 | 31.9 |
| 475 frames | 475 / 475 | 0.5M | 26.7 | 35.0 | 36.6 | 17.7 | 22.5 | 29.9 |
| 475 frames + EHR | 475 / 475 | 0.7M / 1.0M | 34.5 | 30.0 | 48.9 | 83.8 | 40.0 | 49.3 |
| TRACER harness, all tools (Table 1 run) | – | – | 36.4 | 52.5 | 63.7 | 96.2 | 45.0 | 61.0 |
Open-weight question answering for Socratic baselines.
Using fixed Flash Lite captions, open-weight Socratic answerers score 23.9–29.3%, versus 57.4–67.9% with their harness configurations. Context limits reduce caption coverage to roughly 183–391 minutes per stay. The result combines representation/tool benefits with much broader accessible evidence.
Accessible table, original values and citation links
| Answering model | Minutes | 5 min | 5–30 min | 30 min | EHR | Cross | All | TRACER |
|---|---|---|---|---|---|---|---|---|
| Gemini 3.1 FL (1M) | 2.8k / 3.4k | 34.5 | 32.5 | 39.2 | 16.2 | 47.5 | 33.7 | 61.0 |
| gpt-oss-20b (131K) | 379 / 391 | 33.3 | 26.3 | 22.7 | 14.6 | 20.0 | 23.9 | 57.4 |
| Gemma-4 31B (131K) | 379 / 377 | 33.9 | 25.0 | 31.4 | 23.1 | 22.5 | 29.3 | 67.9 |
| Qwen3.5 9B (65K) | 183 / 185 | 30.3 | 28.8 | 30.8 | 18.5 | 10.0 | 27.1 | 61.6 |
A.8 Model Instantiation and Computational Cost
All model roles remain frozen. Flash Lite supplies cohort perception and hourly reports, although hourly reports are generated only for the two QA stays. Authorized proprietary processing is restricted to the Gemini family by the study’s data-use agreement; this is an implementation constraint, not an endorsement of arbitrary data upload.
Computational cost.
Background construction costs two perception calls per patient-minute plus one report call per hour. Per-question Gemini 3.5 Flash use is reported as ~160k agent tokens plus ~132k video-tool tokens, about 292k total, with a heavy tail. Reusable preprocessing is excluded from that per-question comparison. The appendix’s 9.8-turn average and Table 11’s 9.0 mean calls describe the reported accounting without a full reconciliation.
Sampling and serving settings.
Gemini uses API-default generation settings. Open-weight perception uses temperature .6, top-p .95 and top-k 20; baselines use greedy decoding, with thinking disabled in vLLM. Video sampling differs by model family. Exact model/version and sampling choices matter for reproduction and comparisons.
A.9 Perception Benchmark Details
Perception analysis covers submission-time cost, label prevalence and annotation reliability. It does not validate every predicted cohort minute against human ground truth.
Accuracy-cost tradeoff.
Figure 4 preserves the September-2026 API list-price cost axis and separate self-hosted MI250X device-hour axis. Those units should not be combined as a universal monetary ranking. Flash Lite is chosen for this study’s accuracy/cost tradeoff, rather than highest detection F1.
Prevalence.
Common benchmark actions occur in about 70% of selected clips, while rare categories are under 5%. Actual cohort prevalence is lower because benchmark sampling intentionally spans activity levels. Aggregate performance on that enriched set may not transfer directly to precision at routine low prevalence.
Annotation reliability.
A blinded author reannotates twenty randomly selected clips with the finalized rubric. Across 300 binary decisions, agreement is 92.7% and κ=.81, with clip-bootstrap 95% interval [.74,.87]. This supports label consistency on a small audit and does not replace full-cohort validation.
A.10 Hourly Report Faithfulness
Ten reports contain 153 verifiable claims: 129 supported and 24 errors. Twelve errors partially mismatch timing; three each misstate movement level, action absence, medication absence and repeated medication absence. Named medications match recorded details, but false absence propagates between sections. The audit checks reports against model minute records/EHR rather than independently verifying every raw video action, and cannot detect omissions. Its 84.3% support rate is not an end-to-end clinical correctness rate.
Accessible table, original values and citation links
| Report section | ||||||||
|---|---|---|---|---|---|---|---|---|
| Movement | Interaction | Moments | Medication | Alignment | Summary | All | % | |
| Verifiable claims | 41 | 36 | 28 | 16 | 12 | 20 | 153 | |
| Supported | 31 | 32 | 25 | 13 | 9 | 19 | 129 | 84.3 |
| Errors | 10 | 4 | 3 | 3 | 3 | 1 | 24 | 15.7 |
| Only partial timing match | 7 | 1 | 3 | 0 | 0 | 1 | 12 | 7.8 |
| Movement summary misleading | 3 | 0 | 0 | 0 | 0 | 0 | 3 | 2.0 |
| Incorrect action absence | 0 | 3 | 0 | 0 | 0 | 0 | 3 | 2.0 |
| Incorrect medication absence | 0 | 0 | 0 | 3 | 0 | 0 | 3 | 2.0 |
| Carry over of incorrect information | 0 | 0 | 0 | 0 | 3 | 0 | 3 | 2.0 |
| Support rate (%) | 76 | 89 | 89 | 81 | 75 | 95 | 84 | |
A.11 How does TRACER Perform Across Different Agent Models?
The twelve configurations separate agent and video-tool changes. Gemini 3.5 Flash falls 76.9→70.7 with weaker fixed video perception and reaches the call cap on 10.7% of trials in that condition. Open-weight agents can be competitive when proprietary perception remains. Approximate placement-level McNemar significance is retained with its dependence caveat.
Accessible table, original values and citation links
| Agent model | All | 5 min | 5–30 min | 30 min | EHR | Cross | Avg. Calls |
|---|---|---|---|---|---|---|---|
| Gemini 2.5 Flash Lite | 49.2 | 32.7 | 35.0 | 53.9 | 69.2 | 45.0 | 3.7 |
| Gemini 2.5 Flash | 60.3 | 33.9 | 56.2 | 64.0 | 90.8 | 50.0 | 4.7 |
| Gemini 3 Flash | 70.8 | 44.8 | 66.3 | 77.0 | 91.5 | 72.5 | 12.3 |
| Gemini 3.1 Flash Lite | 61.0 | 36.4 | 52.5 | 63.7 | 96.2 | 45.0 | 8.9 |
| Gemini 3.5 Flash Lite | 61.1 | 40.0 | 52.5 | 59.8 | 99.2 | 52.5 | 6.8 |
| Gemini 3.5 Flash | 76.9 | 54.5 | 75.0 | 80.7 | 99.2 | 72.5 | 9.0 |
| Qwen3.5 9B | 57.9 | 26.7 | 43.8 | 63.3 | 95.4 | 52.5 | 10.4 |
| Qwen3-VL 8B | 42.1 | 19.4 | 32.5 | 45.7 | 68.5 | 42.5 | 4.7 |
| gpt-oss-20b† | 57.4 | 32.1 | 41.2 | 62.5 | 93.1 | 40.0 | 3.7 |
| Qwen3.5 9B† | 61.6 | 35.2 | 45.0 | 67.3 | 95.4 | 50.0 | 9.0 |
| Gemma-4 31B† | 67.9 | 40.0 | 61.2 | 73.7 | 100.0 | 47.5 | 5.4 |
| Gemini 3.5 Flash† | 70.7 | 35.8 | 57.5 | 80.3 | 98.5 | 77.5 | 9.5 |
A.12 Per-Model Cumulative Ablation
All five agents’ cumulative tables retain question-type and stay-A/B results. Performance differs markedly between stays: main Flash Lite is 75.8% on A and 50.8% on B. Reports do not improve every category/agent significantly; the Gemini 3.5 report increment has approximate p=.06. Adding video also reduces some Qwen category/overall scores. Two-stay variation limits broad claims from pooled averages.
Accessible table, original values and citation links
| Video | EHR | Cross-modal | Stay | All | ||||
| Agent tools | 5 min | 5–30 min | 30 min | Record only | Video + EHR | A | B | |
| Gemini 3.5 Flash | ||||||||
| Text-only | 37.0 | 32.5 | 51.4 | 21.5 | 27.5 | 35.9 | 41.6 | 39.3 |
| + timeline | 37.6 | 57.5 | 67.5 | 42.3 | 20.0 | 62.0 | 45.6 | 52.4 |
| + reports | 40.6 | 72.5 | 70.5 | 38.5 | 30.0 | 70.2 | 45.9 | 55.8 |
| + EHR | 39.4 | 76.2 | 71.1 | 100.0 | 75.0 | 84.7 | 59.5 | 69.9 |
| + video (TRACER) | 54.5 | 75.0 | 80.7 | 99.2 | 72.5 | 88.5 | 68.9 | 76.9 |
| Gemini 3.1 Flash Lite | ||||||||
| Text-only | 24.2 | 18.8 | 40.2 | 26.9 | 25.0 | 26.8 | 33.8 | 30.9 |
| + timeline | 23.0 | 32.5 | 55.3 | 26.2 | 22.5 | 50.5 | 29.8 | 38.3 |
| + reports | 24.8 | 48.8 | 62.9 | 31.5 | 37.5 | 57.9 | 36.9 | 45.5 |
| + EHR | 23.6 | 36.2 | 58.6 | 96.2 | 47.5 | 71.2 | 42.5 | 54.3 |
| + video (TRACER) | 36.4 | 52.5 | 63.7 | 96.2 | 45.0 | 75.8 | 50.8 | 61.0 |
| Qwen3.5 9B | ||||||||
| Text-only | 31.5 | 27.5 | 43.9 | 20.0 | 40.0 | 35.3 | 34.4 | 34.7 |
| + timeline | 21.8 | 40.0 | 59.3 | 22.3 | 32.5 | 52.1 | 32.2 | 40.4 |
| + reports | 26.1 | 45.0 | 58.0 | 23.8 | 27.5 | 51.9 | 34.0 | 41.4 |
| + EHR | 32.1 | 42.5 | 62.5 | 95.4 | 62.5 | 74.8 | 48.4 | 59.2 |
| + video (TRACER) | 26.7 | 43.8 | 63.3 | 95.4 | 52.5 | 74.6 | 46.4 | 57.9 |
| Qwen3-VL 8B | ||||||||
| Text-only | 29.7 | 28.7 | 42.6 | 25.4 | 32.5 | 34.9 | 34.1 | 34.4 |
| + timeline | 21.2 | 27.5 | 39.9 | 22.3 | 35.0 | 39.3 | 24.9 | 30.8 |
| + reports | 19.4 | 48.8 | 46.7 | 18.5 | 22.5 | 39.5 | 30.6 | 34.2 |
| + EHR | 17.0 | 33.8 | 43.5 | 70.0 | 25.0 | 53.8 | 30.6 | 40.1 |
| + video (TRACER) | 19.4 | 32.5 | 45.7 | 68.5 | 42.5 | 55.7 | 32.7 | 42.1 |
| gpt-oss-20b (video tool served by Gemini 3.1 Flash Lite) | ||||||||
| Text-only | 22.4 | 17.5 | 36.5 | 22.3 | 42.5 | 27.5 | 29.9 | 28.9 |
| + timeline | 22.4 | 36.2 | 56.5 | 16.2 | 35.0 | 47.4 | 31.4 | 38.0 |
| + reports | 25.5 | 38.8 | 62.1 | 27.7 | 30.0 | 55.8 | 34.3 | 43.1 |
| + EHR | 26.7 | 40.0 | 59.3 | 90.8 | 52.5 | 74.8 | 41.2 | 55.0 |
| + video (TRACER) | 32.1 | 41.2 | 62.5 | 93.1 | 40.0 | 73.4 | 46.4 | 57.4 |
A.13 Single Tool Family Ablations
Single-family ablations retain all reported model-specific category results. EHR alone excels on record questions; video alone helps short moments but is weaker on long spans. These family effects depend on the shared scaffold and prompts, with incomplete independent factorial control of all interactions.
Accessible table, original values and citation links
| Agent tools | All | 5 min | 5–30 min | 30 min | EHR | Cross |
|---|---|---|---|---|---|---|
| Gemini 3.5 Flash | ||||||
| Reports only | 51.0 | 44.2 | 72.5 | 55.4 | 36.9 | 47.5 |
| EHR only | 59.6 | 49.1 | 47.5 | 49.5 | 100.0 | 72.5 |
| Video only | 46.4 | 54.5 | 76.2 | 43.9 | 30.0 | 27.5 |
| All tools | 76.9 | 54.5 | 75.0 | 80.7 | 99.2 | 72.5 |
| Gemini 3.1 Flash Lite | ||||||
| Reports only | 37.5 | 27.9 | 38.8 | 44.5 | 31.5 | 40.0 |
| EHR only | 45.3 | 26.1 | 20.0 | 40.7 | 95.4 | 47.5 |
| Video only | 34.5 | 38.2 | 45.0 | 36.8 | 22.3 | 20.0 |
| All tools | 61.0 | 36.4 | 52.5 | 63.7 | 96.2 | 45.0 |
| Qwen3.5 9B | ||||||
| Reports only | 37.7 | 29.1 | 46.2 | 42.2 | 31.5 | 42.5 |
| EHR only | 43.0 | 23.6 | 27.5 | 33.5 | 93.8 | 60.0 |
| Video only | 30.2 | 26.7 | 31.2 | 34.3 | 23.1 | 35.0 |
| All tools | 57.9 | 26.7 | 43.8 | 63.3 | 95.4 | 52.5 |
| gpt-oss-20b (video tool served by Gemini 3.1 Flash Lite) | ||||||
| Reports only | 32.8 | 26.7 | 43.8 | 33.9 | 30.0 | 37.5 |
| EHR only | 37.3 | 26.1 | 27.5 | 23.4 | 93.1 | 27.5 |
| Video only | 28.4 | 28.5 | 43.8 | 32.3 | 14.6 | 12.5 |
| All tools | 57.4 | 32.1 | 41.2 | 62.5 | 93.1 | 40.0 |
A.14 Delirium and Atomic Action Frequency Analysis
Equation 1 pools detected action minutes within each ever-positive/never-positive group, weighting longer-observed patients more. Equal-patient weighting preserves the largest purposeful-action reductions but shifts several near-one ratios. Patient bootstrap intervals and Holm/BH-adjusted per-patient tests provide complementary uncertainty measures: head turning’s bootstrap interval excludes no difference, but its Mann–Whitney p=.094 is not significant. Six reduced actions survive Holm at .05; the three strongest survive below 1e-4, while gesticulation’s adjusted p=.0015 remains below .05. Delirium patients have longer stays and more deeply sedated assessments; effects are not multivariable-adjusted or aligned to positive-screen periods. Complete Tables 14–16 and all formulas remain.
Equation 1
| (1) |
Original formal definition and mathematical context
where denotes the set of patients in group , is the number of patient-minutes in which action is detected for patient , and is the total number of observed minutes for that patient; group frequencies are thus pooled over all observed minutes within each group (minute-weighted).
Accessible table, original values and citation links
| Action | Med. del. | Med. non-del. | MWU | Holm | BH | Perm. (mw / ew) |
|---|---|---|---|---|---|---|
| Using non-medical objects | 0.016 | 0.080 | / | |||
| Object exchange | 0.004 | 0.010 | / | |||
| Leg ADL movement | 0.002 | 0.007 | / | |||
| Clinician–patient contact | 0.423 | 0.349 | / | |||
| Other arm movements | 0.165 | 0.243 | / | |||
| Gesticulating | 0.030 | 0.053 | / | |||
| Grabbing medical equipment | 0.081 | 0.102 | 0.034 | / | ||
| Head turning | 0.363 | 0.399 | 0.094 | 0.56 | 0.15 | / |
| Position change | 0.027 | 0.029 | 0.11 | 0.56 | 0.16 | 0.21 / 0.59 |
| In-bed leg movement | 0.022 | 0.019 | 0.23 | 0.93 | 0.29 | 0.95 / 0.028 |
| Adjusting bedding | 0.037 | 0.038 | 0.25 | 0.93 | 0.29 | 0.23 / 0.52 |
| Communication | 0.240 | 0.257 | 0.27 | 0.93 | 0.29 | 0.080 / 0.20 |
| Looking at others | 0.330 | 0.320 | 0.74 | 0.93 | 0.74 | 0.54 / 1.0 |
| Purposeful activity (any of four) | 0.053 | 0.146 | – | – |
Accessible table, original values and citation links
| Characteristic | Delirium () | Non-delirium () |
|---|---|---|
| ICU LOS, days, median [IQR] | 4.62 [3.01, 8.40] | 3.09 [1.89, 4.84] |
| Observed minutes, median [IQR] | 7,560 [4,680, 12,840] | 4,920 [2,580, 7,440] |
| Invasive mechanical ventilation, (%) | 31 (58.5) | 126 (55.5) |
| Per-patient median RASS, median [IQR] | 0.00 [, 0.00] | 0.00 [, 0.00] |
| RASS assessments , % | 5.0 (of 6,979) | 2.0 (of 18,402) |
Accessible table, original values and citation links
| Minute-weighted | Equal patient weight | |||
| Action | FR | FR [95% CI] | FR | FR [95% CI] |
| Using non-medical objects | 0.41 | 0.39 | ||
| Leg ADL movement | 0.46 | 0.47 | ||
| Object exchange | 0.48 | 0.47 | ||
| Gesticulating | 0.60 | 0.68 | ||
| Other arm movements | 0.62 | 0.69 | ||
| Grabbing medical equipment | 0.67 | 0.80 | ||
| Head turning | 0.85 | 0.90 | ||
| Adjusting bedding | 0.85 | 1.07 | ||
| Position change | 0.85 | 1.06 | ||
| Looking at others | 0.96 | 1.00 | ||
| In-bed leg movement | 1.01 | 1.30 | ||
| Communication | 0.88 | 0.94 | ||
| Clinician–patient contact | 1.20 | 1.18 | ||
| Not in evaluation set (detection performance not validated) | ||||
| Other torso movement | 0.90 | 1.08 | ||
| Clinician restraining | 1.54 | 1.33 | ||
A.15 Ethics, Data, and Code Availability
The appendix states the IRB, access and release boundaries. Publicly shareable code/prompts are distinct from the protected study data.
Human subjects review.
Stanford IRB approval covers the retrospective study, with restricted study-personnel access and secure storage/processing. The paper does not publish raw identifiable bedside recordings.
Data and code availability.
Video and linked records remain unavailable publicly under privacy and IRB restrictions. The stated repository releases taxonomy, prompts and inference/evaluation code. The preserved de-identified question examples are the public manuscript examples, without inventing patient dates or times.