Literature
Display
← All papers

CloudTrack: Scalable UAV Tracking with Cloud Semantics

Yannik Blei · Michael Krawez · Nisarga Nilavadi · Tanja Katharina Kaiser · Wolfram Burgard

· ICRA 2025 ↗

Robotics6 min text · CondensedOriginal paper ↗
Contents

At a glance

CloudTrack combines a remote open-vocabulary detector and vision-language verifier with a lightweight onboard single-object tracker. Text descriptions guide initial detection; the tracker then follows an accepted candidate without rerunning the large models every frame. Experiments measure semantic detection on search-and-rescue imagery and tracking on adapted video sequences. The approach reduces onboard computation while depending on a network and incurring multi-second detection latency.

I INTRODUCTION

A missing-person description can specify attributes beyond a detector’s fixed class vocabulary. Large vision-language models can interpret such descriptions but are expensive and slow on small drone computers. CloudTrack separates semantic initialization from fast local tracking and releases software plus added descriptions/labels for two datasets. It requires no mission-specific weight training, but prompts and reinitialization thresholds are empirically tuned.

Refer to caption
Fig. 1: Detection of a person in UAV footage based on a semantic description. An image is recorded by the UAVs onboard camera. The image is then processed in the cloud by an open vocabulary object detector and a VLM. If the VLM confirms a match with the description, the object is tracked by an onboard real-time object tracker.
Refer to caption
Fig. 2: Overview of the object tracking pipeline. We run an OV object detector and a vision language model in the back-end. UAVs are equipped with a camera and a lightweight object tracker. The onboard object tracker is initialized via the back-end. It ensures close to real-time frame rates on limited hardware. Multiple agents can share the same back-end to achieve better scalability in search and rescue scenarios.

II RELATED WORK

II-A Detection and Tracking for UAVs

Earlier aerial systems use compact detectors, motion models, remote verification, or human feedback. CloudTrack retains a remote verification stage but uses a vision-language model instead of a human to accept initial candidates. Its benchmark evaluates semantic detection and tracking, rather than demonstrating a complete autonomous rescue mission.

II-B Model-free Object Tracking

The evaluated trackers follow a supplied bounding box and do not independently identify the requested person. They include online discriminative models, correlation-filter approaches, and compact learned trackers. Their role is short-term continuity after the semantic system chooses a candidate.

II-C Open-Vocabulary Detection and Tracking

Open-vocabulary systems reuse broad image/text pretraining to accept descriptions beyond fixed class labels. CloudTrack combines Grounding DINO [21] with a separate VLM verification stage instead of training a new end-to-end referring tracker. The chosen models’ capabilities and names refer to this paper’s evaluation configurations.

III METHODS

III-A Overview

A user supplies a description, the server identifies a matching candidate, and an onboard tracker follows it while awaiting human review. The architecture could serve multiple drones from one server, but the paper does not report a multi-drone scaling experiment. A working connection between drone and server is an explicit assumption.

III-B Object-Detection Back-End

Grounding DINO first receives a broad category such as person, because detailed prompts can return people without correctly checking attributes. Each proposed box is cropped with a 50-pixel margin and passed to a VLM with the full description. Accepted boxes initialize tracking. This addresses small-object limitations of whole-image VLM queries, while making runtime grow with the number of candidate boxes. Prompts are tuned separately for each model. The reported rescue prompt explicitly favors saying yes when uncertain, which affects the recall/false-positive tradeoff.

Original system prompt: You are an intelligent AI assistant that helps a drone in a search and rescue mission. If in doubt, rather say yes.

Original user prompt: The drone needs to find an injured person. Would you recommend sending help to the person in the image? Start your justification with ‘Lets analyze the image’.

III-C Object Tracking Front-End

Streaming every frame through the remote semantic detector would make tracking sensitive to network delay and throughput. CloudTrack instead targets at least ten frames per second locally on a Raspberry Pi 4 or Jetson Orin NX. Six lightweight trackers are benchmarked, and the large models are used primarily for initial detection or recovery.

III-D Re-Initialization

When a tracker supplies a confidence score, a threshold triggers remote reinitialization. Trackers without such a score receive no reinitialization in these experiments. If several accepted boxes are available, the method minimizes the sum of center-coordinate and box-size differences from the previous target. This geometric association preserves continuity heuristically; it is not an identity verifier. Thresholds are selected on the same adapted VOT benchmark used for the reported tracking results, rather than a separately described held-out tuning set.

Equation 1
Cbox​(p,i)=∑d∈{x,y}‖cd,p−cd,i‖+‖sd,p−sd,i‖C_{\text{box}}(p,i)=\sum_{d\in\{x,y\}}\|c_{d,p}-c_{d,i}\|+\|s_{d,p}-s_{d,i}\| (1)

IV DATASETS

The authors extend two existing datasets with semantic annotations appropriate for their pipeline. Detection and tracking are evaluated on different data.

IV-A Search and Rescue Image Dataset for Person Detection (SARD)

SARD contains 6,532 actor instances in 1,981 aerial images. Shirt colors are first proposed by GPT-4o on crops and then manually checked. Potential help/injury labels begin from selected pose categories, followed by manual classification of 3,255 candidate instances. These labels describe visible poses interpreted as pain, injury, or unconsciousness; they do not establish actual clinical injury or whether assistance is medically necessary. This annotation rule also shapes the meaning of the reported “injury” detection score.

IV-B Referring VOT22

The original VOT22 benchmark has 62 sequences. The authors add referring expressions and remove sequences where the target cannot be uniquely described, such as one specific ant among a colony. This produces Referring VOT22, but the text does not state the final retained sequence count. Removing ambiguous targets narrows the evaluated referring task.

V EXPERIMENTAL EVALUATION

The first evaluation isolates semantic detection on SARD. The second measures the complete detection-plus-tracking pipeline on Referring VOT22, testing component choices and recovery thresholds.

V-A Hardware Set-up

The server uses an RTX A6000 workstation. Front ends run on a 2 GB Raspberry Pi 4 or 16 GB Jetson Orin NX. A wired Gigabit connection is throttled to 5 Mbps to approximate the assumed field connection. This controls bandwidth but does not by itself reproduce variable cellular latency, outages, or a multi-drone network.

V-B OV Detection Evaluation

Eight detection objectives cover any person, three shirt colors, three poses, and possible need for help. Grounding DINO generates candidates; four VLM configurations verify them. Reported average precision is averaged across these eight tasks. Per-object time reflects one verification, while per-frame time also depends on how many candidate people require checking. The strongest detection configuration and the later full-pipeline configuration use different VLMs.

Table I: OV object detection evaluation. For each task and VLM, we report the Average Precision in the top part of the table. We then calculate the mean among all experiments to obtain the mean Average Precision (mAP) metric for each VLM configuration. We further benchmark the per-frame tft_{f} and per-object to​b​jt_{obj} processing times.
TaskVLM GPT4-mini LLaVA13b LLaVA7b PaliGemma
person 75.23% 82.95% 74.96% 82.88%
shirt_gray 51.61% 44.96% 38.38% 40.35%
shirt_green 53.13% 40.62% 37.53% 3.7%
shirt_blue 54.76% 34.04% 35.72% 7.85%
pose_laying 42.77% 28.29% 40.66% 13.77%
pose_standing 49.4% 45.9% 39.9% 39.97%
pose_sitting 30.08% 21.18% 25.57% 9.22%
injury 38.1% 40.38% 37.4% 4.92%
mAP 49.39% 42.29% 41.26% 25.33%
tft_{\text{f}} 6.955s 4.915s 3.856s 0.866s
tobjt_{\text{obj}} 2.239s 1.478s 1.16s 0.26s

VLM Benchmarking

The paper’s “GPT4-mini” configuration achieves 49.39% mean AP, versus 42.29% for LLaVA13b, 41.26% for LLaVA7b, and 25.33% for PaliGemma. Average detection time per frame is 6.955, 4.915, 3.856, and 0.866 seconds respectively. PaliGemma handles generic-person detection relatively well but loses accuracy on finer attributes. GPT4-mini has higher injury-label recall than LLaVA13b, 77.47% versus 73.28%, but lower injury AP because it also recommends assistance for more cases outside the annotation rule. This illustrates differing decision criteria, not a demonstrated clinical assessment capability. The authors favor the API-based configuration when their semantic criterion and internet access matter, and LLaVA13b for local server operation.

Baseline Comparison

CloudTrack’s 49.39% mean AP exceeds Grounding DINO’s 30.19% and the tested GLEE variants’ 28.27% and 25.78%. This benefit comes with much higher latency: Grounding DINO alone averages 0.284 seconds per frame. CloudTrack does not lead every individual task—GLEE plus is slightly higher for green shirts and GLEE pro is slightly higher for the injury label. These experiments compare the specific eight-description aerial benchmark, rather than all open-vocabulary detection or tracking settings.

Table II: OV object detection baseline comparison. Here, CloudTrack  is our method with GPT4-mini as VLM. We calculate the Average Precision score (AP) for each experiment. We then calculate the mean among all experiments to obtain the mean Average Precision (mAP) metric for each baseline. CloudTrack performs best with an mAP score of 49.39 %49.39\text{\,}\%.
TaskMethod CloudTrack gDINO GLEE plus GLEE pro
person 75.23% 72.6% 34.46% 30.99%
shirt_gray 51.61% 24.79% 29.61% 21.09%
shirt_green 53.13% 31.74% 54.67% 35.63%
shirt_blue 54.76% 16.48% 17.73% 18.05%
pose_laying 42.77% 17.45% 42.4% 37.06%
pose_standing 49.4% 27.33% 15.78% 19.43%
pose_sitting 30.08% 14.11% 4.17% 5.61%
injury 38.1 37.05% 27.3% 38.36%
mAP 49.39% 30.19% 28.27% 25.78%
tft_{\text{f}} 6.955s 0.284s 0.319s 1.156s
tobjt_{\text{obj}} 2.239s 0.103s 0.265s 0.666s

V-C Full-Pipeline Evaluation

The full pipeline uses LLaVA13b, not the highest-scoring GPT4-mini detector. Confidence thresholds from 0.3 to 0.95 are swept in 0.05 increments for score-enabled trackers. Nano at threshold 0.7 gives the best displayed combination: mean IoU 0.378 on both platforms, with overall throughput 35.526 FPS on Jetson and 18.101 FPS on Raspberry Pi. Its local-only rates are higher, 66.067 and 29.487 FPS; those numbers exclude the full recovery/detection overhead measured by the overall rate. The vit tracker reaches similar mean IoU, 0.376, with lower throughput and a smoother threshold tradeoff. Several alternatives fail the ten-FPS overall target, especially on Raspberry Pi. A mean IoU near 0.38 indicates substantial remaining tracking error despite adequate throughput. The two Figure 3 plots retain the complete speed/accuracy tradeoffs.

0010102020303040405050FPS0.320.320.330.330.340.340.350.350.360.360.370.370.380.38mIoUnanovitdaSiamRpntc=0.3t_{c}=0.3tc=0.95t_{c}=0.95tc=0.3t_{c}=0.3tc=0.95t_{c}=0.95tc=0.3t_{c}=0.3tc=0.95t_{c}=0.95
0010102020FPS0.320.320.330.330.340.340.350.350.360.360.370.370.380.38mIoUnanovitdaSiamRpntc=0.3t_{c}=0.3tc=0.95t_{c}=0.95tc=0.3t_{c}=0.3tc=0.95t_{c}=0.95tc=0.3t_{c}=0.3tc=0.95t_{c}=0.95
Fig. 3: Sensitivity study for performance in terms of mean IoU (mIoU) and FPS under variation of the confidence threshold tct_{c} for re-initialization. We analyze threshold-enabled trackers on a Jetson Orin NX (left) and a Raspberry Pi 4 (right) platform.
Table III: Results with fine-tuned threshold values from the sensitivity studies on the Jetson Orin NX and Raspberry Pi 4 platforms.
Tracker tc,optt_{c,\text{opt}} mIoU ↑\uparrow FPS ↑\uparrow FPSEdge↑\text{FPS}_{\text{Edge}}\uparrow
Jetson Orin NX nano 0.7 0.378 35.526 66.067
vit 0.4 0.376 13.355 37.099
DaSiamRPN 0.9 0.346 10.988 31.263
csrt — 0.229 17.586 26.864
mil — 0.153 2.69 3.019
GOTURN — 0.091 16.875 22.602
Raspberry Pi 4 nano 0.7 0.378 18.101 29.487
vit 0.4 0.376 9.034 20.442
DaSiamRPN 0.9 0.34 1.434 2.122
csrt — 0.229 7.276 9.704
mil — 0.153 6.034 7.202
GOTURN — 0.091 2.722 3.006

VI CONCLUSION AND FUTURE WORK

Separating semantic detection from local tracking makes a description-conditioned pipeline feasible on limited edge hardware. The main tradeoffs are slow remote initialization, network dependence, imperfect semantic labels and tracking, benchmark-tuned recovery, and no demonstrated fleet-level scaling. The authors propose faster semantic verification and multi-object support. The current system is a single-target pipeline evaluated under the described connection and benchmark conditions.

Concept demonstration

Open image ↗