At a glance
CloudTrack combines a remote open-vocabulary detector and vision-language verifier with a lightweight onboard single-object tracker. Text descriptions guide initial detection; the tracker then follows an accepted candidate without rerunning the large models every frame. Experiments measure semantic detection on search-and-rescue imagery and tracking on adapted video sequences. The approach reduces onboard computation while depending on a network and incurring multi-second detection latency.
I INTRODUCTION
A missing-person description can specify attributes beyond a detector’s fixed class vocabulary. Large vision-language models can interpret such descriptions but are expensive and slow on small drone computers. CloudTrack separates semantic initialization from fast local tracking and releases software plus added descriptions/labels for two datasets. It requires no mission-specific weight training, but prompts and reinitialization thresholds are empirically tuned.
II RELATED WORK
II-A Detection and Tracking for UAVs
Earlier aerial systems use compact detectors, motion models, remote verification, or human feedback. CloudTrack retains a remote verification stage but uses a vision-language model instead of a human to accept initial candidates. Its benchmark evaluates semantic detection and tracking, rather than demonstrating a complete autonomous rescue mission.
II-B Model-free Object Tracking
The evaluated trackers follow a supplied bounding box and do not independently identify the requested person. They include online discriminative models, correlation-filter approaches, and compact learned trackers. Their role is short-term continuity after the semantic system chooses a candidate.
II-C Open-Vocabulary Detection and Tracking
Open-vocabulary systems reuse broad image/text pretraining to accept descriptions beyond fixed class labels. CloudTrack combines Grounding DINO [21] with a separate VLM verification stage instead of training a new end-to-end referring tracker. The chosen models’ capabilities and names refer to this paper’s evaluation configurations.
III METHODS
III-A Overview
A user supplies a description, the server identifies a matching candidate, and an onboard tracker follows it while awaiting human review. The architecture could serve multiple drones from one server, but the paper does not report a multi-drone scaling experiment. A working connection between drone and server is an explicit assumption.
III-B Object-Detection Back-End
Grounding DINO first receives a broad category such as person, because detailed prompts can return people without correctly checking attributes. Each proposed box is cropped with a 50-pixel margin and passed to a VLM with the full description. Accepted boxes initialize tracking. This addresses small-object limitations of whole-image VLM queries, while making runtime grow with the number of candidate boxes. Prompts are tuned separately for each model. The reported rescue prompt explicitly favors saying yes when uncertain, which affects the recall/false-positive tradeoff.
Original system prompt: You are an intelligent AI assistant that helps a drone in a search and rescue mission. If in doubt, rather say yes.
Original user prompt: The drone needs to find an injured person. Would you recommend sending help to the person in the image? Start your justification with ‘Lets analyze the image’.
III-C Object Tracking Front-End
Streaming every frame through the remote semantic detector would make tracking sensitive to network delay and throughput. CloudTrack instead targets at least ten frames per second locally on a Raspberry Pi 4 or Jetson Orin NX. Six lightweight trackers are benchmarked, and the large models are used primarily for initial detection or recovery.
III-D Re-Initialization
When a tracker supplies a confidence score, a threshold triggers remote reinitialization. Trackers without such a score receive no reinitialization in these experiments. If several accepted boxes are available, the method minimizes the sum of center-coordinate and box-size differences from the previous target. This geometric association preserves continuity heuristically; it is not an identity verifier. Thresholds are selected on the same adapted VOT benchmark used for the reported tracking results, rather than a separately described held-out tuning set.
Equation 1
| (1) |
IV DATASETS
The authors extend two existing datasets with semantic annotations appropriate for their pipeline. Detection and tracking are evaluated on different data.
IV-A Search and Rescue Image Dataset for Person Detection (SARD)
SARD contains 6,532 actor instances in 1,981 aerial images. Shirt colors are first proposed by GPT-4o on crops and then manually checked. Potential help/injury labels begin from selected pose categories, followed by manual classification of 3,255 candidate instances. These labels describe visible poses interpreted as pain, injury, or unconsciousness; they do not establish actual clinical injury or whether assistance is medically necessary. This annotation rule also shapes the meaning of the reported “injury” detection score.
IV-B Referring VOT22
The original VOT22 benchmark has 62 sequences. The authors add referring expressions and remove sequences where the target cannot be uniquely described, such as one specific ant among a colony. This produces Referring VOT22, but the text does not state the final retained sequence count. Removing ambiguous targets narrows the evaluated referring task.
V EXPERIMENTAL EVALUATION
The first evaluation isolates semantic detection on SARD. The second measures the complete detection-plus-tracking pipeline on Referring VOT22, testing component choices and recovery thresholds.
V-A Hardware Set-up
The server uses an RTX A6000 workstation. Front ends run on a 2 GB Raspberry Pi 4 or 16 GB Jetson Orin NX. A wired Gigabit connection is throttled to 5 Mbps to approximate the assumed field connection. This controls bandwidth but does not by itself reproduce variable cellular latency, outages, or a multi-drone network.
V-B OV Detection Evaluation
Eight detection objectives cover any person, three shirt colors, three poses, and possible need for help. Grounding DINO generates candidates; four VLM configurations verify them. Reported average precision is averaged across these eight tasks. Per-object time reflects one verification, while per-frame time also depends on how many candidate people require checking. The strongest detection configuration and the later full-pipeline configuration use different VLMs.
| GPT4-mini | LLaVA13b | LLaVA7b | PaliGemma | |
| person | 75.23% | 82.95% | 74.96% | 82.88% |
| shirt_gray | 51.61% | 44.96% | 38.38% | 40.35% |
| shirt_green | 53.13% | 40.62% | 37.53% | 3.7% |
| shirt_blue | 54.76% | 34.04% | 35.72% | 7.85% |
| pose_laying | 42.77% | 28.29% | 40.66% | 13.77% |
| pose_standing | 49.4% | 45.9% | 39.9% | 39.97% |
| pose_sitting | 30.08% | 21.18% | 25.57% | 9.22% |
| injury | 38.1% | 40.38% | 37.4% | 4.92% |
| mAP | 49.39% | 42.29% | 41.26% | 25.33% |
| 6.955s | 4.915s | 3.856s | 0.866s | |
| 2.239s | 1.478s | 1.16s | 0.26s |
VLM Benchmarking
The paper’s “GPT4-mini” configuration achieves 49.39% mean AP, versus 42.29% for LLaVA13b, 41.26% for LLaVA7b, and 25.33% for PaliGemma. Average detection time per frame is 6.955, 4.915, 3.856, and 0.866 seconds respectively. PaliGemma handles generic-person detection relatively well but loses accuracy on finer attributes. GPT4-mini has higher injury-label recall than LLaVA13b, 77.47% versus 73.28%, but lower injury AP because it also recommends assistance for more cases outside the annotation rule. This illustrates differing decision criteria, not a demonstrated clinical assessment capability. The authors favor the API-based configuration when their semantic criterion and internet access matter, and LLaVA13b for local server operation.
Baseline Comparison
CloudTrack’s 49.39% mean AP exceeds Grounding DINO’s 30.19% and the tested GLEE variants’ 28.27% and 25.78%. This benefit comes with much higher latency: Grounding DINO alone averages 0.284 seconds per frame. CloudTrack does not lead every individual task—GLEE plus is slightly higher for green shirts and GLEE pro is slightly higher for the injury label. These experiments compare the specific eight-description aerial benchmark, rather than all open-vocabulary detection or tracking settings.
| CloudTrack | gDINO | GLEE plus | GLEE pro | |
| person | 75.23% | 72.6% | 34.46% | 30.99% |
| shirt_gray | 51.61% | 24.79% | 29.61% | 21.09% |
| shirt_green | 53.13% | 31.74% | 54.67% | 35.63% |
| shirt_blue | 54.76% | 16.48% | 17.73% | 18.05% |
| pose_laying | 42.77% | 17.45% | 42.4% | 37.06% |
| pose_standing | 49.4% | 27.33% | 15.78% | 19.43% |
| pose_sitting | 30.08% | 14.11% | 4.17% | 5.61% |
| injury | 38.1 | 37.05% | 27.3% | 38.36% |
| mAP | 49.39% | 30.19% | 28.27% | 25.78% |
| 6.955s | 0.284s | 0.319s | 1.156s | |
| 2.239s | 0.103s | 0.265s | 0.666s |
V-C Full-Pipeline Evaluation
The full pipeline uses LLaVA13b, not the highest-scoring GPT4-mini detector. Confidence thresholds from 0.3 to 0.95 are swept in 0.05 increments for score-enabled trackers. Nano at threshold 0.7 gives the best displayed combination: mean IoU 0.378 on both platforms, with overall throughput 35.526 FPS on Jetson and 18.101 FPS on Raspberry Pi. Its local-only rates are higher, 66.067 and 29.487 FPS; those numbers exclude the full recovery/detection overhead measured by the overall rate. The vit tracker reaches similar mean IoU, 0.376, with lower throughput and a smoother threshold tradeoff. Several alternatives fail the ten-FPS overall target, especially on Raspberry Pi. A mean IoU near 0.38 indicates substantial remaining tracking error despite adequate throughput. The two Figure 3 plots retain the complete speed/accuracy tradeoffs.
| Tracker | mIoU | FPS | |||
|---|---|---|---|---|---|
| Jetson Orin NX | nano | 0.7 | 0.378 | 35.526 | 66.067 |
| vit | 0.4 | 0.376 | 13.355 | 37.099 | |
| DaSiamRPN | 0.9 | 0.346 | 10.988 | 31.263 | |
| csrt | — | 0.229 | 17.586 | 26.864 | |
| mil | — | 0.153 | 2.69 | 3.019 | |
| GOTURN | — | 0.091 | 16.875 | 22.602 | |
| Raspberry Pi 4 | nano | 0.7 | 0.378 | 18.101 | 29.487 |
| vit | 0.4 | 0.376 | 9.034 | 20.442 | |
| DaSiamRPN | 0.9 | 0.34 | 1.434 | 2.122 | |
| csrt | — | 0.229 | 7.276 | 9.704 | |
| mil | — | 0.153 | 6.034 | 7.202 | |
| GOTURN | — | 0.091 | 2.722 | 3.006 |
VI CONCLUSION AND FUTURE WORK
Separating semantic detection from local tracking makes a description-conditioned pipeline feasible on limited edge hardware. The main tradeoffs are slow remote initialization, network dependence, imperfect semantic labels and tracking, benchmark-tuned recovery, and no demonstrated fleet-level scaling. The authors propose faster semantic verification and multi-object support. The current system is a single-target pipeline evaluated under the described connection and benchmark conditions.