Literature
Display
← All papers

TrackOcc: Camera-Based 4D Panoptic Occupancy Tracking

Zhuoguang Chen · Kenan Li · Xiuyu Yang · Tao Jiang · Yiming Li · Hang Zhao

· ICRA 2025 ↗

Computer Vision9 min text · CondensedOriginal paper ↗
Contents

At a glance

TrackOcc predicts a semantic label for each 3D voxel and maintains object identities across time using only surround-view camera inputs at inference. Persistent track queries interact with image-derived volume features, while new queries detect emerging objects and describe background regions. Centroid supervision improves the positions from which queries sample features. On the authors’ Occ3D-Waymo tracking benchmark, TrackOcc scores 20.0 OccSTQ versus 16.2 for the strongest adapted baseline; the association component rises from 9.0 to 13.5. The evidence concerns one dataset and three tracked object categories, with instance labels constructed from existing 3D boxes.

I INTRODUCTION

Box tracking supplies persistent identities but coarse geometry; semantic occupancy supplies voxel-level geometry and classes but no persistent instance identities. Prior 4D LiDAR methods associate segmented point-cloud volumes over time. TrackOcc combines dense semantic occupancy with identity tracking directly from camera streams. Its contribution is the task formulation, a persistent-query architecture, a localization loss, and adapted metrics/baselines. “4D” here means three-dimensional spatial predictions linked over time, rather than a separate continuous four-dimensional reconstruction.

Original PDF illustration S1.F1
Fig. 1: Comparison of tasks for scene understanding. (a) Outputs bounding box tracks from multi-view image sequences. (b) Predicts semantic labels for the occupancy volume from multi-view images. (c) Applies 3D panoptic segmentation on aggregated 4D LiDAR point volumes, followed by post-processing instance matching. (d) Our proposed task: Predicts temporally consistent panoptic labels of the occupancy from multi-view image sequences.

II RELATED WORK

The work joins occupancy prediction, video panoptic segmentation, query-based 3D tracking, and temporal LiDAR segmentation. These research areas provide complementary representations and association mechanisms.

II-A 3D Occupancy Prediction

Occ3D [10] supplies dense occupancy labels; SparseOcc and PaSCo extend voxel representations to panoptic segmentation. These distinguish semantic regions and individual objects in a frame but do not by themselves preserve identities across long sequences. Occupancy can represent geometry beyond standard boxes, although the present experiment tracks only vehicles, pedestrians, and cyclists.

II-B Video Panoptic Segmentation

Video panoptic segmentation unifies semantic and instance labels over time. Older pipelines combine separate networks and post-processing, while MinVIS [15] and CTVIS [17] use query associations. Their 2D masks and video-clip designs do not directly solve streaming 3D voxel tracking; the paper adapts their association ideas as baselines.

II-C 3D Object Tracking

Traditional 3D tracking links detected boxes, while MUTR3D, ADA-Track, and DQTrack [1] learn query-based associations. They motivate persistent representations but still output boxes and focus on countable objects. TrackOcc predicts both tracked object voxels and untracked background categories in one representation.

II-D 4D Panoptic LiDAR Segmentation

4D-PLS [7] and Mask4Former [8] segment aggregated LiDAR scans, then associate volumes. TrackOcc instead propagates learned queries online. The comparison in this paper uses camera-derived occupancy with an adapted association rule, not a direct sensor-matched test against the original LiDAR pipelines.

Original PDF illustration S2.F2
Fig. 2: Overall pipeline of TrackOcc. At each timestep, the encoder extracts multi-view image features and transforms them into 3D volume features. These volume features serve as critical context for updating the 4D panoptic queries via a designed decoder. The query propagation module facilitates efficient object tracking in a streaming, end-to-end manner. The symbol ⨂\bigotimes denotes matrix multiplication.

III METHOD

The model encodes each camera timestep as a 3D feature volume, updates a joint set of emerging and persistent queries, predicts class-specific voxel masks, and propagates selected queries to the next timestep.

III-A Problem Definition

Every voxel receives a pair consisting of a semantic class and an instance ID. Countable “thing” classes require persistent IDs; “stuff” regions do not. Free space is treated as a special stuff class. The output grid has fixed spatial dimensions, and objects should retain their IDs throughout the sequence. This is a closed set of semantic classes; the general geometric representation does not establish open-vocabulary instance tracking.

III-B Overall Architecture

Figure 2 preserves the complete encoder, decoder, query propagation, and output-mask dataflow. An encoder creates volume features from multi-view images, the decoder lets queries interact with those features, and matrix multiplication produces voxel masks. Tracking is integrated through query persistence rather than an external box-association step.

III-C Encoder

A multiscale image backbone and a 2D-to-3D transformation form a volume with D features per voxel. The implementation uses COTR’s encoder [3]. LSS, BEVDepth, BEVFormer, and FB-Occ are discussed as potential compatible alternatives, but the paper does not evaluate interchangeable encoders.

III-D 4D Panoptic Query

A fixed learned set of emerging queries handles both stuff and previously unseen objects. Persistent track queries handle already recognized instances, so their number changes over time. Only emerging queries exist at the first frame. Subsequent frames concatenate the persistent and emerging sets, decode them together, and propagate selected updated queries.

During training, emerging queries matched to ground-truth things become track queries; stuff queries never persist. A track is removed after three consecutive frames without a match. During inference, a thing prediction enters tracking above classification confidence 0.3, and a track exits after three consecutive frames below 0.25. These thresholds provide hysteresis and short-term persistence, not a learned guarantee of reidentification after arbitrary occlusion. The source does not separately evaluate long disappearances or explain all ego-motion/reference-coordinate update details.

III-E Decoder

Self-attention exchanges information among queries. Deformable volume cross-attention then samples a small neighborhood around each query’s reference point instead of attending to the full voxel grid. Emerging queries predict their reference point with an MLP; track queries maintain corresponding points over time. Equation 1 retains the weighted sum of sampled volume features with learned offsets and weights. The implementation uses four sample points per reference.

A linear classifier with sigmoid produces class scores, including a no-object category. Query–volume multiplication followed by sigmoid yields voxel-mask probabilities. Combining classes and masks creates the panoptic occupancy output; predictions from the same persistent query retain the same instance ID. This mechanism ties identity to query continuity, while segmentation and localization quality still constrain tracking accuracy.

Equation 1
VCA​(q,F)=∑j=1Jwj⋅F⁡(p+Δ​pj),\text{VCA}(q,F)=\sum_{j=1}^{J}w_{j}\cdot F(p+\Delta p_{j}), (1)

III-F Loss function

Once a query becomes a track, its target identity stays fixed across subsequent training frames. Emerging predictions are assigned to ground truth through Hungarian bipartite matching following Mask2Former [32]. The source’s statement that emerging queries do not need explicit per-timestep assignment sits alongside this matching description; it does not fully specify the assignment schedule. The mask/class objective combines a binary-mask loss and multiclass classification loss.

Localization-aware supervision applies an L1 loss between predicted reference positions and object centroids, only for thing-associated queries. Equations 2–3 preserve this term and its addition to the mask/class loss. Position supervision is intended to make deformable attention sample the relevant object, rather than relying on mask classification alone to discover useful reference points. The ablation measures the combined effect on tracking and segmentation; it does not independently measure reference-point localization error.

Equations 2–3
ℒl​o​c=L1​(P′,P),\mathcal{L}_{loc}=\text{L1}(P^{\prime},P), (2)
ℒ=ℒm​a​s​k−c​l​s+ℒl​o​c.\mathcal{L}=\mathcal{L}_{mask-cls}+\mathcal{L}_{loc}. (3)
TABLE I: Performance Comparison Between SparseOcc and COTR on 3D Panoptic Prediction. Under the revision metric PQ∗, COTR outperforms SparseOcc in both small objects and stuff regions. C.C represents construction cone.

Methods Metrics Overall vehicle pole C.C. road
SparseOcc [6] PQ 5.0 12.4 0.1 2.6 13.6
 PQ∗ 12.8 21.6 7.0 13.8 34.4
COTR [3] PQ 13.6 13.7 0.5 0.6 76.9
 PQ∗ 22.4 24.3 15.0 12.0 78.2

TABLE II: Camera-based 4D Panoptic Occupancy Tracking Performance on the Occ3D-Waymo Dataset [10]. MinVIS and CTVIS are query-based association methods, while AB3DMOT refers to the tracking-by-detection method. 4D-LCA denotes 4D LiDAR cross-volume association method in [7]. E2E represents end-to-end. Due to space limitations, only representative categories are selected. GO denotes General Objects.

Method E2E OccSTQ OccAQ OccSQ
Overall Vehicle Pedestrian Cyclist Overall Sign Building Vegetation GO
MinVIS [15] ✓ 9.8 3.3 4.0 1.5 3.1 29.1 10.6 39.1 34.0 11.7
CTVIS [17] ✓ 10.7 4.3 5.3 2.1 1.6 26.5 13.2 36.9 30.9 8.1
AB3DMOT [34] ✗ 12.4 5.3 6.8 1.7 2.9 29.1 10.7 39.2 34.0 11.7
4D-LCA [7] ✗ 16.2 9.0 11.4 3.4 4.1 29.1 10.7 39.2 34.0 11.7
TrackOcc (ours) ✓ 20.0 13.5 18.1 3.1 4.6 29.4 13.6 43.2 40.0 9.2

TABLE III: Ablation Studies. Comparison of results with different settings including Localization-aware Loss and the number of training frames.

w/ Loc. Loss #frames OccSTQ OccSQ OccAQ
✗ 3 16.5 28.9 9.4
✓ 3 20.0 29.4 13.5
✓ 5 20.6 29.5 14.4

IV EXPERIMENTS

All experiments use a newly constructed temporal panoptic version of Occ3D-Waymo, with camera inputs and existing semantic occupancy plus box-track annotations for supervision. The paper evaluates segmentation and association separately as well as jointly.

IV-A Dataset

The dataset contains 798 training scenes and 202 validation scenes sampled at 2 Hz. A 0.4 m grid spans −40 to 40 m in x/y and −1 to 5.4 m in z, giving 200 × 200 × 16 voxels. Existing semantic labels are augmented with Waymo’s tracked vehicle, pedestrian, and cyclist boxes [35]. Thing voxels inside a box receive its instance ID; thing voxels outside all boxes are assigned to the nearest box. Every other category is stuff.

These are box-derived instance annotations, not independent dense instance-track labels. The nearest-box rule can introduce assignment ambiguity, and no separate annotation-quality audit is reported. Camera-only describes model input; training still depends on the provided occupancy, box, and tracking supervision. General objects can receive semantic occupancy labels but are not a fourth tracked thing class.

IV-B Implement Details

ResNet-50 processes 256 × 704 images. Self-attention and volume cross-attention each have one layer; queries have 256 dimensions, and each frame receives 200 emerging queries. VCA uses four samples. The entrance/exit thresholds are 0.3/0.25 with a three-frame retirement window. Training runs 24 epochs with AdamW, learning rate 2 × 10⁻⁴, batch size 1, and eight RTX 3090 GPUs. Main results use three-frame training; a five-frame variant is evaluated separately.

IV-C Evaluation Metrics

OccSTQ is the geometric mean of occupancy segmentation quality (OccSQ) and association quality (OccAQ), adapting STQ [39] to space-time voxels. OccSQ is semantic mean IoU. OccAQ measures how consistently predicted instance tracks overlap ground-truth tracks across the sequence: Equation 4 averages overlap-weighted track IoU over ground-truth instances, and Equation 5 combines the two components. Association matching may cross thing classes, because semantic misclassification is penalized in OccSQ rather than again in OccAQ.

The reported metrics cover only the visible region specified by Occ3D. Extending the same metric beyond visible voxels is proposed as an option, not an evaluated scene-completion result. The separation matters: a large association improvement can coexist with almost unchanged semantic accuracy.

Equations 4–5
OccAQ=1|G|​∑g∈G1|g​ti​d​(g)|​∑p∈Pp∩g≠0TPA​(p,g)⋅IoU​(p,g).\textit{OccAQ}\!=\!\frac{1}{|G|}\!\!\sum_{g\in G}\!\frac{1}{|gt_{id}(g)|}\!\!\!\!\sum_{\begin{subarray}{c}p\in P\\ p\cap g\neq 0\end{subarray}}\!\!\!\textit{TPA}(p,g)\!\cdot\!\text{IoU}(p,g). (4)
O​c​c​S​T​Q=O​c​c​S​Q×O​c​c​A​Q.OccSTQ=\sqrt{OccSQ\times OccAQ}. (5)
Original PDF illustration S4.F3
Fig. 3: Qualitative results of our method and 4D-LCA. The same color indicates identical instances across different time steps. Our method effectively tracks both moving and stationary vehicles, while the 4D-LCA method encounters difficulties with moving vehicles. For clarity, only images from the front camera are displayed, although both methods process data from surrounding cameras.

IV-D Baselines

COTR is selected as the shared single-frame occupancy foundation. Table I compares it with SparseOcc using standard panoptic quality and a modified PQ* that replaces the hard IoU > 0.5 pairing rule with maximum-weight bipartite matching. COTR’s overall PQ* is 22.4 versus 12.8 and road score 78.2 versus 34.4. Its construction-cone PQ* is lower, 12.0 versus 13.8, so the caption’s small-object advantage is not universal; PQ* also should not be conflated with standard PQ.

Four association baselines extend COTR: MinVIS matches queries between frames; CTVIS adds contrastive training for query association; AB3DMOT [34] tracks boxes fitted around predicted instance voxels; and 4D-LCA adapts the cross-volume association procedure from 4D-PLS. These are the authors’ adaptations to the new task. Table II retains the end-to-end flags and all reported representative class scores.

IV-E Main results

TrackOcc reaches 20.0 OccSTQ versus 16.2 for 4D-LCA, a 3.8-point gain. OccAQ rises from 9.0 to 13.5, a 50% relative increase, while OccSQ changes only from 29.1 to 29.4. Vehicle association improves from 11.4 to 18.1 and cyclist association from 4.1 to 4.6, but pedestrian association drops from 3.4 to 3.1. General-object semantic quality also decreases from 11.7 to 9.2. Thus the aggregate tracking gain does not mean every class improves.

MinVIS and CTVIS obtain 9.8 and 10.7 OccSTQ; AB3DMOT reaches 12.4. The stronger spatial-association baselines and the localization ablation support the usefulness of position information. The reported speed is 7.5 FPS on one RTX 3090 for a stream of about 40 images. The paper gives no full latency distribution, sustained memory profile, or longer-stream scaling measurement, and this single setup does not establish a general deployment throughput.

IV-F Ablation Studies

With three training frames, adding centroid localization loss raises OccSTQ from 16.5 to 20.0 and OccAQ from 9.4 to 13.5—3.5 points and about 43.6% relative association improvement—while OccSQ rises from 28.9 to 29.4. Keeping the loss and training on five frames yields 20.6/29.5/14.4 for OccSTQ/OccSQ/OccAQ. This is a modest gain from one longer training window, not evidence of unlimited temporal scalability. No repeated-run uncertainty is reported.

IV-G Qualitative Comparison

Figure 3 retains all three timesteps and all four columns: front camera, ground truth, TrackOcc, and 4D-LCA. Colors indicate persistent instance IDs; annotations highlight moving and stationary vehicles. The shown sequence supports the qualitative claim of fewer identity inconsistencies for TrackOcc, while both models use surrounding cameras despite only the front images being displayed. It is one selected illustration rather than a measured breakdown by motion or occlusion.

V CONCLUSIONS

Persistent voxel-mask queries and explicit centroid supervision improve aggregate camera-based occupancy tracking on the constructed Waymo benchmark, primarily through association quality. Limits include coarse voxel resolution, box-derived instance labels, only three tracked thing categories, visible-region evaluation, modest absolute association scores, and missing uncertainty and long-stream stress tests. Integration with planning, transfer to other datasets, unobserved-space completion, and broad open-world tracking are future directions rather than demonstrated outcomes.

ACKNOWLEDGMENT

The authors acknowledge the National Key R&D Program of China and Tsinghua University’s Initiative Scientific Research Program.

Concept demonstration

Open image ↗