1 Introduction↩︎

Holistic 4D scene understanding from cameras remains a key challenge for autonomous systems. Camera-based 4D panoptic occupancy tracking (4D-POT) has recently emerged as a unified formulation that jointly models geometry, semantics, and object identities over time. By jointly reasoning over them, recent approaches enable rich spatiotemporal awareness from multi-view imagery alone, bringing camera-only perception closer to persistent 3D scene understanding [1]–[3]. Such unified 4D scene representations are particularly appealing for navigation and motion planning in dynamic environments, where both static structure and moving objects must be modeled reliably. However, maintaining temporally consistent volumetric representations in these settings remains challenging, especially under partial observations and occlusions.

Most recent 4D-POT approaches [4], [5] adopt mask-based pipelines that combine a dense 3D feature volume with a set of decoder queries for semantic and instance prediction. In these formulations, the volumetric feature grid primarily encodes geometric scene structure, while decoder queries produce embeddings that are matched against the feature volume to predict voxel masks. For thing classes, instance queries are propagated across frames to preserve object identity, whereas stuff semantics are predicted from per-frame queries without temporal persistence. More importantly, the underlying 3D feature volume, which carries the bulk of geometric information, is recomputed at each timestep. As a result, temporal information is not explicitly preserved in the volumetric representation, and geometric consistency across frames is not explicitly enforced. This limitation becomes particularly pronounced under occlusion and for static scene elements that are not directly associated with tracked object queries.

To address these limitations, we propose a streaming Gaussian encoder that maintains a persistent and temporally consistent volumetric scene representation (illustrated in 1), complementing decoder-level tracking with representation-level temporal coherence. Concretely, our encoder maintains a fixed-size set of latent Gaussian queries, which are propagated across frames via ego-motion compensation and iteratively updated using the current multi-view evidence. To keep the latent state both compact and expressive, we employ confidence-guided state management. Gaussian opacities are shaped to approximate per-frame visibility, and confidence accumulates this visibility over time, thereby retaining consistently supported structures while removing obsolete ones. Discarded queries are replaced by newly initialized queries sampled from strong feature responses in the lifted 3D volume. The resulting queries are jointly refined by a point-based transformer and decoded into Gaussian primitives that are splatted into the 3D feature grid. This streaming formulation preserves geometric consistency across frames while remaining fully compatible with existing mask-based 4D-POT pipelines.

In summary, our contributions are fourfold:

• Persistent Gaussian state for 4D-POT. We introduce a streaming Gaussian encoder that maintains a fixed-budget, queryable latent scene representation across timesteps, enabling representation-level temporal coherence beyond decoder-only tracking.

• Confidence- and visibility-guided state management. We interpret Gaussian opacity as a proxy for per-frame visibility and accumulate it as evidence over time to guide pruning and refreshment of the latent state.

• Streaming-consistent supervision. We align opacity with geometric visibility via depth-based regularization and adopt a warmup-based multi-frame training strategy to stabilize extended temporal aggregation.

• State-of-the-art performance with negligible overhead. Our approach achieves state-of-the-art results on Occ3D nuScenes and Waymo while maintaining virtually identical inference throughput to prior mask-based pipelines.

2 Related Work↩︎

We review prior work on camera-based occupancy prediction, temporal scene modeling, and Gaussian-based 3D representations for 4D-POT.

Camera-based 4D Occupancy Prediction: Camera-based 3D occupancy prediction has emerged as a powerful paradigm for holistic scene understanding. With the availability of large-scale benchmarks such as nuScenes [6] and Waymo [7], together with dense 3D occupancy annotations from Occ3D [8], recent work has made substantial progress in reconstructing semantic scene geometry from multi-view imagery. Methods including TPVFormer [9], SurroundOcc [10], OccFormer [11], and SparseOcc [12] demonstrate strong performance on camera-only semantic occupancy prediction. More recent approaches extend this formulation toward panoptic and temporal reasoning, culminating in camera-based 4D panoptic occupancy tracking (4D-POT) as formalized by TrackOcc [4].

Mask-based Panoptic Occupancy: Most recent 4D-POT approaches adopt mask-based architectures that combine dense volumetric features with query-based decoders, a paradigm first popularized in 2D segmentation by MaskFormer [13] and Mask2Former [14]. OccFormer [11] and COTR [15] bring this query-driven formulation to 3D perception, which is further developed into panoptic occupancy approaches, including SparseOcc [12] and PaSCo [16]. Recent 4D-POT systems, such as TrackOcc [4] and LaGS [5] build on this design, in which decoder queries produce mask embeddings that are matched against a dense 3D feature volume. While propagating instance queries across frames preserves object identities, the underlying volumetric features are typically recomputed at each timestep. Some methods incorporate short-term temporal fusion by aligning and aggregating features from adjacent frames [15], [17], [18]. However, the resulting 3D representation remains largely frame-centric rather than persistently maintained. Consequently, temporal consistency is largely handled at the decoder level, and geometric coherence of the volumetric representation is not explicitly enforced. This limitation becomes particularly pronounced under occlusion and for static scene elements that are not directly associated with tracked object queries.

Temporal Scene Modeling: A large body of work explores temporal fusion for camera-based 3D perception. BEV-based methods such as BEVFormer [18], BEVDet4D [17], and SOLOFusion [19] aggregate multi-frame image features to improve bird’s-eye-view representations. Subsequent approaches including StreamPETR [20], Sparse4D [21], and UniOcc [22] investigate streaming-friendly and memory-enhanced multi-frame perception. In parallel, occupancy-focused methods such as OccFlowNet [23] and ForecastOcc [24] incorporate additional temporal cues for dynamic scene reasoning. Despite these advances, most approaches rely on short-term feature aggregation or recurrent feature-volume updates, yielding frame-centric representations that are recomputed at each timestep. In contrast, our approach maintains a persistent volumetric scene state via latent Gaussian queries.

Gaussian Scene Representations: Gaussian-based representations have recently emerged as an efficient alternative for 3D scene modeling. 3D Gaussian Splatting [25] demonstrates high-fidelity real-time rendering using explicit Gaussian primitives, but focuses primarily on offline scene reconstruction, with extensions for motion-aware reconstruction [26]. Subsequent works such as GaussianFormer [27], GaussianFormer-2 [28], GaussianWorld [29], and Chorus [30] adapt Gaussian representations for online scene understanding and occupancy prediction. Most closely related to our work, LaGS [5] introduces latent Gaussian queries within a per-frame encoder and employs mask-based decoding for 4D-POT. However, temporal consistency is primarily handled at the decoder level, leaving the encoded scene state itself temporally independent across frames. In contrast, our approach maintains temporal coherence directly within the volumetric representation, enabling representation-level temporal consistency.

Overall, existing methods predominantly rely on frame-centric volumetric representations, with temporal reasoning largely confined to the decoder, where geometric scene structure is only indirectly represented. Persistent scene representations for streaming 4D-POT remain largely unexplored.

3 Method↩︎

We address camera-based 4D panoptic occupancy tracking with a temporal Gaussian encoder that maintains a persistent, queryable 3D scene representation over time. 3.1 formalizes the task, followed by an architectural overview in 3.2. 3.3 then details the proposed encoder, including the latent state representation, feature-aware ego-motion compensation, and confidence-guided state management under a fixed query budget. Finally, 3.4 describes the depth-regularized and streaming-aware training strategy.

3.1 Task Definition↩︎

Camera-based 4D panoptic occupancy tracking [4] requires joint prediction of 3D occupancy, semantics, and temporally consistent instance assignments from multi-view image observations. In this work, the task is formulated as a streaming prediction problem. At each timestep \(t\), the input consists of synchronized multi-view RGB images \(\boldsymbol{I}_t = \{I_t^j\}_{j=1}^{N}\), where \(N\) is the number of cameras, together with known camera intrinsics and extrinsics, as well as ego-motion from \(t-1\) to \(t\). Approaches may optionally maintain an aggregated temporal state. Formally, given a set of classes \(\mathcal{C}\) (including free space, tracked thing- and static/background stuff-type categories), the task is to assign a tuple \((c_{\boldsymbol{x}, t}, i_{\boldsymbol{x}, t}) \in \mathcal{C} \times \mathbb{N}\) to each spatial location \(\boldsymbol{x} \in \mathcal{V}\), where \(\mathcal{V} \subset \mathbb{R}^3\) denotes the spatial region of interest. Here, \(c_{\boldsymbol{x},t}\) represents the semantic occupancy label and \(i_{\boldsymbol{x},t}\) the associated instance identity. Notably, instance assignments are defined only on thing classes and must remain temporally consistent across timesteps. For stuff classes, instance identities are not defined. \(\mathcal{V}\) is discretized as a voxel grid of size \(X \times Y \times Z\).

3.2 Overall Architecture↩︎

Our approach builds on an existing 4D panoptic occupancy pipeline [5] and introduces a novel temporal Gaussian encoder (3.3) to learn temporally consistent volumetric representations. Given a set of multi-view input images \(\boldsymbol{I}_t\) at timestep \(t\), image features are first extracted and lifted into 3D using low-resolution depth predictions, yielding an initial feature volume. This volume is refined into a multi-scale 3D feature pyramid, which is subsequently processed by our temporal Gaussian encoder. The encoder aggregates information over time, maintaining a temporally consistent latent state across frames. The resulting representation is converted into a dense 3D feature volume at the target resolution \(X \times Y \times Z\), which serves as input to a mask-based 4D panoptic occupancy decoder [5], to produce semantic occupancy and temporally consistent instance assignments.

3.3 Temporal Gaussian Encoder↩︎

Our temporal Gaussian encoder is designed to learn a temporally consistent, queryable 3D representation by maintaining and updating a set of latent queries over time. These queries act as a persistent latent state (3.3.1) and are decoded into Gaussian primitives that carry localized geometric and semantic information, yielding a continuous volumetric representation. This formulation builds on the temporally independent Gaussian-based LaGS encoder [5], which constructs volumetric representations from multi-view observations via query sampling, refinement, and splatting, and extends it to the temporal domain.

In the single-frame setting, LaGS initializes query features \(\boldsymbol{q}_k\) by sampling from the multi-scale 3D feature pyramid obtained after the lifting step. The queries are serialized via space-filling curves and refined using a point-based transformer, where different scales are processed as separate streams that interact via window-based multi-stream attention. Within each scale, self-attention models interactions between queries, while spatial image cross-attention enables efficient implicit 2D-to-3D lifting. After refinement, each query \(\boldsymbol{q}_k\) is decoded into Gaussian parameters, including center \(\boldsymbol{\mu}_k\), covariance, opacity \(\alpha_k\), and embedding feature \(\boldsymbol{e}_k\). The embedding features \(\boldsymbol{e}_k\) are then splatted into a dense 3D feature volume using the corresponding Gaussian primitives.

Building on this formulation, our temporal encoder introduces a persistent query state and a streaming update mechanism across timesteps. As new observations arrive, the latent queries are iteratively propagated, refreshed, and refined, enabling efficient multi-view aggregation while preserving temporal consistency. The update proceeds in three stages, illustrated in 2. First, queries from the previous timestep are propagated using feature-aware ego-motion compensation to maintain spatial alignment (3.3.2). Second, the latent state is refreshed under a fixed query budget via confidence-guided pruning and magnitude-based re-initialization (3.3.3), allowing obsolete queries to be replaced by newly observed structures. Finally, the resulting set of queries is jointly refined using the same point-based transformer architecture as in the single-frame encoder. The refined queries are then stored for the next timestep, decoded into Gaussian primitives, and splatted into a 3D voxel grid for the downstream 4D-POT decoder.

3.3.1 State Representation↩︎

The encoder maintains a persistent set of \(K\) latent queries \(\{\boldsymbol{q}_k\}_{k=1}^K\) and corresponding reference positions \(\boldsymbol{p}_k \in \mathbb{R}^3\), together forming the temporal state that is propagated and updated over time. Each query can be decoded into a Gaussian primitive that provides an explicit geometric interpretation of the latent state. Specifically, a query \(\boldsymbol{q}_k\) parameterizes a Gaussian with center \(\boldsymbol{\mu}_k\), covariance (represented via scale and rotation), opacity \(\alpha_k\), embedding features \(\boldsymbol{e}_k\), and semantic predictions (class logits) \(\boldsymbol{z}_k\). The Gaussian center is defined relative to the reference position, i.e., \(\boldsymbol{\mu}_k = \boldsymbol{p}_k + \boldsymbol{\delta}_k\) where \(\boldsymbol{\delta}_k\) is decoded from the Gaussian, enabling stable spatial tracking across frames. These Gaussian attributes are used throughout the update procedure: centers define spatial locations for ego-motion compensated propagation, opacities provide a measure of importance for pruning, and embedding features contribute to the final volumetric representation.

3.3.2 Feature-Aware Ego-Motion Compensation↩︎

Each query represents a persistent spatial feature across timesteps. As the camera moves, ego-motion compensation is therefore required during propagation to preserve this correspondence and maintain spatial alignment of the latent state. Given the ego-motion between frames \(t-1\) and \(t\) as a rigid transform \(T = [R \mid \boldsymbol{t}]\), updated query reference positions are obtained by transforming the decoded Gaussian centers: \[\boldsymbol{p}_k^{(t)} = T \cdot \boldsymbol{\mu}_k^{(t-1)}.\] To update position-dependent feature components, we employ a FiLM-based modulation scheme [31]. Global modulation parameters \(\boldsymbol{\gamma}, \boldsymbol{\beta} \in \mathbb{R}^d\) are predicted from ego-motion magnitude \(m = \|\boldsymbol{t}\|_2\) and rotation \(\theta = \arccos\left(\frac{(\operatorname{tr}(R) - 1)}{2}\right)\), providing a compact and sufficient summary of the global motion, as \([\boldsymbol{\gamma}; \boldsymbol{\beta}] = \operatorname{MLP}_{\text{mod}}([m;\theta])\), with \(\boldsymbol{\gamma}\) bias-initialized to \(\boldsymbol{1}\) and \(\boldsymbol{\beta}\) to \(\boldsymbol{0}\). Local motion cues are encoded per query as \[\boldsymbol{m}_k = \operatorname{MLP}_{\text{mot}}\left(\boldsymbol{p}_k^{(t)} - \boldsymbol{\mu}_k^{(t-1)}\right)\] and used to compute feature updates \[\boldsymbol{\delta}_k = \operatorname{MLP}_{\text{upd}}\left(\left[\boldsymbol{q}_k^{(t-1)};\, \boldsymbol{m}_k\right]\right).\] Finally, query features are updated via a gated, FiLM-modulated residual step: \[\begin{align} \boldsymbol{q}_k^{(t)} = \boldsymbol{q}_k^{(t-1)} + \sigma(g)\cdot \left( \boldsymbol{\gamma} \odot \boldsymbol{\delta}_k + \boldsymbol{\beta} \right), \end{align}\] where \(\sigma(\cdot)\) is the sigmoid function and \(g\) is a learned scalar initialized to yield small updates at early stages of training, promoting stable learning dynamics.

3.3.3 Confidence-Guided Pruning↩︎

While ego-motion compensation preserves spatial alignment across frames, the latent state must be refreshed under a fixed query budget as new regions become visible and old ones become obsolete. However, pruning based solely on instantaneous observations is insufficient, as temporarily occluded yet valid structures may be removed prematurely. To address this, we assign each query a confidence score \(\rho_k\) that accumulates visibility evidence over time. Here, Gaussian opacity \(\alpha_k\) serves as a proxy for per-frame visibility, whose semantics are shaped through depth-based supervision (cf. 3.4.1).

Confidence is recursively updated to reflect temporal reliability. We compute a per-frame observation score \(o_k\) from Gaussian opacity \(\alpha_k\), modulated by distance-based falloff \(d_k\) and gated by field-of-view visibility: \[o_k = \alpha_k \cdot d_k \cdot \mathbf{1}_{\text{FoV}}(\boldsymbol{\mu}_k), \quad d_k = \frac{1}{1 + \frac{\|\boldsymbol{\mu}_k\|_2}{\lambda_{\text{dist}}}},\] where \(\lambda_{\text{dist}}\) controls the distance-based decay. The confidence is then updated as \[\rho_k^{(t)} = \rho_k^{(t-1)} + \lambda \, o_k^{(t)} \left(1 - \rho_k^{(t-1)}\right), \quad \rho_k^{(0)} = 0,\] with \(\lambda\) controlling accumulation speed.

Since ego-motion compensation accounts only for camera motion, Gaussians belonging to dynamic classes (e.g., vehicles or pedestrians) may become spatially inconsistent over time. We therefore apply an additional per-frame decay: \[\rho_k^{(t)} = \rho_k^{(t)} \cdot \begin{cases} \lambda_{\text{dyn}}, & \text{if } \hat{c}_k \in \mathcal{C}_{\text{dyn}}, \\ 1, & \text{otherwise}, \end{cases}\] where \(\hat{c}_k = \arg\max \boldsymbol{z}_k\) denotes the predicted class of the \(k\)-th Gaussian and \(\lambda_{\text{dyn}}\) controls dynamic decay strength.

Budget-Constrained State Refreshment: Given the confidence estimates \(\rho_k\), the latent state is refreshed under a fixed budget of \(K\) queries. Queries are first filtered using temporal confidence, instantaneous opacity, and spatial relevance: \[\mathcal{K}' = \left\{\; k \;\middle|\; \left( \rho_k^{(t)} \ge \tau_{\rho} \;\lor\; \alpha_k^{(t)} \ge \tau_\alpha \right) \;\land\; \boldsymbol{p}_k \in \mathcal{V}_p \;\right\}.\] If \(|\mathcal{K}'| > K - M\), we enforce the query budget via hybrid rank-based sampling. Otherwise, all eligible queries are retained. To favor queries supported by either persistent confidence or strong instantaneous evidence, we define the hybrid rank \[r_k = \max\!\left\{ \operatorname{rank}\left(\rho_k\right)\!, \,\operatorname{rank}\left(\alpha_k\right) \right\}.\] The retained set of \(K - M\) queries is then sampled as \[\mathcal{K}_{p} \sim \mathcal{M}(\boldsymbol{\pi},\, K - M), \quad \pi_k = \frac{r_k}{\sum_j r_j},\] where \(\mathcal{M}(\cdot)\) denotes multinomial sampling without replacement, ensuring that at least \(M\) slots remain available for new queries in the current frame. In practice, we find that \(\tau_\alpha{=}0.01\) by itself is a reliable discriminator for visibility. Consequently, combining this with \(\tau_\rho{=}0.01\), \(\lambda{=}0.8\), and \(\lambda_{\text{dist}}{=}20\) retains Gaussians that are either visible in the current frame or have been observed well in the past. Additionally, we set \(\lambda_{\text{dyn}}{=}0.9\).

Initialization of New Queries: To replenish the pruned slots, new queries are initialized at locations with strong responses in the lifted 3D feature volume. Given per-voxel feature vectors \(\boldsymbol{f}_k\), we define sampling priorities based on feature magnitude and sample indices via multinomial sampling: \[\mathcal{K}_{n} \sim \mathcal{M}(\boldsymbol{\pi},\, K - |\mathcal{K}_p|), \quad \pi_k = \frac{\|\boldsymbol{f}_k\|_2}{\sum_j \|\boldsymbol{f}_j\|_2}.\] The final query set is given by \(\mathcal{K} = \mathcal{K}_{p} \cup \mathcal{K}_{n}\), ensuring a fixed budget of \(|\mathcal{K}| = K\) queries at each timestep.

3.4 Training and Supervision↩︎

Our temporal encoder is trained end-to-end as part of the overall model. To supervise the persistent latent state, we combine geometric supervision with a streaming-aware training strategy. Specifically, we employ (i) a depth splatting loss that regularizes the lifted 3D representation and shapes Gaussian opacities, and (ii) a multi-frame training procedure that exposes the encoder to realistic streaming inputs while keeping memory overhead low.

3.4.1 Depth Splatting Supervision↩︎

Since confidence aggregates observation quality over time, confidence-guided pruning requires opacity to reflect per-frame visibility. We therefore align opacity with geometric support by regularizing Gaussian opacities using sparse LiDAR depth. To this end, decoded Gaussians are splatted onto the image plane, encouraging surface-consistent Gaussians to attain high opacity while suppressing those inconsistent with measured geometry or located in occluded regions. Consequently, opacity serves as a proxy for per-frame visibility during confidence updates (cf. the characteristic shadows in 1).

Depth is rendered by alpha compositing the decoded Gaussians onto the image plane and supervised using \[\mathcal{L}_{\text{depth}} = \frac{1}{|\Omega|} \sum_{\boldsymbol{u} \in \Omega} \left| \hat{D}(\boldsymbol{u}) - D(\boldsymbol{u}) \right|,\] where \(\hat{D}(\boldsymbol{u})\) denotes the rendered depth at pixel \(\boldsymbol{u}\), \(D(\boldsymbol{u})\) the corresponding sparse LiDAR depth, and \(\Omega\) the set of pixels with valid LiDAR measurements.

3.4.2 Multi-Frame Training↩︎

We follow a two-stage training protocol of LaGS, which consists of (i) single-frame pre-training for detection and (ii) multi-frame training for instance tracking. During the tracking stage, gradients are detached across consecutive frames to bound memory usage, while supervision is provided by the panoptic decoder as well as Gaussian-based auxiliary losses [5]. However, this training scheme does not explicitly initialize the temporal state of the proposed encoder. To expose the model to a well-formed temporal context, we introduce warmup frames in both training stages. Specifically, a small number of preceding frames are processed by the temporal encoder to initialize its latent state. These frames are excluded from loss computation and are not passed to the decoder. Gradients are detached during warmup steps, ensuring they serve only to initialize temporal context without increasing memory consumption. This strategy enables training under realistic streaming conditions, and we find that, despite the absence of cross-frame gradient flow, it is sufficient for the encoder to learn temporally consistent update dynamics.

4 Experiments↩︎

We evaluate our proposed streaming Gaussian encoder on camera-based 4D panoptic occupancy tracking using the Occ3D-extended nuScenes and Waymo benchmarks. [ssec:datasets] [ssec:implementation] describe datasets, metrics, and implementation details. Main quantitative results are presented in 4.3, followed by comparisons to single-frame occupancy methods (4.4), qualitative analyses (4.5), and ablations (4.6) assessing contributions of each component.

4.1 Datasets and Evaluation Metrics↩︎

We evaluate on the nuScenes [6] and Waymo [7] benchmarks using occupancy annotations from Occ3D [8]. We follow the standard camera-based 4D panoptic occupancy tracking protocol established by TrackOcc [4]. For both datasets, the evaluation volume spans −40 m to 40 m in the horizontal plane and −1 m to 5.4 m vertically, discretized with a voxel size of 0.4 m. Performance is primarily measured using Segmentation and Tracking Quality (STQ) [4], [32], defined as the geometric mean of Segmentation Quality (SQ) and Association Quality (AQ), jointly evaluating semantic occupancy and temporal instance consistency. To assess single-frame panoptic performance without temporal association, we additionally report STQ\(_\mathrm{1}\) and AQ\(_\mathrm{1}\), the non-temporal counterparts of STQ and AQ [5]. We report the binary IoU for free/occupied occupancy prediction.

4.2 Implementation Details↩︎

Following LaGS [5], we use an ImageNet-pretrained VoVNetV2-99 (V99) backbone [33] with input resolutions of \(800{\times}320\) for nuScenes and \(704{\times}256\) for Waymo, and adopt the same multi-view lifting, 3D feature construction, architecture, optimizer, learning schedule, and loss weights. Unless stated otherwise, all experiments use the V99 backbone; additional results with a ResNet-50 (R50) backbone are reported for comparison. Based on empirical analysis, we employ our encoder only at the fine stream of the two-stream LaGS variant (LaGS-2s), as the coarse stream fails to produce meaningful Gaussian primitives. We maintain a fixed query budget of \(K{=}8192\), with at least \(M{=}2048\) newly initialized queries per frame and latent dimension \(d{=}256\).

We train the models on \(8\) NVIDIA L40 GPUs for \(24\) epochs (12 detection pre-training and 12 tracking epochs), with a batch size of \(1\) per GPU. Each training sequence contains \(8\) consecutive frames. During detection pre-training, sequences comprise \(7\) warmup frames followed by one supervised frame, while tracking training uses \(5\) warmup and \(3\) supervised frames. Gradients are detached between frames. The Gaussian depth loss is applied with weight \(1.0\) at \(\frac{1}{4}\) input resolution.

4.3 Main Results↩︎

Results for 4D panoptic occupancy tracking on Occ3D-nuScenes and Occ3D-Waymo are presented in [tab:results_nusc,tab:results_waymo]. Our method outperforms prior work across all major temporal (STQ, AQ) and single-frame (STQ\(_\mathrm{1}\), AQ\(_\mathrm{1}\)) metrics. On Occ3D-nuScenes, our approach improves STQ from 32.3 to 34.4 over LaGS-2s, with corresponding gains in AQ, semantic quality, and binary occupancy IoU. Improvements are observed for both thing and stuff classes, indicating that the proposed streaming representation benefits both dynamic object reasoning and static scene modeling. On Occ3D-Waymo, our method similarly advances the state of the art, achieving 21.9 STQ (0.7 over LaGS-2s) together with improvements in AQ, STQ\(_\mathrm{1}\), AQ\(_\mathrm{1}\), and mIoU. Although the gains are smaller than on nuScenes, they are observed across both temporal and semantic metrics. We attribute the reduced margin to Waymo’s predominantly forward-driving scenarios and lower occlusion frequency, which diminish the benefits of extended temporal aggregation.

2.5pt

2.5pt

Despite introducing temporal state management, our encoder incurs negligible computational overhead. On Occ3D-nuScenes, our method processes 2.94 samples/s at inference on an NVIDIA L40 GPU, compared to 3.00 for LaGS, corresponding to a marginal slowdown of approximately 2 %. This demonstrates that persistent volumetric modeling can be achieved with virtually no throughput penalty while substantially improving temporal consistency.

4.4 Comparison to Single-Frame 3D Occupancy Methods↩︎

9pt

While our primary focus is camera-based 4D panoptic occupancy tracking, we additionally compare against semantic occupancy methods on Occ3D (¿tbl:tab:results95occ3d?). Because 4D-POT jointly predicts geometry, semantics, and temporal identities, it constitutes a strictly more challenging task than semantic occupancy alone. Nevertheless, our method achieves 43.5 mIoU and 71.7 IoU, substantially outperforming prior 4D-POT approaches such as TrackOcc and LaGS-2s. Despite optimizing for the more demanding panoptic and temporal setting, our model remains highly competitive with dedicated semantic occupancy approaches, even surpassing BEVDet4D+COTR without coarse-to-fine semantic grouping (CFSG) [15], the closest purely semantic counterpart. These results significantly narrow the gap between 3D and 4D occupancy approaches.

4.5 Qualitative Results↩︎

Qualitative comparisons on the Occ3D-nuScenes and Occ3D-Waymo datasets are shown in 3 4. Notably, our streaming Gaussian encoder yields more stable volumetric predictions under partial observations and occlusions. On nuScenes, the persistent representation enables the model to retain static scene layout even after regions become occluded by vegetation or terrain, whereas LaGS progressively loses road extent and fine structural details. On Waymo, similar behavior is observed in cluttered parking scenarios, where our method better preserves both parked vehicles and surrounding structures under vegetation and structural occlusions. These results indicate that enforcing temporal coherence at the representation level improves robustness to occlusion and promotes consistent scene modeling over time.

4.6 Ablation Studies↩︎

2.5pt

We conduct controlled ablations to quantify the contribution of each component of our proposed streaming Gaussian encoder. Specifically, we analyze (i) core architectural elements, (ii) pruning strategies, (iii) query initialization schemes, and (iv) temporal context length. Unless stated otherwise, results are reported on the Occ3D-nuScenes validation split.

Core Architectural Components: Component ablations are shown in ¿tbl:tab:ablations95components? (top block). Starting from a baseline without ego-motion compensation (EMC) or depth supervision (DS), adding EMC substantially improves STQ\(_\mathrm{1}\) and AQ\(_\mathrm{1}\), highlighting the importance of spatial alignment for persistent state propagation. Without EMC, propagated queries drift, limiting temporal aggregation. Adding DS further improves semantic and geometric metrics by shaping opacity to reflect geometric visibility (cf. 5), strengthening visibility-driven confidence accumulation. Together, EMC and DS produce clear cumulative gains. We additionally evaluate an explicit temporal consistency loss (CL) on Gaussian parameters, enforcing cross-frame consistency of opacity, geometry, and semantic logits via \(\mathcal{L}_2\) and KL penalties. However, this provides no measurable benefit. As the model is trained in a streaming regime with repeated query propagation and refinement, it already learns stable update dynamics. Thus, representation-level temporal coherence emerges naturally from streaming refinement and visibility aggregation, rendering additional parameter-level constraints unnecessary.

Pruning Strategy: We next compare pruning variants (¿tbl:tab:ablations95components?, middle block). Replacing hybrid rank-based pruning \(\mathcal{M}(r)\) with opacity-only pruning \(\mathcal{M}(\alpha)\) consistently reduces performance, suggesting that instantaneous visibility alone is insufficient for robust state management. Substituting confidence with a learned semantic accuracy score, \(\mathcal{M}(r_{\text{acc}})\), leads to a further performance drop, indicating that accumulated visibility provides a stronger signal for query retention than instantaneous correctness estimates. Finally, deterministic retention \(\operatorname{top}_k(r)\) performs slightly worse than stochastic multinomial sampling, suggesting that while the hybrid ranking criterion is the primary contributor to performance, stochasticity provides a small additional benefit.

6pt

Query Initialization: By default, new queries are initialized via multinomial sampling proportional to feature magnitude \(\|\boldsymbol{f}\|\), favoring well-supported regions. To encourage exploration of newly observed or underexplored regions, we additionally evaluate density-aware weighting \(\|\boldsymbol{f}\| \cdot e^{-\beta \boldsymbol{d}}\), where \(\boldsymbol{d}\) is the accumulated Gaussian density from the previous timestep. This biases initialization towards uniform coverage (¿tbl:tab:ablations95components?, bottom block). However, we find that moderate density-aware weighting performs nearly identically to magnitude-based initialization while stronger weighting degrades performance, suggesting that confidence-guided pruning and iterative refinement already provide sufficient adaptability.

Temporal Context Size: We analyze the number of warm-up frames used to initialize the persistent state (¿tbl:tab:ablations95temporal?). Increasing the context from 2 to 7 frames consistently improves performance, after which gains largely saturate. We therefore adopt 7 warm-up frames as a practical trade-off between performance and training efficiency.

5 Conclusion↩︎

We presented a streaming Gaussian encoder for camera-based 4D panoptic occupancy tracking that maintains a persistent and temporally coherent volumetric scene representation. Moving beyond frame-centric feature recomputation, our method propagates and updates a fixed-budget set of latent Gaussian queries, enabling representation-level temporal consistency with negligible inference overhead. Central to our approach is the interpretation of Gaussian opacity as a proxy for per-frame visibility, whose accumulation over time enables principled confidence-driven state management. Together with ego-motion-aware propagation and streaming refinement, this results in a compact yet expressive scene representation that remains stable under occlusion, partial observation, and dynamic changes. Extensive experiments on Occ3D-extended nuScenes and Waymo demonstrate state-of-the-art performance for camera-based 4D-POT. More broadly, our findings highlight the potential of persistent volumetric representations as a step toward continuous 4D scene modeling in camera-only systems.

References↩︎