1 Introduction↩︎

Autonomous driving operates in partially observable environments where actors frequently become fully unobservable due to occlusion or sensor limits. Our analysis of nuScenes shows that approximately 30% of actors are fully unobservable at each timestep, including many within close proximity to the ego vehicle. Therefore, safe driving requires maintaining persistent hypotheses about actors even when they are temporarily unobservable. This capability corresponds to object permanence: the ability to reason about objects that continue to exist despite missing observations.

Recent end-to-end autonomous driving (E2E-AD) systems jointly model perception, prediction, and planning within a unified architecture. However, most existing approaches implicitly couple actor existence to instantaneous observations. When an actor becomes fully unobservable, its representation often degrades or disappears entirely (Fig. 1), removing it from the scene representation used by downstream prediction and planning modules. As a result, the planner must reason only over currently visible agents, leading to reactive or brittle behavior in occlusion-heavy scenarios such as intersections and crosswalks. While temporal aggregation improves short-term stability, it does not explicitly model actor persistence during extended observability gaps.

Current perception benchmarks implicitly equate observability with existence: when an actor becomes fully occluded, it typically disappears from both supervision and evaluation. This prevents models from learning persistent representations across observability gaps. To address this limitation, we introduce nuScenes-Permanence, an extension of nuScenes that provides annotations and evaluation protocols for unobservable actors, enabling systematic training and evaluation of object permanence.

Building on this benchmark, we introduce BeyondSight, a permanence-aware end-to-end driving framework that explicitly decouples actor existence from instantaneous observability by maintaining persistent actor hypotheses over time. BeyondSight extends sparse-query scene representations with temporal propagation and observation-conditioned updates, allowing actor hypotheses to persist through observability gaps. Propagated and observation-conditioned hypotheses are fused into a unified representation that contains both observable and unobservable actors. This persistent scene representation provides richer context for downstream motion prediction and planning, improving decision-making in occlusion-heavy scenarios.

Our contributions are summarized as follows:

• We formalize object permanence for end-to-end autonomous driving, defining the requirement that actors previously observed remain represented even when temporarily unobservable in both model representations and dataset annotations.

• We introduce nuScenes-Permanence, extending nuScenes with unobservable actor annotations and an observability-conditioned evaluation protocol that enables systematic training and evaluation of persistence-aware models.

• We propose BeyondSight, a permanence-aware extension to sparse-query end-to-end driving architectures that maintains persistent actor hypotheses through temporal propagation and observability-aware fusion.

Extensive experiments on the nuScenes benchmark demonstrate the effectiveness of permanence-aware reasoning. On nuScenes, BeyondSight improves planning error from 0.61 to 0.54 L2\(_{avg}\) while maintaining competitive perception performance. On the proposed nuScenes-Permanence benchmark, it dramatically improves reasoning about occluded actors, increasing \(\text{mAP}_\text{unobs}\) from 0 to 0.249. Qualitative results further show that the model maintains stable hypotheses for actors that become fully occluded and produces safer planning behavior in complex scenes with prolonged occlusions.

2 Related Work↩︎

2.0.1 Vision-Based End-to-End Autonomous Driving.↩︎

Vision-based end-to-end autonomous driving (E2E-AD) jointly optimizes perception, motion forecasting, and planning within a single differentiable stack. Most modern E2E-AD systems operate in BEV, combining camera-to-BEV lifting with temporal aggregation to support long-horizon planning. Lift-Splat-Shoot [1], ST-P3 [2], and BEVFormer [3] established the core recipe of BEV scene encoding and recurrent or attention-based temporal fusion. Building on this, UniAD [4] unified detection, tracking, prediction, and planning in a planning-centric architecture, while VAD [5] introduced vectorized scene representations that improve efficiency and downstream controllability. Robust motion prediction remains a central requirement for deployable autonomy, especially under uncertainty, distribution shift, and interaction-heavy driving scenarios [6].

Recent work targets scalability and latency by replacing dense BEV supervision with sparse instance-centric representations. SparseDrive [7] uses sparse queries to represent agents and map elements for joint detection, prediction, and planning. SSR [8] further prunes supervision and computation via navigation-guided sparse tokens, while DriveAdapter [9] and DriveTransformer [10] reduce coupling and improve streaming behavior through teacher–student decoupling and unified query processing. Despite architectural progress, most E2E-AD stacks still rely on training signals and metrics defined over currently observable actors by filtering annotations with zero LiDAR points, effectively removing fully occluded actors from supervision and evaluation. As a result, actor hypotheses often degrade or disappear after missed detections or prolonged occlusion, leading to weak temporal consistency and limited recovery once the actor reappears.

2.0.2 Temporal Modeling and Query Propagation.↩︎

Temporal reasoning in vision-based driving is commonly implemented as (i) BEV feature recurrence or (ii) object-centric query propagation. BEVFormer [3] aggregates historical BEV features with spatiotemporal attention, improving temporal stability for BEV reasoning. In object-centric pipelines, StreamPETR [11] propagates instance queries across frames to enable efficient temporal association and multi-view 3D detection. More recent E2E-AD systems extend these ideas to multi-task temporal coherence: BridgeAD [12] leverages historical prediction to inform present perception and planning, MomAD [13] introduces momentum objectives to stabilize planning, and ForeSight [14] shares memory between detection and forecasting to improve streaming consistency. Related multi-object tracking methods also study hypothesis persistence through missed observations and association ambiguity. SWTrack [15] maintains multiple 3D tracking hypotheses in a sliding window, while SCATr [16] mitigates query suppression for joint detection and tracking.

These approaches improve short-horizon stability, association, and interaction modeling, but query survival is still largely driven by recent observations and evaluation is typically defined only where ground truth remains observable. As a result, hypotheses for fully unobservable actors often decay or disappear, yielding missing forecasts and incomplete context for planning. BeyondSight instead treats persistence as a first-class requirement: temporally propagated hypotheses remain valid independent of immediate sensor evidence, and are explicitly supervised and evaluated across observability gaps.

2.0.3 Occlusion and Object Permanence.↩︎

Object permanence under occlusion has been studied in video understanding and tracking, where learning to preserve identity and state through occlusions requires explicit memory and supervision. OPNet [17] and PermaTrack [18] show that tracking through occlusion benefits from objectives that encourage a persistent latent state. Subsequent work further demonstrates that heavy occlusion demands mechanisms and losses beyond short-term feature aggregation [19], [20].

In autonomous driving perception, several methods improve robustness to partial occlusion by strengthening features or reasoning about difficult conditions (e.g., CorrBEV [21] and ReasonNet [22]). However, in end-to-end driving systems, actors that become fully unobservable are typically excluded from supervision and evaluation under standard protocols. Temporal propagation may occasionally recover actors after brief observation gaps, but persistence across longer occlusions is neither explicitly supervised nor evaluated. BeyondSight closes this gap by combining (i) permanence-aware supervision for fully unobservable actors, (ii) architectural state propagation across observability gaps, and (iii) an observability-conditioned evaluation protocol on nuScenes-Permanence.

3 Problem Formulation↩︎

3.0.1 Partial Observability.↩︎

We define the observability of an actor using a binary indicator \(o_t^i \in \{0,1\}\) for actor \(i\) at time \(t\). Following the nuScenes protocol, observability is approximated by sensor support: \[o_t^i \approx \mathbb{I}\!\left(n_{\text{pts},t}^i > 0\right), \label{eq:observability}\tag{1}\] where \(n_{\text{pts},t}^i\) is the number of associated LiDAR and radar returns. A scene exhibits partial observability when an actor that was previously observable becomes fully unobservable due to occlusion or sensor resolution.

3.0.2 Object Permanence.↩︎

We define object permanence as the requirement that actors previously observed remain represented even when temporarily unobservable. Formally, a representation satisfies permanence if \[\forall i,t:\; \Big(o_t^i = 0 \;\land\; \exists\, t' < t : o_{t'}^i = 1\Big) \;\Rightarrow\; \exists\, r_t^i \in \mathcal{R}_t, \label{eq:permanence}\tag{2}\] where \(\mathcal{R}_t\) denotes the set of actor representations maintained at time \(t\) and \(r_t^i\) corresponds to actor \(i\). Equation 2 expresses permanence as a representation-level constraint that decouples actor existence from instantaneous observation.

In a driving model, \(\mathcal{R}_t\) corresponds to the maintained hypothesis set, requiring actor hypotheses to persist across observability gaps. In a dataset, \(\mathcal{R}_t\) corresponds to annotated actor states, requiring trajectories to remain defined during unobservable intervals. BeyondSight enforces Eq. 2 architecturally (Sec. 4), while the nuScenes-Permanence benchmark enforces it through supervision and evaluation (Sec. 5).

4 Method↩︎

BeyondSight builds upon SparseDrive [7], a sparse query-based end-to-end driving architecture that jointly performs detection, prediction, and planning from multi-view camera inputs. The model maintains a set of instance queries representing dynamic actors and map elements, which are iteratively refined using deformable feature aggregation and temporal attention.

4.1 BeyondSight Model↩︎

To satisfy the actor persistence constraint defined in Sec. 3, BeyondSight introduces a persistence-aware actor inference pipeline while retaining the SparseDrive backbone and objectives. Actor queries are updated through three stages resembling a Bayesian filtering approach: \[\begin{align} q_t^{\text{prior}} &= D_{\text{prior}}(q_{t-1}) \\ q_t^{\text{obs}} &= D_{\text{obs}}(F_t, q_t^{\text{prior}}) \\ q_t^{\text{post}} &= D_{\text{fusion}}(q_t^{\text{prior}}, q_t^{\text{obs}}) \end{align}\] where \(F_t\) denotes BEV features extracted from the perception backbone. The terms prior, observation, and posterior are used in a filtering-inspired sense: they denote learned latent query representations for temporal propagation, image-conditioned update, and fused refinement, rather than calibrated probability distributions.

This formulation separates temporal hypothesis propagation from observation updates, enabling actor representations to persist during observability gaps. Unlike standard sparse-query propagation, where previous detections are refined through the observation decoder and remain tied to recent visual evidence, BeyondSight propagates motion-conditioned hypotheses in an image-feature-free temporal prior decoder, then fuses them with observation-conditioned queries. This keeps unobservable actors available to prediction and planning without requiring the visual branch to hallucinate non-visible evidence.

4.1.1 Temporal Prior Decoder.↩︎

The Temporal Prior Decoder propagates actor queries from the previous frame to produce a motion-conditioned hypothesis set. Historical detection queries and motion-predicted queries are fused through a lightweight MLP and refined using transformer-style self-attention operating solely on the queries. This stage updates actor hypotheses based on temporal consistency without accessing image features: \[q_t^{\text{prior}} = D_{\text{prior}}(q_{t-1}).\]

4.1.2 Observation Decoder.↩︎

The temporal prior is combined with newly initialized object queries and passed to the standard SparseDrive detection decoder, which attends to BEV features to produce observation-conditioned hypotheses: \[q_t^{\text{obs}} = D_{\text{obs}}(F_t, q_t^{\text{prior}}).\] This stage is supervised only using observable ground-truth actors, allowing the propagated prior to maintain hypotheses for actors that remain temporarily unobservable.

4.1.3 Posterior Fusion Decoder.↩︎

The Posterior Fusion Decoder reconciles propagated and observation-conditioned hypotheses. For each actor, the proposal with higher classification confidence is retained and refined through a final query refinement stage: \[q_t^{\text{post}} = D_{\text{fusion}}(q_t^{\text{prior}}, q_t^{\text{obs}}).\] The resulting actor set \(q_t^{\text{post}}\) forms the scene representation used by the downstream motion prediction and planning modules. Unlike SparseDrive, which supervises only observable actors, BeyondSight also supervises actors that are temporarily unobservable, allowing occluded actors to influence trajectory prediction and planning.

4.2 Optimization↩︎

We follow the two-stage training protocol of SparseDrive. In Stage 1, the sparse perception module is trained to learn the scene representation. In Stage 2, perception, motion prediction, and planning modules are trained jointly end-to-end. The overall objective is \[\mathcal{L} = \mathcal{L}_{\text{det}} + \mathcal{L}_{\text{det}}^\prime + \mathcal{L}_{\text{map}} + \mathcal{L}_{\text{motion}}^\prime + \mathcal{L}_{\text{plan}} + \mathcal{L}_{\text{depth}} .\]

The standard detection loss \(\mathcal{L}_{\text{det}}\) supervises the Observation Decoder using observable actors, while the modified detection loss \(\mathcal{L}_{\text{det}}^\prime\) supervises the temporal and fusion decoders using both observable and unobservable actors.

4.2.1 Permanence-Aware Supervision.↩︎

Let \(G_{\text{obs}}\) and \(G_{\text{unobs}}\) denote observable and unobservable ground-truth actors at time \(t\), with \[G_{\text{all}} = G_{\text{obs}} \cup G_{\text{unobs}} .\] Motion supervision is extended to unobservable actors by applying the motion loss over the full actor set: \[\mathcal{L}_{\text{motion}}^\prime = \mathcal{L}_{\text{motion}}(G_{\text{all}}).\]

To explicitly decouple existence from observability, the posterior decoder predicts an observability state \(\hat{o}_t^i\) for each actor query. This is supervised using binary cross-entropy: \[\mathcal{L}_{\text{obs}} = \sum_{i \in G_{\text{all}}} \mathrm{BCE}(\hat{o}_t^i, o_t^i).\]

The final detection objective becomes \[\mathcal{L}_{\text{det}}^\prime = \mathcal{L}_{\text{det}}(G_{\text{all}}) + \lambda_{\text{obs}} \mathcal{L}_{\text{obs}}.\]

5 Permanence Benchmark↩︎

Standard perception benchmarks such as nuScenes [23] evaluate only actors that are observable at each timestamp, typically defined by the presence of LiDAR or radar returns. When an actor becomes fully occluded or moves outside sensor range, annotations and evaluation are usually discontinued. As a result, current benchmarks implicitly equate observability with existence, preventing models from learning or being rewarded for maintaining actor hypotheses during observability gaps. To support training and evaluation of persistence-aware models, we introduce nuScenes-Permanence, an extension of the nuScenes dataset that provides supervision for actors that remain present in the scene but are unobservable. The benchmark extension targets approximately continuous actor motion, while abrupt hidden maneuvers that cannot be reliably inferred from surrounding observations are outside its scope.

5.1 Annotation Extension.↩︎

We extend actor trajectories through intervals where sensor support is absent by reconstructing missing states offline using physically grounded motion priors. Observable segments remain unchanged, while unobservable intervals are completed using interpolation between observed states or short-horizon extrapolation when actors leave the sensor field of view. Annotated 3D boxes with zero LiDAR or radar points are retained rather than filtered, enabling supervision for heavily occluded actors that remain present in the scene.

Applying this procedure expands the nuScenes annotations from 932k to 1.33M boxes (+30%), transforming the dataset into an occlusion-aware benchmark that supports supervision across observability gaps. The trajectory completion model operates strictly offline and is not used at inference time. BeyondSight must therefore learn to maintain persistent hypotheses without access to completed trajectories.

To validate the generated annotations, we perform a holdout reconstruction study in which observable trajectory intervals are masked, reconstructed with the same offline pipeline, and compared against the original ground truth. We use the resulting reconstruction errors to calibrate an occlusion-duration-dependent matching tolerance for unobservable evaluation. Additional validation details are provided in the supplementary material.

5.2 Observability-Conditioned Evaluation.↩︎

Standard nuScenes evaluation penalizes predictions of unobservable actors as false positives, discouraging persistence across occlusion intervals. To evaluate object permanence, we introduce an observability-conditioned protocol that partitions ground truth into observable (\(G_{\text{obs}}\)) and unobservable (\(G_{\text{unobs}}\)) subsets. Each metric specifies a target ground-truth set \(G\) and an ignored set \(I\). Predictions are first matched to \(G\) using the standard nuScenes matching procedure; remaining predictions matched to \(I\) are removed prior to metric accumulation. This symmetric ignore rule prevents cross-penalization between observable and unobservable actors. In addition to the standard detection metric \(\text{mAP}\), we report:

• \(\text{mAP}_{\text{obs}}\): performance on continuously observable actors,

• \(\text{mAP}_{\text{unobs}}\): performance during unobservable intervals,

• \(\text{mAP}_{\text{all}}\): performance across the full actor set.

Because reference trajectories for unobservable actors are partially extrapolated, we use an adaptive matching tolerance that accounts for accumulated motion uncertainty during unobservability. This tolerance reduces to the standard nuScenes threshold when the unobservability duration approaches zero, ensuring compatibility with the official evaluation protocol. Details of the trajectory completion model and tolerance estimation are provided in the supplementary material.

6 Experiments↩︎

6.1 Experimental Setup↩︎

6.1.1 Dataset.↩︎

We evaluate on nuScenes [23], a large-scale autonomous driving dataset containing 1,000 driving scenes of approximately 20 s each. Following standard practice, we use the official train/validation split (700/150 scenes). Each keyframe provides synchronized multi-view imagery from six cameras together with 3D bounding box annotations at 2 Hz for ten object classes. In addition to the standard dataset, we evaluate on nuScenes-Permanence, our extended benchmark introduced in Section 5. The extension augments nuScenes with supervision for actors that remain physically present but become temporarily unobservable due to occlusion, sensor range limits, or field-of-view constraints. This extension increases the number of annotated actor states by approximately 30%, enabling systematic evaluation of persistence-aware models.

6.1.2 Metrics.↩︎

We report the standard nuScenes metrics to ensure comparability with prior work. We evaluate detection using mAP and NDS together with the standard nuScenes error metrics mATE, mASE, mAOE, mAVE, and mAAE [23]. Tracking performance is evaluated using AMOTA, AMOTP, recall, and ID switches. For motion forecasting we report minADE, minFDE, Miss Rate (MR), and End-to-end Prediction Accuracy (EPA) following the VIP3D protocol [24] and UniAD-style evaluation [4]. Planning quality is measured using open-loop trajectory error and collision rate following the VAD evaluation protocol [5].

To evaluate persistence under partial observability, we additionally report the proposed observability-conditioned metrics on nuScenes-Permanence: \(\text{mAP}_{\text{obs}}\), \(\text{mAP}_{\text{unobs}}\), \(\text{mAP}_{\text{all}}\). These metrics isolate performance on observable and unobservable actors while applying symmetric ignore rules to prevent penalization. For unobservable intervals whose reference trajectories are extrapolated, we use the adaptive matching tolerance described in Section 5.

6.1.3 Implementation Details.↩︎

BeyondSight is implemented on top of SparseDrive and retains its backbone architecture, sparse scene representation, motion prediction, and planning modules. Unless otherwise specified, all architectural hyperparameters follow the corresponding SparseDrive configuration. The Temporal Prior Decoder and Posterior Fusion Decoder are implemented as a lightweight transformer decoder using self-attention, two-layer MLPs, and a detection head operating on the query embeddings, introducing less than \(5\%\) additional parameters.

Training follows a two-stage protocol. First, the sparse perception module is trained to learn the scene representation. Second, perception, motion prediction, and planning modules are trained jointly in an end-to-end manner. We extend detection and motion supervision to unobservable timestamps and introduce an observability classification loss with weight \(\lambda_{\text{obs}}=1.0\). Models are trained using AdamW with weight decay \(10^{-3}\), cosine learning-rate scheduling, and 500 warm-up iterations. Stage 1 uses learning rate \(4\times10^{-4}\), while Stage 2 uses learning rate \(3\times10^{-4}\). The image backbone learning-rate multiplier is set to 0.5 in Stage 1 and 0.1 in Stage 2. Full optimization details are provided in the supplementary material.

6.2 Main Result↩︎

We compare BeyondSight to state-of-the-art vision-based end-to-end driving baselines that jointly model perception, forecasting, and planning. Our primary baseline is SparseDrive [7], since BeyondSight directly extends its sparse-query representation and training protocol. We additionally compare to representative end-to-end driving systems including UniAD [4], VAD [5], and recent efficient E2E-AD approaches when supported by public code and evaluation settings. All methods are evaluated under the same nuScenes splits and metrics; where necessary, we re-train models under a unified pipeline to ensure fair comparison.

6.2.1 End-to-end Performance.↩︎

We first report standard nuScenes leaderboard metrics to ensure comparability with prior work (Tables 1 and 2). BeyondSight consistently improves the strongest SparseDrive baseline across multiple tasks while maintaining compatibility with the official evaluation protocol.

On perception, BeyondSight improves detection from \(0.415\) to \(0.427\) mAP and from \(0.526\) to \(0.536\) NDS. Tracking performance also improves, increasing AMOTA from \(0.372\) to \(0.401\). Motion forecasting performance remains competitive while slightly improving EPA. These results indicate that persistence-aware reasoning improves the quality and stability of the scene representation without degrading standard perception metrics.

BeyondSight also improves downstream planning performance. Planning error decreases from \(0.61\) to \(0.54\) L2\(_{\text{avg}}\), while the collision rate decreases from \(0.08\%\) to \(0.07\%\). Importantly, these gains are achieved without modifying the planner itself. Instead, improvements arise from the richer scene representation produced by permanence-aware actor modeling.

Crucially, the permanence-aware model does not incur additional false positives under the official nuScenes evaluation protocol. Standard mAP and NDS remain competitive with state-of-the-art baselines, demonstrating that maintaining persistent actor hypotheses does not negatively impact conventional benchmark performance.

6.2.2 Object Permanence Performance.↩︎

We next evaluate BeyondSight using the proposed observability-conditioned protocol on the nuScenes-Permanence benchmark. Table 3 reports detection performance for observable actors, unobservable actors, and the full actor set.

BeyondSight substantially improves performance during occlusion. The model achieves the highest mAP\(_{\text{unobs}}\), improving over the strongest baseline by a large margin. In contrast, performance on continuously observable actors remains nearly unchanged. This demonstrates that permanence-aware supervision specifically improves reasoning about unobservable actors without degrading standard detection performance. The aggregated metric mAP\(_{\text{all}}\) further confirms improved holistic scene understanding. By maintaining persistent hypotheses across observability gaps, BeyondSight produces a more complete representation of the dynamic environment.

6.3 Ablation Study↩︎

We conduct ablation experiments to analyze the contribution of the key components introduced in BeyondSight. All ablations are evaluated on the nuScenes validation split using the same training protocol as the full model. Table 4 reports cumulative component additions under two annotation settings: standard nuScenes annotations and nuScenes-Permanence annotations.

Under standard nuScenes annotations, unobservable actors are not explicitly supervised, and all configurations obtain zero mAP\(_{\text{unobs}}\). Adding the Temporal Prior Decoder slightly improves mAP\(_{\text{obs}}\), mAP\(_{\text{all}}\), and planning error, but does not enable detection of fully unobservable actors by itself. With nuScenes-Permanence annotations, the same temporal prior provides a substantial gain on mAP\(_{\text{unobs}}\), showing that temporal propagation is most effective when paired with permanence-aware supervision.

The Posterior Fusion Decoder further improves both observable and unobservable detection by reconciling propagated hypotheses with observation-conditioned queries. Adding unobservable-actor supervision improves the persistence of occluded actors and reduces planning error. The full BeyondSight model achieves the best overall performance, improving mAP\(_{\text{unobs}}\) from 0.021 to 0.249 over the SparseDrive baseline trained on nuScenes-Permanence, while also improving mAP\(_{\text{obs}}\), mAP\(_{\text{all}}\), L2\(_{\text{avg}}\), and CR\(_{\text{avg}}\).

6.4 Planning under Occlusion↩︎

We qualitatively evaluate BeyondSight in scenarios involving partial observability and prolonged occlusions. Figure 3 shows a planning-relevant case where an actor becomes fully occluded by another vehicle. SparseDrive drops the hidden actor once direct evidence disappears, causing the ego plan to intersect the actor’s ground-truth future trajectory. BeyondSight maintains a persistent hypothesis for the occluded actor and produces a plan with additional clearance. This example illustrates how permanence-aware scene representations can provide useful context to downstream planning when hidden actors remain decision-relevant.

To quantify this behavior beyond a single qualitative example, we evaluate a high-occlusion subset containing 600 validation samples with occluded dynamic agents whose future trajectories approach the ego plan within 12 m over a 4 s horizon. On this subset, SparseDrive degrades to 0.64 L2\(_{\text{avg}}\) and 0.163 CR\(_{\text{avg}}\), while BeyondSight achieves 0.56 L2\(_{\text{avg}}\) and 0.08 CR\(_{\text{avg}}\). These results suggest that persistent actor hypotheses are most useful when unobservable actors remain relevant to the ego plan.

7 Conclusion↩︎

We presented BeyondSight, a permanence-aware end-to-end driving framework designed to operate reliably under partial observability. Unlike existing systems that implicitly couple actor existence to instantaneous observations, BeyondSight maintains persistent actor hypotheses across occlusion intervals through temporally propagated and observation-conditioned updates. We further introduced nuScenes-Permanence, an extension of the nuScenes benchmark that enables supervision and evaluation of actors during unobservable intervals through observability-conditioned metrics. This benchmark provides the first systematic evaluation protocol for object permanence in end-to-end autonomous driving.

Limitations. BeyondSight’s persistent hypotheses can become stale during long occlusions, leading to false persistence or localization drift when actors deviate from the learned temporal prior. nuScenes-Permanence unobservable annotations target occlusions with approximately continuous motion and may not capture abrupt hidden maneuvers such as sudden stops, turns, or intent changes. Future work should explore uncertainty-aware memory, simulation-based unobservable annotations with privileged labels, and multi-observer datasets for direct occlusion evaluation.

8 Additional Implementation Details↩︎

8.0.1 Architecture.↩︎

Our implementation builds on SparseDrive [7] with a ResNet-50 backbone pretrained on ImageNet and an FPN producing four feature levels with strides \(\{4,8,16,32\}\) and 256 channels. Multi-view inputs from six cameras are resized to \(704 \times 256\). The transformer hidden dimension is \(d=256\) with \(8\) attention heads throughout.

8.0.2 Perception Head.↩︎

The 3D detection head follows the Sparse4D design with 900 anchor queries initialised via \(k\)-means clustering on the training set. The decoder contains six layers: the first processes single-frame features, while the remaining layers incorporate temporal context via a temporal attention over up to 600 propagated instances with confidence decay. Each layer applies graph attention, layer normalisation, deformable feature aggregation with twelve 3D keypoints per anchor (six fixed, six learnable), an FFN, and box/class refinement. An auxiliary visibility prediction head is trained with Focal Loss (\(\gamma{=}0\), \(\alpha{=}0.4\)). Detection supervision uses Focal Loss for classification (\(\gamma{=}2\), \(\alpha{=}0.25\), weight \(2.0\)) and sparse 3D box regression (weight \(0.25\)).

8.0.3 Motion and Planning.↩︎

The motion-planning module maintains an InstanceQueue storing the top-50 detected agents across \(L{=}4\) frames with ego-motion compensation applied to anchors. Future agent trajectories are predicted over \(T_{\text{fut}}{=}12\) steps (\(\approx 6\) s at 2 Hz) with \(K{=}6\) modes. Ego plans are predicted over \(T_{\text{ego}}{=}6\) steps with six modes. The decoder applies three blocks of temporal attention, agent–agent attention, layer normalisation, and FFN before a final refinement stage. No cross-attention to map features is used. Motion and planning are supervised using Focal Loss for mode classification (weights \(0.2\) and \(0.5\)) and L1 regression losses (weights \(0.2\) and \(1.0\)). An additional L1 loss (weight \(1.0\)) supervises ego-status regression.

8.0.4 Training.↩︎

Models are trained in two stages on the nuScenes trainval split using 8 NVIDIA A100 GPUs. Stage 1 trains the sparse perception module for 100 epochs with a total batch size of 64, learning rate \(4\times10^{-4}\), image-backbone learning-rate multiplier 0.5, and weight decay \(10^{-3}\). Stage 2 jointly trains perception, motion prediction, and planning for 10 epochs with total batch size of 48, a learning rate of \(3\times10^{-4}\), an image-backbone learning-rate multiplier of 0.1, and weight decay of \(10^{-3}\). Both stages use AdamW, cosine learning-rate scheduling with 500 warm-up iterations, and mixed precision (FP16). Data augmentation includes random resize (\([0.40,0.47]\)), horizontal flip, and photometric distortion. Actors beyond 55 m from the ego vehicle are filtered during training.

9 Additional Results↩︎

9.0.1 Additional Backbone Experiments.↩︎

To evaluate the robustness of BeyondSight across backbone architectures, we conduct additional experiments using a ResNet-101 backbone. All training settings remain identical to the ResNet-50 configuration described in the main paper. Table 5 compares SparseDrive and BeyondSight under both backbone configurations. This comparison evaluates whether the improvements from permanence-aware modeling persist when using a stronger visual encoder.

9.0.2 Extended Metrics Results.↩︎

For brevity, several perception and prediction metrics were omitted from the main paper. This section reports the complete set of evaluation metrics for reference. Table 6 reports the vectorized map prediction metrics. Table 7 reports detailed trajectory prediction metrics for vehicles and pedestrians. We report Average Displacement Error (ADE), Final Displacement Error (FDE), Miss Rate (MR), and End-to-end Prediction Accuracy (EPA).

9.0.3 Latency Analysis.↩︎

We measure inference latency on a single NVIDIA RTX 3090 GPU with batch size 1 and input resolution \(704\times256\). BeyondSight introduces a slight increase in computational overhead due to temporal propagation and permanence reasoning. Runtime decreases slightly from 6.1 FPS to 6.0 FPS (-1.8%) compared to the SparseDrive baseline, indicating that permanence-aware modeling introduces minimal additional computational cost.

9.0.4 Occlusion-Duration Analysis.↩︎

We additionally stratify object-permanence performance by occlusion duration. For each unobservable actor, we compute the time since it was last observable and report detection quality, permanence recall, and false discovery rate. Here, TPR\(_{\text{unobs}}\) measures the fraction of unobservable actors that are maintained, while FDR\(_{\text{unobs}}\) measures the fraction of maintained unobservable predictions that do not match a valid unobservable actor. Table 8 shows that performance degrades with longer occlusions: recall decreases and false persistence increases as hypotheses become stale. This highlights long-duration occlusion as a key remaining challenge for permanence-aware models.

10 nuScenes-Permanence Annotation Generation↩︎

This section provides implementation and validation details for the offline annotation generation procedure used to construct nuScenes-Permanence, together with additional analysis of the observability-conditioned evaluation protocol. We focus on details omitted from the main paper for brevity. Figure 4 summarizes the prevalence of full observability gaps in nuScenes. The left panel reports the distribution of full-occlusion durations, while the right panel reports the corresponding ego-distance distribution. These statistics show that complete sensor absence occurs frequently across a broad range of distances and is not limited to rare edge cases.

10.0.1 Annotation Generation Process.↩︎

To supervise persistence through observability gaps, annotations must remain defined when actors are physically present but fully unobservable. Since such intervals are not consistently retained under the standard nuScenes filtering rules, we construct additional annotations offline using a trajectory completion procedure.

For each actor track, we identify contiguous intervals of missing observability and estimate the latent actor state during those intervals. Each generated state is assigned a provenance label \[\tau_t^i \in \{\texttt{observable},\texttt{zero-point},\texttt{interpolated},\texttt{extrapolated}\},\] which is stored and later used for analysis and evaluation.

(1) Zero-point boxes. Some annotated boxes remain temporally valid despite having zero LiDAR or radar support at a given timestamp. Rather than discarding these boxes, we retain them as direct supervision for actors that are heavily occluded but still present in the scene.

(2) Temporary gaps: interpolation. For actors that disappear and later reappear within the scene horizon, we reconstruct the intermediate states by fitting a motion prior between the last observable state and the first subsequent observable state. We use a constant turn-rate and acceleration model in SE(2), constrained by the endpoint positions, headings, and velocities: \[\mathbf{x}_t = (x_t, y_t, \theta_t, v_t).\] Let \(t_a\) and \(t_b\) denote the last observable and first re-observed timestamps surrounding a gap. Intermediate states for \(t \in (t_a, t_b)\) are generated by solving for a smooth kinematic trajectory consistent with the endpoint conditions.

(3) Terminal gaps: extrapolation. For actors that do not reappear before the scene ends, future states are estimated using a pretrained trajectory forecasting prior. This prior predicts short-horizon future motion from the last observable history. For stationary or near-stationary actors, or when the forecast confidence is below a threshold, we instead apply analytic kinematic propagation to avoid implausible drift: \[\mathbf{x}_{t+1} = f_{\text{kin}}(\mathbf{x}_t).\] Table 9 summarizes the composition of the resulting annotation set and the relative contribution of each trajectory completion mechanism.

10.0.2 Forecasting Model.↩︎

The motion forecasting model is trained on the nuScenes v1.0-trainval observable trajectory annotations. The model is adapted from SparseDrive [7] but removes the planning head and replaces the detection and mapping heads with ground truth object and map information. The model is trained on all detection class categories. The model is trained to predict 6 s futures, sampled at 2 Hz. Map context is encoded from six polyline types—lanes, stop signs, road edges, road lines, crosswalks, and speed bumps—within a \(150\,\text{m}\) radius, centered \(30\,\text{m}\) ahead of the focal agent. The \(z\)-axis and physical size attributes are masked from the input, reducing the task to bird’s-eye-view trajectory prediction. The model uses a ResNet-50 image encoder and 4-layer transformer decoder (\(d{=}128\), 8 heads) and trained with the same hyperparameter settings as BeyondSight.

Generated trajectories are filtered for physical consistency. In particular, we reject or clip trajectories that exhibit implausible velocity spikes, abrupt heading changes, or large off-drivable-area deviations for vehicle classes. Bounding-box dimensions and semantic class labels are retained from the nearest reliable observable state unless evidence suggests a valid state transition. The annotation model is used strictly offline during dataset construction and is never used at inference time.

10.0.3 Validation of New Annotations.↩︎

To assess the quality of the generated annotations, we perform a holdout reconstruction study on observable trajectory intervals, Table 10. For each selected interval, we remove the actor states from the annotation sequence, reconstruct them using the same interpolation or extrapolation procedure used for nuScenes-Permanence, and compare the reconstructed states to the original nuScenes ground truth. Errors are stratified by annotation provenance, object class, occlusion duration, and motion pattern. We then prevalence-weight the strata according to the distribution of generated unobservable labels used in evaluation. This provides an empirical estimate of the uncertainty introduced by the annotation-generation process.

The results show that reconstruction quality is highest for retained zero-point boxes and interpolation, and degrades with longer extrapolation horizons and dynamic motion. This behavior motivates the adaptive matching tolerance used in observability-conditioned evaluation. Reconstruction error increases smoothly with horizon and remains bounded over the occlusion durations most commonly observed in nuScenes. These results indicate that the generated supervision reflects controlled motion uncertainty, making it suitable for training and evaluation under temporary unobservability.

Figure 5 shows representative examples of retained zero-point boxes, interpolated segments, and extrapolated terminal segments across a range of traffic scenarios, including urban intersections, turns, and highway scenes. In each case, the completed annotations remain temporally smooth and geometrically consistent with the road layout.

11 Evaluation Metrics and nuScenes-Permanence↩︎

For clarity, Table 11 summarizes the evaluation metrics used throughout the paper. Standard nuScenes detection metrics include mAP, mATE, mASE, mAOE, mAVE, mAAE, and NDS. Tracking is evaluated using AMOTA, AMOTP, recall, and identity switches. Motion prediction is evaluated using minADE, minFDE, Miss Rate, and EPA, while planning is evaluated using open-loop trajectory error and collision rate. For nuScenes-Permanence, these metrics are further partitioned into observable, unobservable, and all-actor subsets using the observability-conditioned protocol described below.

The main paper introduces the observability-conditioned evaluation protocol. Here we summarize the implementation details used in our experiments.

11.0.1 Observability-Conditioned Evaluation.↩︎

At each timestamp the ground-truth actor set is partitioned into observable and unobservable subsets \[G_{\mathrm{obs}}, \qquad G_{\mathrm{unobs}}.\] Evaluation is performed by selecting a target set \(G\) and defining the complementary subset as an ignore set \(I\). Predictions are first matched to \(G\) using the standard nuScenes class-aware distance matching rule. Remaining predictions that match any element of \(I\) are removed before false-positive accumulation. This symmetric ignore rule prevents predictions corresponding to the complementary subset from being incorrectly penalized.

We report the following evaluation subsets: \[\begin{align} \mathrm{mAP}_{\mathrm{obs}} &: \text{performance on } G_{\mathrm{obs}}, \\ \mathrm{mAP}_{\mathrm{unobs}} &: \text{performance on } G_{\mathrm{unobs}}, \\ \mathrm{mAP}_{\mathrm{all}} &: \text{performance on } G_{\mathrm{obs}} \cup G_{\mathrm{unobs}} . \end{align}\]

11.0.2 Adaptive Matching Threshold.↩︎

Evaluation against extrapolated ground truth introduces additional localization uncertainty. To avoid conflating model error with extrapolation drift, we augment the standard nuScenes matching threshold with an adaptive tolerance:

\[d_{\mathrm{match}}(v,a,t,\ell) = d_{\mathrm{nuScenes}}(\ell) + \hat{d}_\ell(v,a,t),\]

where \(t\) denotes the duration of unobservability and \(\ell\) the object class. The additional tolerance is modeled as

\[\hat{d}_\ell(v,a,t) = \alpha_\ell t + \beta_\ell v t + \gamma_\ell a t^2,\]

capturing drift due to velocity and acceleration uncertainty during occlusion, where \(v\) and \(a\) denote the actor’s ground truth speed and acceleration magnitude at the last observable state. The coefficients \((\alpha_\ell,\beta_\ell,\gamma_\ell)\) are estimated using non-negative 90th-percentile quantile regression on approximately \(1.5\times10^6\) extrapolated box–horizon pairs generated by masking observable nuScenes annotations.

As \(t \rightarrow 0\), \(\hat{d}_\ell \rightarrow 0\), recovering the official nuScenes evaluation protocol for observable actors.

12 Limitations and Failure Cases↩︎

While BeyondSight improves reasoning under partial observability, several limitations remain. First, very long occlusion intervals (\(>6\) s) can lead to drift in propagated hypotheses when actor motion deviates from the assumed dynamics. Second, abrupt behavioral changes (e.g., sudden stops or turns after occlusion) remain difficult to anticipate because the model relies primarily on historical motion priors. Third, persistence may occasionally maintain low-confidence hypotheses for actors that have permanently exited the scene, which can introduce minor localization noise until the hypothesis is pruned by downstream confidence filtering. Addressing these challenges may require stronger interaction-aware forecasting or explicit uncertainty modeling during long observability gaps.

References↩︎