1 Introduction↩︎

Fusing complementary signals captured by different sensors is essential for autonomous driving perception systems. Leveraging the distinct characteristics of each sensor helps reduce prediction uncertainty, leading to more accurate and robust perception outcomes [1]–[3]. Since different sensors present data in varying formats, such as cameras providing perspective semantic data and Lidar capturing 3D spatial information, multi-modal fusion faces significant challenges due to these view discrepancies. To address this, some methods [2], [4]–[6] achieve multi-modal 3D object detection through point-level fusion. However, point-level fusion strategies are generally unsuitable for semantics-oriented 3D perception tasks like 3D semantic occupancy prediction. Consequently, recent approaches aim to construct unified representations for multi-modal feature fusion.

r8cm

Recently, the shared Bird’s Eye View (BEV) space has emerged as a promising direction for fusing cross-modal features to enable task-agnostic learning. Several existing fusion methods [1], [7], such as BEVFusion [1], integrate multimodal information via CNNs and feature concatenation, while MetaBEV [8] mitigates cross-modal feature misalignment by introducing meta-BEV queries. However, despite its widespread adoption in multimodal 3D perception, the BEV representation inherently suffers from limitations in information expression.

BEV directly discretizes and quantizes data, leading to inevitable information loss. During feature extraction, perception data are projected onto a fixed-resolution BEV grid, which compresses spatial information. This issue becomes particularly severe when the BEV resolution is low, as it directly impacts model performance by failing to adequately preserve fine-grained scene structures. While increasing the BEV resolution will bring unacceptable computational overhead, as shown in Table 1. Additionally, BEV fusion strategies often rely on simple feature concatenation or weighted summation, which are insufficient for effective cross-modal feature interaction and alignment, ultimately leading to suboptimal fusion performance, as illustrated in Fig. 1 (a).

To address these challenges, we introduce a fusion approach based on 3D Gaussian Splatting (3DGS) [9] to achieve more fine-grained information modeling and more natural multimodal alignment. As shown in Fig. 1 (b), 3DGS employs continuous Gaussian distributions to represent the scene, preserving rich geometric and semantic information in the Gaussian stage and preventing the early quantization-induced information loss seen in BEV-based methods. Unlike direct BEV quantization, 3DGS aggregates information before its final projection onto the BEV grid, allowing cross-modal features to interact at a higher-dimensional level and capturing finer spatial structures prior to quantization, Table 1 shows the effectiveness of this strategy. Moreover, the covariance matrices of Gaussians enable adaptive modeling of uncertainty, enhancing the representation of object shapes and boundaries.

Specifically, inspired by [10], we propose a forward projection Gaussian initialization strategy to better initialize camera Gaussian representations in 3D space rather than using random initialization [11]. To further achieve continuous alignment and cross-modal feature enhancement, we construct a shared Gaussian encoder. The shared Gaussian encoder supports cross-feature learning of 3D Gaussian features from different modalities, where the covariance matrix of each 3D Gaussian adaptively captures feature differences between modalities and iteratively updates the Gaussian parameters. Camera and LiDAR Gaussians are naturally fused via a Gaussian mixture model, and a high-performance Gaussian-to-voxel fusion module aggregates surrounding Gaussians to generate voxel features, enabling task-agnostic 3D perception. We conduct extensive experiments on BEV object detection and 3D occupancy prediction tasks to validate the generality of the GaussianFusion. Main contributions are as follows:

• We propose the first unified 3D Gaussian representation multi-modal fusion framework, where cross-view and cross-modal Gaussian representations are naturally aggregated through the Gaussian mixture model.

• A progressive update strategy is designed to optimize the multi-modal 3D Gaussian properties iteratively.

• The shared 3D Gaussian encoder enables alignment and complementary enhancement of cross-modal features, allowing Gaussian representations from both modalities to achieve consistent uncertainty within a unified space.

• Our GaussianFusion achieves state-of-the-art benchmarks in task-agnostic methods on various 3D perception tasks within the nuScenes dataset.

2 Related work↩︎

2.1 Multimodal 3D Perception↩︎

We categorize current multimodal fusion methods into object-centric methods and dense BEV methods. Object-centric methods [4]–[6], [12]–[14] are specifically designed for tasks such as 3D object detection or tracking. Advanced object-centric [5], [6] methods typically use 2D detection results on camera to enhance multi-modal fusion 3D detection. Additionally, some works [3], [15] use query-based 3D detection decoders to learn features from perspective images and Lidar BEV features directly. However, these object-centric methods cannot easily generalize to dense semantic tasks such as BEV map segmentation and 3D occupancy prediction. Dense BEV methods [1], [7], [8], [16]–[20] naturally adapt to various tasks. Both BEVFusion [1] and UniTR[7] achieve multi-modal BEV fusion perception through CNN and feature concatenation in the BEV space. In addition, MetaBEV [8] proposed a learnable cross-attention mechanism to generate unified BEV features. BEV or 3D voxel also provides a unified representation for 3D occupancy prediction. Some methods design fusion modules [21]–[23] based on voxel, such as adaptive fusion [22], etc. for multi-modal 3D occupancy prediction. There are also many camera-only methods [24]–[33] for 3D semantic occupancy prediction based on voxel representation. However, discrete voxel representations may result in significant detail loss and hinder effective multimodal complementary fusion.

2.2 3D Gaussian Splatting↩︎

3D Gaussian Splatting (3DGS) [9] combines the advantages of implicit neural radiance fields [34] and voxel-based explicit radiance fields [35], [36] and is widely applied in 3D reconstruction of 2D image. 3DGS uses a set of Gaussian functions to capture the geometric shapes and semantic of different objects or regions within a scene, effectively representing the scene. Based on this, some works [37], [38] leverage 3DGS’s multi-view synthesis capabilities to achieve 3D scene segmentation.

Recent studies [11], [39]–[42] have applied 3DGS to vision-only 3D semantic occupancy prediction, BEV segmentation, and end-to-end autonomous driving. However, these methods rely on randomly initialized Gaussians. For example, GaussianFormer [11] randomly initializes and re-predicts Gaussian parameters in each iteration, making fine-tuning difficult and limiting accurate scene representation. Moreover, these methods do not fully exploit the advantages of Gaussian Mixture Models (GMM) for seamless multimodal Gaussian fusion. In contrast, we propose a forward-projection-based 3DGS parameter initialization and a shared optimization model, leveraging GMM to fuse multi-view camera and LiDAR features within a shared space, ultimately enabling dense semantic understanding and object-centric multitask perception.

3 Methods↩︎

3.1 Overall Architecture↩︎

The overall pipeline of GaussianFusion is illustrated in Fig. 2, with the goal of fusing multimodal features through 3D Gaussian representations, which naturally preserve both geometric and semantic information. We first initialize separate 3D Gaussian representations for camera and Lidar, denoted as \(\mathcal{G}_{\text{c}} \sim Q_{\text{c}}\) and \(\mathcal{G}_{\text{L}}\sim Q_{\text{L}}\), within a unified space. Then, the multimodal Gaussian sets are processed through a shared Gaussian Encoder, enabling the integration of semantic and geometric information from both modalities. Finally, the learned 3D Gaussian sets \(\hat{\mathcal{G}}_{\text{c}} \sim \hat{Q}_{\text{c}}\) and \(\hat{\mathcal{G}}_{\text{L}} \sim \hat{Q}_{\text{L}}\) are fused within the unified Gaussian space and fed into task-specific heads to perform 3D perception.

3.2 Gaussian initialization↩︎

BEVFusion [1] projects multimodal features into discrete BEV space for fusion. In contrast, inspired by the Gaussian mixture model paradigm of 3DGS [9] for modeling scene geometry and semantics, we utilize two Gaussian sets, \(\mathcal{G}_{\text{c}}\) and \(\mathcal{G}_{\text{L}}\), within a shared space to represent surround-view camera and Lidar information, respectively, to achieve a seamless multimodal fusion.

Camera Gaussian Initialization with Forward Projection. The properties of every single 3D Gaussian function are defined by a mean \(\boldsymbol{\mu} \in \mathbb{R}^3\), scale \(\boldsymbol{s}\in \mathbb{R}^3\), and rotation vectors \(\boldsymbol{r} \in \mathbb{R}^4\). Given \(N\) surround camera view, each camera’s view can be represented by a set of 3D Gaussian distributions \(G_{c,i} \in \mathbb{R}^{D_g \times D_c \times H_c \times W_c} | i = 1, 2,\cdots, N\), where \(D_g = \boldsymbol{\mu} + \boldsymbol{s} + \boldsymbol{r}\), \(D_c\) denotes the number of discrete depths for camera, as shown in Fig. 2 (a), a Gaussian function is assigned to a depth point. It is worth noting that this is completely different from GausianFormer [11], which randomly initializes a set of Gaussians in space, which makes model optimization more difficult. Specifically, inspired by [1], [10], [43], [44], given surround camera input features \(F_{c,i} \in \mathbb{R}^{C \times H_c \times W_c}, i = 1, 2, \cdots, N\), where \(C\), \(H_c\) and \(W_c\) represent the channel, height, and width of the camera features. \(F_{c,i}\) is fed to LSS [10] to obtain the depth distribution \(D_i \in \mathbb{R}^{D_c \times H_c \times W_c}, i = 1, 2, \cdots, N\), \(D_i\) are then used as the initial mean \(\boldsymbol{\mu}\) of the Gaussian, which is the location of every single Gaussian center. For \(\boldsymbol{s}\) and \(\boldsymbol{r}\), we initialize them randomly. And \(\mathcal{G}_c = \{ G_{c,i} \}, \mathcal{G}_c \in \mathbb{R}^{N \times D_g \times D_c \times H_c \times W_c}\).

The image features \(F_{c,i}\) are processed through a context network composed of multiple convolutional layers to obtain the semantic features \(F_{c,i}^{{\prime}}\). Next, an inner product is computed between \(D_i\) and \(F_{c,i}^{{\prime}}\) to derive the features at each depth point in 3D space, denoted as initial query features \(Q_c = \{Q_{c,i} \in \mathbb{R}^{C \times D_c \times H_c \times W_c}| i=1, 2, \cdots, N\}\). Then, \(Q_{c}\) are associated with Gaussian \((\mathcal{G}_{c} \sim Q_{c})\). For a given Gaussian set \((g_c\sim q_c, g_c \in \mathcal{G}_c, q_c \in Q_c)\), the feature at a point \(\boldsymbol{p} = (x, y, z)\) within its elliptical space are: \[\;g_c (\mathbf{p}; \boldsymbol{\mu},\mathbf{s},\mathbf{r}) = {\rm{exp}}\big(-\frac{1}{2}(\mathbf{p}-\boldsymbol{\mu})^T \mathbf{\Sigma}^{-1} (\mathbf{p}-\boldsymbol{\mu})\big)q_c, \label{eq:32gaussian32dist}\tag{1}\] where \(\mathbf{\Sigma} = \mathbf{R}\mathbf{S}\mathbf{S}^T\mathbf{R}^T, \quad \mathbf{S} = {\rm{diag}}(\mathbf{s})\), and \(\quad \mathbf{R} = {\rm{q2r}}(\mathbf{r})\). \(\mathbf{\Sigma}, \rm{diag}(\cdot)\), and \(\rm{q2r}(\cdot)\) represent the covariance matrix, the function that constructs a diagonal matrix from a vector, and the function that transforms a quaternion into a rotation matrix, respectively.

Lidar Gaussian initialization. Lidar’s BEV space naturally provides an initialization for the Gaussian mean \(\boldsymbol{\mu}\). And, s and r are initialized randomly. The Gaussian initialization for Lidar is formulated as \(\mathcal{G}_\text{L} \in \mathbb{R}^{C \times H_\text{L} \times W_\text{L}}\). Since directly extracting information from the massive raw lidar point clouds to construct 3D Gaussian representations is both difficult and computationally intensive, grid-based representations offer an effective approach to alleviate these challenges. For query, given the Lidar features \(B_\text{L} \in \mathbb{R}^{C \times H_\text{L} \times W_\text{L}}\), where \(C, H_\text{L}\), and \(W_\text{L}\) represent the channel, height, and width of the BEV features, respectively. Then, we fed the BEV feature \(B_\text{L}\) into a multilayer perceptron (MLP) to obtain Lidar query \(Q_\text{L}\), which is associated with the initial query features for each Gaussian function \((\mathcal{G}_\text{L} \sim Q_\text{L})\).

Each voxel in Lidar BEV grid is centered at a 3D location \(\boldsymbol{\mu} \in \mathbb{R}^3\), and a Gaussian is initialized at \(\boldsymbol{\mu}\) with learnable scale s and rotation r. For a Gaussian \((\mathcal{G}_\text{L} \sim Q_\text{L})\) associated with LiDAR feature \(Q_L \in \mathbb{R}^3\), the feature response at a 3D point \(\boldsymbol{p} = (x, y, z)\) within its support is computed as:

\[\;g_L (\mathbf{p}; \boldsymbol{\mu},\mathbf{s},\mathbf{r}) = {\rm{exp}}\big(-\frac{1}{2}(\mathbf{p}-\boldsymbol{\mu})^T \mathbf{\Sigma}^{-1} (\mathbf{p}-\boldsymbol{\mu})\big)q_L, \label{onjflzwv}\tag{2}\]

3.3 Gaussian Encoder↩︎

The 3D Gaussian distribution effectively represents the scene, and we have designed a Gaussian encoder to optimize both the properties and query features. This encoder includes a deformable attention with Gaussian module and a Gaussian Updating module. The Gaussian encoder is stacked multiple times to update the Gaussian properties in an iterative refinement paradigm. Additionally, to better fuse multi-modal information, we employ a shared Gaussian encoder to simultaneously process the Gaussian distributions \(\mathcal{G}_{\text{c}}\) and \(\mathcal{G}_{\text{L}}\), as the two modalities are ultimately intended to converge towards similar Gaussian distributions. Specifically, we merge \(\mathcal{G}_{\text{c}}\) and \(\mathcal{G}_{\text{L}}\) into the batch dimension.

Deformable Attention with Gaussian. As shown in Fig. 2 (b), after obtaining the Gaussian distribution sets and corresponding query features (\(\mathcal{G}_{\text{c}} \sim Q_{\text{c}}, \mathcal{G}_{\text{L}} \sim Q_{\text{L}}\), we first encode \(\mathcal{G}_i \sim Q_i, i={\text{c, L}},\) into a new query \(\hat{Q}\). Specifically, to capture the position and geometric information of the Gaussian functions within the feature maps, we employ an MLP to encode the Gaussian properties. The new query \(\hat{Q}\) is then obtained by adding the encoded properties \(P_q\) to the original query \(Q_i\): \[\hat{Q}_i=\text{MLP}(\mathcal{G})+Q_i, i={\text{c, L}},\] where the MLP as the position embedding (PE) transforms the dimensions from \(\mathbb{R}^{D_g}\) to \(\mathbb{R}^{C}\). We obtain sets \(\mathcal{G}_i \sim \hat{Q}_i\).

Furthermore, we adopt deformable attention with Gaussian [11], [45] to extract features, as shown in Fig.3. Vanilla deformable attention (Fig.3 (b)) initializes sampling locations with an approximately "square/kernel-like" region and learns offsets to cover the regions of interest. However, this initial distribution lacks inherent geometric priors about object shape. In contrast, our deformable attention with Gaussian (Fig.3 (a)) directly inherits and leverages the shape properties of Gaussians: by projecting the 3D Gaussian distributions onto the BEV feature map (projection in Figure 2b), we obtain a prior sampling distribution that encodes orientation, scale, and covariance structure. In other words, the initial sampling points are not uniformly spaced on a grid, but instead follow a Gaussian distribution aligned with the underlying object geometry—such as aspect ratio, orientation, and spatial uncertainty. This Gaussian prior enables better alignment of cross-modal features to the "likely object extent," thereby enhancing fusion effectiveness—a capability absent in conventional square-shaped initialization.

Evidently, the properties of the Gaussian functions effectively describe the shape of the potential objects or regions. For each Gaussian function \(g\), we calculate a set of offsets \(\Delta \boldsymbol{\mu}=(\Delta x, \Delta y, \Delta z)\) based on the covariance matrix. These offsets, combined with the mean \(\boldsymbol{\mu}\), yield the corresponding reference points \(\boldsymbol{\mu} + \Delta \boldsymbol{\mu}\). We then project the 3D reference points onto the BEV feature map, where each Gaussian query \(q_i \in Q_i\) is updated through deformable attention, expressed as: \[\operatorname{DeformAtt} (q_i,B_i)=\sum_{k=1}^{K} A_{k}\cdot W_k B_i(\boldsymbol{\mu} + \Delta \boldsymbol{\mu}),\] where \(B_i\) is the camera BEV feature \(B_\text{c}\) or Lidar BEV feature \(B_\text{L}\), \(W_k\) is the weights obtained by linear layer, \(A_{k}\) is the attention weights and \(A_{k}\in [0, 1]\), \(K\) is the number of sampling points, \(B(\boldsymbol{\mu} + \Delta \boldsymbol{\mu})\) are the sampled features.

Gaussian Updating. To update the Gaussian properties, we propose an iterative optimization strategy of predicting offsets instead of predicting a set of new Gaussian distributions as adopted in GaussianFormer [11]. In Lidar and camera fusion perception, incrementally updating the Gaussian parameters allows for better handling of the discrepancies caused by different modalities when perceiving the same object. This approach is particularly effective in handling fusion uncertainties caused by such as signal attenuation, depth prediction uncertainties, or multi-modal signal discrepancies, as demonstrated by the ablation experiments. Incremental updates across layers allow the model to gradually reduce the disparity between modalities, improving fusion accuracy. Specifically, by predicting the offsets \(\Delta \boldsymbol{\mu}\), \(\Delta \boldsymbol{s}\), and \(\Delta \boldsymbol{r}\) for the Gaussian mean \(\boldsymbol{\mu}\), scale \(\boldsymbol{s}\), and rotation \(\boldsymbol{r}\) using an MLP, we refine the Gaussian distribution without having to predict a completely new set of properties. The updated Gaussian \(\hat{\mathcal{G}}_i\) as follows: \[\hat{\mathcal{G}}_i=\text{MLP}(\hat{Q})+\mathcal{G}_i=(\Delta \boldsymbol{\mu}+\boldsymbol{\mu},\Delta \boldsymbol{s}+\boldsymbol{s},\Delta \boldsymbol{r}+\boldsymbol{r}).\]

3.4 Multi-Sensor Fusion↩︎

By the Gaussian Encoder model, we obtain the multimodal 3D Gaussian representations \(\hat{\mathcal{G}}_{\text{c}}\) and \(\hat{\mathcal{G}}_{\text{L}}\) within the shared 3D space, respectively. We merge \(\hat{\mathcal{G}}_{\text{c}}\) and \(\hat{\mathcal{G}}_{\text{L}}\) into a unified set \(\hat{\mathcal{G}}\), and we can easily fuse them. Although the 3D Gaussian distributions can effectively represent the scene, to handle the irregular distribution of Gaussian points, we need to voxelize these Gaussian distributions to achieve task-independent 3D perception.

Specifically, given the unified Gaussian sets \(\hat{\mathcal{G}} \sim \hat{Q}\), we divide the Gaussian space into a voxel grid \(H \times W\). For a non-empty voxel \(V\) that contains \(M\) Gaussian means \(\boldsymbol{\mu}\), the Gaussian set for that voxel is \(V = \{(\hat{g}_1, \hat{g}_2, \cdots, \hat{g}_M) \sim (\hat{q}_1, \hat{q}_2, \cdots, \hat{q}_M)\}\). To ensure real-time performance and the receptive field of the voxel, we use MeanVFE [46] to downsample the Gaussian within the voxel, as illustrated in Fig. 2 (c). After Gaussian pooling, each voxel contains one Gaussian distribution \((\hat{g} \sim \hat{q})\): \[\hat{g}= \frac{1}{M}[\sum \boldsymbol{\mu}_m ,\sum \boldsymbol{s}_m ,\sum \boldsymbol{r}_m ], \hat{q} = \frac{1}{M} \sum \hat{q}_m .\]

Furthermore, the Gaussian mixture model [9] can naturally aggregate multiple Gaussian distributions into a finer-grained distribution, unifying multi-modal Gaussian representations and elegantly capturing the complexity of autonomous driving scenes. Thus, if the total number of Gaussian distributions covering point \(\boldsymbol{p}\) in the entire scene is \(J\), the feature \(f(\mathbf{p})\) at point \(\mathbf{p}\) is composed of the cumulative contributions of each individual Gaussian: \[f(\mathbf{p}) = \sum_{i=1}^{J} \hat{g}_i(\mathbf{p}; \boldsymbol{\mu},\mathbf{s},\mathbf{r}) \hat{q}_i. \label{eq:gaussiandist}\tag{3}\]

Since each voxel may be associated with multiple 3D Gaussian distributions, following the strategy in [11], we calculate the neighborhood radius based on the scale property of each Gaussian. The indices of the Gaussians and the voxels within their neighborhood are paired as tuples and appended to a list. This list is then sorted by voxel indices, determining which 3D Gaussians each voxel should focus on. Furthermore, for each voxel using 3 , we can get the fused feature \(B_\text{F}\). Finally, a simple convolutional network is used to further optimize \(B_\text{F}\).

3.5 Perception Task Setup↩︎

Without loss of generality, we follow BEVFusion [1], GaussianFusion can be applied to most 3D perception tasks based on \(B_\text{F}\). We evaluate the performance of GaussianFusion on 3D object detection and 3D semantic occupancy prediction tasks. We adopt the same Transformer-based detection head as BEVFusion [1], [2] and the occupancy head consistent with BEVDet [44]. We also constructed a camera-only version, GaussianFusion-C, containing only the camera branch shown in Fig. 2.

4 Experiments↩︎

4.1 Dataset↩︎

The nuScenes dataset [47] provides annotation data for tasks such as semantic segmentation, object detection, and 3D occupancy (Occ) prediction. It is a large-scale multimodal dataset officially split into 700/150/150 scenes for training, validation, and testing, respectively. Each scene includes annotated Lidar point cloud data captured by a 32-beam scanner, along with 6 perspective camera views, offering comprehensive 360-degree coverage at each timestamp. We evaluate our method on the 3D object detection and Occ. In our task, we down-sample the input camera images to \(704\times256\) and voxelize the point cloud to 0.075m for detection, following BEVFusion [1] and UniTR [7]. For 3D object detection, the perception range of the point cloud is set to \([-51.2m, 51.2m]\) along the \(X\) and \(Y\) axes, and \([-5m, 3m]\) along the \(Z\) axis. For 3D Occ, we evaluate within the region of \([-50m, 50m] \times [-50m, 50m]\) around the ego vehicle, following [1], [7], [48].

4.2 Implementation Details↩︎

We adopt VoxelNet [46] and Swin-T [49] as the Lidar and camera backbone to extract the multimodal features following BEVFusion [1]. The dimensions of Lidar features, image features, and 3D Gaussian query features are all set to 128. Depth of image \(D_c\)=41. Image feature \(F_{c,i}\) dimensions: 8\(\times\)​22\(\times\)​6. We set Gaussian Encoder blocks to 4, see the Appendix for experiments. The BEV size \(H \times W\) is set to \(200 \times 200\).

During training, we follow BEVFusion to adapt the aligned multimodal data augmentation strategy and the class-balanced sampling strategy from CBGS [50]. GaussianFusion is trained on 8 NVIDIA A800 GPUs. We use AdamW [51] optimizer with a weight decay 0.01. We adopt the one-cycle learning rate policy [52] with a maximum learning rate of \(2e^{-4}\). Both BEV object detection and 3D semantic occupancy prediction are trained for 20 epochs, following the same settings as BEVFusion and GaussianFormer [11], respectively.

4.3 3D Object Detection↩︎

Setting. We utilize the official evaluation metric nuScenes Detection Score (NDS) and mean Average Precision (mAP) for 3D detection.

Results. To highlight the effect of Gaussian representation, we only compare the BEV-based method. As shown in Table 2, GaussianFusion achieves SOTA results compared to previous discrete BEV representation multimodal fusion methods[1], [7], [8], [56] on nuScenes dataset, achieving 74.0 NDS and 71.7 mAP on the val split. Specifically, compared with BEVFusion [1], our GaussianFusion achieves +2.6 NDS and +3.2 mAP on val split by exploring a more natural continuous Gaussian cross-modal complementary fusion. In addition, compared with recent SOTA fusion works, such as UniTR [7], EA-LSS [56], and FusionFormer-S [55], GaussianFusion shows superior performance, outperforming them by 1.2, 0.5, and 1.7 respectively in mAP.

r8cm

1mm

Additionally, we provide a comparison with other open-source state-of-the-art (SOTA) methods in inference latency and performance accuracy in Table 3. Benefiting from the unified architecture, it achieves an excellent performance of 71.7 mAP while maintaining lower inference latency (132 ms) and memory consumption (4271 MB) compared to BEVFusion.

r8cm

Here, we design a simple temporal extension, termed GaussianFusion-T: historical Gaussian representations are warped to the current timestamp and then fused via Equations (5) and (6). In Equation (5), the mean \(\mu_m\), scale \(\textit{s}_m\), rotation \(\textit{r}_m\), and query \(q^m\) encompass not only the multi-modal (image and LiDAR) Gaussians and queries, but also those from history frames. Experimental results show that, compared to BEVFusion4D [1], our temporal variant GaussianFusion-T achieves significant improvements. Moreover, even without sophisticated temporal modeling, GaussianFusion-T achieves competitive NDS against advanced temporal fusion methods such as SparseLIF-T [57].

Waymo Open Dataset Result. We further conduct experiments on the Waymo Open Dataset [59] to evaluate the generalization capability of our approach. GaussianFusion adopts the same heatmap-based detection head as BEVFusion for a fair comparison. Table 6 shows that the Gaussian representation outperforms the BEV representation, achieving higher detection accuracy, which demonstrates its greater potential for 3D perception.

4.4 3D Semantic Occupancy Prediction↩︎

Setting. We report the Intersection-over-Union (IoU) of occupied voxels as the evaluation metric of the class-agnostic scene completion task and the mIoU of all semantic classes for the Occ task following SurroundOcc [48].

r8cm

Results. As shown in Table 5, our GaussianFusion achieves SOTA performance at 28.65 mIoU among all single-frame models. GaussianFusion outperforms the multi-modal SOTA method OccFusion [23], which is based on multi-scale voxel fusion, by +1.11 mIoU and significantly surpasses camera-only methods [26], [48], [58]. More importantly, benefiting from our proposed Gaussian initialization strategy and iterative update mechanism, GaussianFusion-C achieves a 1.55 mIoU improvement and nearly 4.5× computational efficiency compared to GaussianFormer, while using only 30% of the Gaussians, as shown in Table 4. GaussianFormer randomly initializes a set of Gaussians in 3D space and predicts new Gaussian parameters for these Gaussians in an update. Extensive experiments demonstrate the effectiveness of our 3D Gaussian representation across multiple tasks, including both object-centric and dense semantic perception.

4.5 Ablation Studies↩︎

Effect of Gaussian Initialization. Table ¿tbl:tab:Initialization? provides a detailed analysis of the performance impact of different Gaussian initialization strategies, including the classic random initialization, our proposed forward projection strategy, the backward projection BEVFormer-based strategy[26], [53], and the strategy of projecting Lidar points onto the image. The initialization details and corresponding Gaussian encoder for the latter two projection strategies are in Appendix. The forward projection and Lidar projection strategies show comparable performance (74.0 NDS \(v.s\) 73.6 NDS), both outperforming the backward projection method (72.4 NDS). Notably, the forward projection strategy brings a significant improvement of +2.8 NDS over random initialization.

Effect of Gaussian Encoder. In Table [tbl:tab:Gaussian95Encoder], we first compare the shared and separate Gaussian Encoders. We find that the shared Gaussian Encoder provides a performance improvement of +0.7 mAP. We attribute this to the unified Gaussian space, which helps the model learn uncertain cross-modal complementary features. And the sharing strategy makes the model leaner. We then conduct an ablation study on the deformable attention module. Results show that deformable attention with Gaussian priors outperforms the vanilla variant by +0.4 NDS, demonstrating that the shape prior encoded by Gaussians facilitates model convergence and enhances detection accuracy. For the deformable attention-based query updating module, the results show that encoding the Gaussian properties as PE into the query leads to a gain of +0.5 mAP. For the Gaussian updating module, predicting the offsets of the properties, rather than the properties themselves, improves multimodal fusion perception by +0.9 mAP. This validates the effectiveness of the Gaussian encoder.

4.6 Visualizations↩︎

As shown in Fig. 4, in BEV object detection, compared to previous BEV-based SOTA methods like UniTR [7] and BEVFusion[1], GaussianFusion achieve higher accuracy for distant or small objects (white marks). Furthermore, UniTR exhibits significant object yaw errors (yellow marks). For Occ, GaussianFusion-C produces sharper object boundaries (red marks) and better class separation (yellow marks) compared to GaussianFormer. See Appendix for more visualizations.

4.7 Limitations↩︎

Several approaches—covering both detection [60] and Occ [61]—employ carefully designed temporal fusion modules to enhance performance. While our method naturally extends to multi-frame settings through simple temporal alignment and already achieves performance comparable to such multi-frame methods, this is likely suboptimal. A promising direction for future work is to explore motion-aware Gaussian updates, for instance by predicting velocity-guided offsets, enabling more coherent 4D scene modeling over time.

5 Conclusion↩︎

We present GaussianFusion, a novel multi-modal fusion perception framework grounded in a unified 3D Gaussian representation that seamlessly integrates camera and LiDAR features in a continuous spatial domain, effectively preserving fine-grained scene details. A shared Gaussian encoder is introduced to facilitate adaptive cross-modal interaction and alignment, with Gaussian properties iteratively refined through optimization. To support task-agnostic applications, we design an efficient Gaussian-to-voxel transformation module incorporating Gaussian pooling and aggregation mechanisms. Extensive experiments across multiple 3D perception tasks on the nuScenes dataset validate the effectiveness of GaussianFusion, achieving state-of-the-art performance among task-agnostic baselines. Although it may slightly lag behind certain task-specialized approaches, our work represents a meaningful step toward generalizable and principled multi-modal fusion. We believe that the proposed Gaussian representation paradigm offers a promising direction for future research in multimodal 3D perception.

References↩︎