Bi-PT: Bidirectional Cross-Attention Point Transformers for Four-Chamber Heart Reconstruction from Sparse Cardiac MRI Data
July 08, 2026
Abstract
We propose Bi-PT, a pipeline for reconstructing 3D four-chamber human heart meshes from clinical sparsely sampled cardiac magnetic resonance imaging (CMR) data. This work addresses the error-prone generation of 3D cardiac shape from a sparse point cloud (SPC) extracted from 2D long-axis and short-axis views used in routine clinical CMR protocols. Bi-PT enables accurate inference of the four-chamber heart mesh from the SPC by learning robust point features via bidirectional point cross-attention between an atlas and the SPC, together with per-point semantic labels that improve correspondence estimation. We formulate the deformation field as a Neural Ordinary Differential Equation (NODE) parameterized by a per-point affine transformation and translation to deform the atlas toward the target heart shape. By learning such a NODE, we can guarantee the deformation field to be a locally affine diffeomorphic deformation. We also integrate a semantic label loss into the Chamfer distance to encourage label-consistent correspondences and add a smoothness regularization to stabilize and improve the learning of the deformation field. Extensive experiments demonstrate that Bi-PT achieves accurate and robust performance compared to baselines.
1 Introduction↩︎
Cardiac magnetic resonance imaging (CMR) is a gold standard for evaluating cardiac morphology and function, and supports a wide range of downstream clinical and research applications. However, CMR acquisition is time-consuming, making dense 3D imaging impractical in routine settings, due to prolonged breath-holding requirement. As a result, standard clinical protocols for CMR often rely on a combination of a few long-axis (LAX) views and a stack of short-axis (SAX) slices to capture the essential cardiac structure. While these views are clinically efficient, they provide only sparse geometric evidence in 3D. In many settings, dense and accurate 3D geometry reconstruction from these sparsely sampled routine views is needed to support image analysis tasks and quantitative measurements, such as mass and volume [1], 3D myocardial wall strain computation [2], [3], image-guided interventions [4], and biomechanics finite element simulations [5].
3D cardiac shape reconstruction from multi-view sparse 2D CMR has a long history [1], [2], [6], [7]. Traditional approaches, such as statistical shape modeling (SSM) and principal component analysis (PCA), typically start from segmentation masks and solve an iterative optimization problem to fit a template, often a mean shape, to sparse observations [8]. These methods rely on repeated registration updates and can be computationally expensive; in practice, inference accuracy can degrade when observations are sparse or noisy. More recently, learning-based methods have been explored to offer more accurate and efficient solutions for 2D-to-3D reconstruction under sparse MRI acquisitions [9]. Existing deep learning methods can generally be grouped into four families: point cloud [10]–[12], mesh-based [13], [14], shape-aware [7], [15], and volumetric models [16]. Across these families, diverse architectures including CNN [11], GCN [17] and point transformer [12] have been adopted, but limitations remain. Specifically, many methods struggle with sparse or irregular input, unstable training, and high computational cost.
In this work, we propose a bidirectional cross-attention Point Transformer (Bi-PT) for reconstructing 3D four-chamber human heart meshes from the sparse point cloud (SPC) extracted from routine clinical sparse CMR data. Our Bi-PT provides a robust feature learning from an atlas and the SPC to predict the locally affine diffeomorphic deformation between the atlas and the target shape parameterized by a Neural Ordinary Differential Equation (NODE) [18]. We further propose a semantic-aware Chamfer distance loss to ensure label-consistent correspondence learning and integrate a smoothness regularization term into the loss to enable high accuracy of four-chamber heart mesh reconstruction.
Our contributions can be summarized as follows: (1) A novel point transformer architecture that captures both local and global information through bidirectional cross-attention between an atlas and the SPC. (2) A locally affine diffeomorphic deformation (LADD) model that produces complex and large deformations and ensures topology preserving of the atlas mesh. (3) Accurate label-aware 3D reconstruction that incorporates semantic labels into the Chamfer distance loss to encourage label-consistent matching and supports chamber-level accuracy of the reconstructed four chamber heart mesh. (4) A Laplacian regularization loss to prevent irregular deformation potentially resulting from extra deformation freedom of LADD.
2 Method↩︎
As shown in Fig. 1, Bi-PT learns an atlas-to-target deformation through four stages: (1) point feature encoding through Point Transformer encoders, (2) bidirectional cross-attention for information fusion, (3) point feature decoding via Point Transformer decoders, and (4) LADD through two NODE blocks. We give details of Bi-PT as follows.
2.1 Bi-PT Encoder and Decoder↩︎
The atlas is shared across all subjects and thus carries no subject-specific evidence; the SPC is the only observation of the individual target. The encoder’s role is to give every point of the atlas or SPC a feature summarizing its local neighborhood, so that the bidirectional cross-attention (Sec. 2.2) can transfer information between subject-specific cues within the SPC and the atlas.
We represent the atlas and the SPC as point sets \(\mathbf{Q}_{a}^{0}\in\mathbb{R}^{N_0\times(3+L_a)}\) and \(\mathbf{Q}_{s}^{0}\in\mathbb{R}^{M_0\times(3+L_s)}\), each point given by its \(3\)D coordinates and a semantic label: a \(3\)-channel triplet (\(L_a=3\)) for the atlas and a \(2\)-channel pair for the SPC (\(L_s=2\)). See more details in Sec. 2.4. These label channels make the point features more discriminative than coordinates alone, sharpening the correspondences estimated downstream.
We encode each stream with an encoder of Point Transformer [19]. After a point-wise embedding, its vector self-attention aggregates every point over its \(k\) nearest neighbors, with attention weights modulated by a learned position encoding of the neighbors’ relative coordinates. The point set is progressively coarsened by \(B\) transition-down stages, each applying set abstraction [20] followed by a self-attention block, doubling the channel width per stage. Applied independently to the two streams, the encoders yields coarse token sets \(\mathbf{Q}_{a}^{1}\in\mathbb{R}^{N\times C}\) and \(\mathbf{Q}_{s}^{1}\in\mathbb{R}^{M\times C}\) (\(N=\frac{1}{256}N_0\), \(M=\frac{1}{256}M_0\), \(C\) is the channel width).
The coordinate-feature pairs at every level of the encoders are retained as a skip connection for the decoders. The decoders mirrors the encoders as a \(B\)-stage upsampling path that propagates the enriched atlas-SPC features from bottleneck tokens back to full resolution.
2.2 Bidirectional Cross-Attention Core Layer↩︎
Each encoder processes the atlas or the SPC independently with only self-attention. The resulting tokens, therefore, describe local geometry within each point set but nothing about the correspondence between them. Because the atlas must be deformed onto the shape carried by the SPC, the two token sets must be put in correspondence. We achieve this with our core contribution, a bidirectional cross-attention layer, that exchanges information between the atlas tokens \(\mathbf{Q}_{a}^{1}\) and the SPC tokens \(\mathbf{Q}_{s}^{1}\) in both directions.
Specifically, for a query token \(\mathbf{x}_{q,i}\) and key/value tokens \(\mathbf{x}_{k,j}\)/\(\mathbf{x}_{v,j}\), we linearly project \(\mathbf{q}_i = f_q(\mathbf{x}_{q,i})\), \(\mathbf{k}_j = f_k(\mathbf{x}_{k,j})\), \(\mathbf{v}_j = f_v(\mathbf{x}_{v,j})\), perform a relational encoding \(\mathbf{p}_{ij} = \mathrm{MLP}_\delta(\mathbf{x}_{q,i} - \mathbf{x}_{k,j})\), and then compute the vector cross-attention [19]: \[\alpha_{ij} = \operatorname{softmax}_{j}\!\big(\mathrm{MLP}_\gamma(\mathbf{q}_i-\mathbf{k}_j+\mathbf{p}_{ij})/\sqrt{d}\big), \label{eq:cross95attn95weight}\tag{1}\] where \(f_q\), \(f_k\) and \(f_v\) are linear layers, \(\mathrm{MLP}_\delta\) and \(\mathrm{MLP}_\gamma\) are two-layer MLPs with ReLU, and \(d\) is the channel dimension. The softmax over key index \(j\) gives each query a per-key weight distribution. Then we aggregate values globally: \[\mathbf{y}_i = \mathbf{x}_{q,i}+f_o\!\Big(\textstyle\sum_{j=1}^{N_k}\alpha_{ij}\odot(\mathbf{v}_j+\mathbf{p}_{ij})\Big), \label{eq:cross95attn95aggregation}\tag{2}\] where \(f_o\) is a linear projection and \(\odot\) is the Hadamard product. Crucially, the summation runs over all \(N_k\) key tokens rather than over a local neighborhood to ensure global information fusion.
In the \(atlas\!\to\!SPC\) direction (\(\mathbf{x}_q=\mathbf{Q}_{a}^{1}\), \(\mathbf{x}_k=\mathbf{x}_v=\mathbf{Q}_{s}^{1}\)), atlas tokens attend to the SPC to absorb local target-shape cues, yielding \(\mathbf{Q}_{a}^{2}\in\mathbb{R}^{N\times C}\); the first decoder upsamples them to the \(N_0\) atlas vertices, producing the per-point features \(\mathbf{Q}_{a}^{3}\in\mathbb{R}^{N_0\times C}\). The reverse \(SPC\!\to\!atlas\) direction (\(\mathbf{x}_q=\mathbf{Q}_{s}^{1}\), \(\mathbf{x}_k=\mathbf{x}_v=\mathbf{Q}_{a}^{1}\)) lets SPC tokens aggregate the atlas as a whole, yielding \(\mathbf{Q}_{s}^{2}\in\mathbb{R}^{M\times C}\); the second decoder upsamples them to the \(M_0\) sparse points \(\mathbf{Q}_{s}^{3}\in\mathbb{R}^{M_0\times C}\), and global average pooling yields a single global target-shape descriptor \(\mathbf{Q}_{s}^{g}\in\mathbb{R}^{1\times C}\). We broadcast and concatenate this descriptor to every atlas feature in \(\mathbf{Q}_{a}^{3}\). The result together with the atlas coordinates \(\mathbf{a}\in\mathbb{R}^{N_0 \times 3}\) give the features \(\mathbf{Q}\in\mathbb{R}^{N_0 \times (3 + 2C)}\), which condition the LADD dynamics (Sec. 2.3). This design combines local SPC cues with global target-shape context in one module, i.e., the bidirectional cross-attention layer, to improve the robustness of the learned correspondence.
2.3 Locally Affine Diffeomorphic Deformation (LADD) Dynamics↩︎
The atlas-SPC fusion features \(\mathbf{Q}\) predict a time-invariant velocity field parameterized by a NODE which gradually deforms the atlas toward a target heart shape. Unlike per-point translation-only models [12], [21], we introduce a per-point local affine transform \(\boldsymbol{A}\) together with a per-point local translation \(\boldsymbol{b}\), which increases the expressiveness of the deformation. We define the flow \(\boldsymbol{\phi}_t\) by \[\frac{d\boldsymbol{\phi}_t(\boldsymbol{p})}{dt} = \boldsymbol{A}(\boldsymbol{\phi}_t(\boldsymbol{p}))\,\boldsymbol{\phi}_t(\boldsymbol{p}) + \boldsymbol{b}(\boldsymbol{\phi}_t(\boldsymbol{p})), \quad \boldsymbol{\phi}_0(\boldsymbol{p})=\boldsymbol{p},\] where \(\boldsymbol{\phi}_t(\boldsymbol{p})\in\mathbb{R}^3\) denotes the position of an initial point \(\boldsymbol{p}\) at time \(t\), and \(\boldsymbol{A}\in\mathbb{R}^{3\times 3}\) and \(\boldsymbol{b}\in\mathbb{R}^{3}\). As in [21], we use MLPs to predict \(\boldsymbol{A}\) and \(\boldsymbol{b}\) from features \(\mathbf{Q}\). Note that in NDM [12], large deformation is modeled by global deformation parameter functions. However, it cannot guarantee topology-preserving. Here, the LADD not only enables complex and large deformation, but also preserves topology of the atlas according to the Cauchy-Lipschitz theorem [22]. Similarly as in NDM [12], we use two consecutive NODE blocks to deform the atlas into the final target heart mesh.
2.4 Semantic-Aware Chamfer Distance and Laplacian Regularization↩︎
2.4.0.1 Semantic-aware Chamfer Distance.
Plain Chamfer distance (CD) [12] across the four chambers ignores chamber identity and can match points across structures. We therefore use a semantic-aware CD (SA-CD) that restricts nearest-neighbor search by the semantic labels. As shown in Fig. 2, we label each point of the atlas/SPC by the semantics that point has. For a query with boundary-pair label \((a,b)\), \(a\neq b\), we first match against points sharing the exact pair \((a,b)\) and, if none exist, relax to points sharing one label (e.g., \((a,c)\) or \((b,c)\)); single-structure labels \((a,a)\) keep a strict match. The same hierarchy applies to the \(3\)-label variant (used in Sec. 2.1 for the atlas). This yields label-consistent correspondences (e.g., boundary points match boundary points) while staying robust when a label combination is rare. The reconstruction loss term combines plain CD and semantic-aware CD: \(\mathcal{L}_{\mathrm{CD}}=\lambda_{plain}\mathcal{L}_{\mathrm{plain}}+\lambda_{\mathrm{sa}}\mathcal{L}_{\mathrm{sa}}\), where \(\lambda_{plain}\) and \(\lambda_{\mathrm{sa}}\) are hyper-parameters. We used the \(2\)-label SA-CD in our default setting, as we found the \(3\)-label variant could degrade the shape reconstruction accuracy.
2.4.0.2 Laplacian regularization.
Since the LADD dynamics can provide extra freedom on the deformation, we add a Laplacian regularization [23] on the final mesh to promote locally coherent and smooth deformation.
2.4.0.3 Deep supervision.
We apply \(\mathcal{L}_{\mathrm{CD}}\) to the output of both NODE blocks. With \(\mathcal{L}_{\mathrm{CD}}^{1}\) and \(\mathcal{L}_{\mathrm{CD}}^{2}\) the losses after the first and second block, respectively. The full objective is \[\mathcal{L}=\lambda_{1}\mathcal{L}_{\mathrm{CD}}^{1}+\lambda_{2}\mathcal{L}_{\mathrm{CD}}^{2}+\lambda_{\mathrm{lap}}\mathcal{L}_{\mathrm{lap}}, \label{eq:total}\tag{3}\] where \(\mathcal{L}_{\mathrm{lap}}\) is the Laplacian regularization term and \(\lambda_{1}\), \(\lambda_{2}\) and \(\lambda_{lap}\) are hyper-parameters.
3 Experiments↩︎
3.1 Dataset and Metrics↩︎
We evaluated Bi-PT on 1K public cardiac CT cases [24]. We used TotalSegmentator V2 [25] to segment each case into multiple anatomical structures: right atrium (RA), left atrium (LA), right ventricle (RV), left ventricle (LV), myocardium (MYO), and major vessels. In this work, our shape reconstruction target is the multi-label four-chamber heart mesh (RA/LA/RV/LV/MYO). To mimic CMR scans, we standardized each case from the patient coordinate space to clinically meaningful cardiac coordinate space [26] and generated sparse point cloud observations with semantic labels as in [12], and finally constructed labeled ground-truth meshes from the segmentation masks for supervision and evaluation. We performed a subject-level split with 800 cases for training, 100 cases for validation and 100 cases held out for testing.
For geometric accuracy, we follow prior work [12] and report three point-based distances: (i) Chamfer distance (CD), (ii) Earth Mover’s Distance (EMD), and (iii) point-to-surface (P2F) distance. We uniformly sample a dense set of surface points from the predicted mesh and the ground-truth surface, respectively, and compute these metrics between the two sets. To evaluate mesh quality, we report: (i) normal consistency (NC), which measures agreement of surface normals, (ii) easy non-manifold face (ENF) ratio, which measures the fraction of faces involved in non-manifold configurations, and (iii) self-intersection (SI) ratio, which measures the fraction of faces involved in triangle-triangle intersections. Note that, we evaluate the shape reconstruction accuracy of individual structure and report the mean result over the five heart structures. All quantitative evaluations were reported in the original image space.
3.2 Implementation Details↩︎
We implemented Bi-PT in PyTorch and trained all models on NVIDIA A100 GPUs (80 GB) with a batch size of \(5\). We optimized using AdamW (learning rate \(5\times10^{-4}\), weight decay \(1\times10^{-4}\)) for 300 epochs with a cosine annealing schedule, and applied gradient clipping with a maximum norm of \(0.1\). We set \(\lambda_{plain}=0.5\), \(\lambda_{sa}=0.5\), \(\lambda_{1}=0.3\), \(\lambda_{2}=0.7\) and \(\lambda_{\mathrm{lap}}=1.0\) empirically. We randomly selected a sample heart mesh from the training dataset as the atlas and set \(N_0=5632\), \(M_0=5632\), \(C=512\), \(d=512\).
3.3 Baselines↩︎
We compared Bi-PT with one traditional iterative method: Coherent Point Drift (CPD) [8] and five learning-based baselines: Neural Mesh Flow (NMF) [21], MR-Net [27], Neural Deformable Model (NDM) [12], Label Transformer Network (LTN) and LTN-DSTN [26]. We used the authors’ public implementations and matched their recommended training settings as closely as possible.
Figure 3: No caption. a — Qualitative comparison of four-chamber whole-heart reconstruction on two representative test cases. For each method, we overlay the predicted mesh on the input SPC; color encodes the per-vertex Euclidean distance to the ground truth, where brighter denotes larger error.
3.4 Results↩︎
| Method | CD \(\downarrow\) | EMD \(\downarrow\) | P2F \(\downarrow\) | NC \(\uparrow\) | ENF \(\downarrow\) | SI(\(\times10^{-5}\)) \(\downarrow\) |
|---|---|---|---|---|---|---|
| CPD [8] | 5.16(1.96) | 12.74(4.99) | 4.56(2.06) | 0.58(0.05) | 0.34(0.00) | 0(0) |
| NMF [21] | 2.99(0.48) | 13.28(2.49) | 2.19(0.56) | 0.57(0.04) | 0.34(0.00) | 0(0) |
| MR-Net [27] | 4.90(2.00) | 12.21(4.96) | 4.28(2.10) | 0.58(0.07) | 0.34(0.00) | 79.4(355) |
| NDM [12] | 3.01(0.47) | 12.51(1.61) | 2.07(0.55) | 0.04(0.03) | 0.01(0.00) | 868.6(967.7) |
| LTN [26] | 2.34(0.36) | 6.36(1.15) | 1.41(0.40) | 0.61(0.04) | 0.35(0.03) | 0.56(3.77) |
| LTN-DSTN [26] | 2.62(0.43) | 6.96(1.18) | 1.77(0.51) | 0.60(0.04) | 0.36(0.03) | 1.17(5.85) |
| Bi-PT (Ours) | 2.28(0.27)\(^\dagger\) | 6.54(1.02) | 1.41(0.34)\(^\dagger\) | 0.66(0.02)\(^\dagger\) | 0.34(0.00) | 0(0)\(^\dagger\) |
0.8pt
In Tab. 1, we report the mean quantitative results for four-chamber reconstruction on the test set with the evaluation metrics in Sec. 3.1. Bi-PT attains the best geometric accuracy while preserving mesh validity. It achieves the lowest CD and P2F distance error, both significant over the second-best method by Wilcoxon test. In terms of NC, it improves a lot compared to the second-best method, indicating a smoother, locally oriented surface. Moreover, Bi-PT produces zero SI, showing that our method retains the atlas-level topology without sacrificing the accuracy. Fig. 3 (a) illustrates examples of the four-chamber reconstruction results. CPD registers the atlas to the SPC purely by minimizing the point-to-point distance, overlaying the atlas as tightly as possible onto the observed points with no awareness of anatomical semantics. NMF, being atlas-based, returns overly smooth heart shape reconstruction results with large error in localized regions such as the MYO (LV epi-surface). MR-Net behaves almost identical with CPD, with large error patches over both atria and parts of the ventricles. NDM uses shape primitives instead of the atlas and cannot accurately recover shape details in the atria and RV. LTN is the strongest baseline over the bi-ventricular region and the LA but retains visible error on the RA. LTN-DSTN performs slightly worse than LTN as it requires stronger smoothness. In contrast, Bi-PT stay low-error across all four chambers which we attribute to the bidirectional cross-attention that injects local cues and global target-shape context of the SPC, the semantic-aware supervision that ties each atlas region to the correct target anatomy, and the Laplacian regularization that keeps smooth shapes.
3.5 Ablation Studies↩︎
From Tab. [tab:ablation], for the reconstruction objective, semantic-aware Chamfer consistently improves correspondence-related metrics over plain CD loss (CD 2.28 vs., EMD 6.54 vs.), indicating that semantic constraints add value beyond geometry-only matching. Bidirectional cross-attention (CA) is critical: replacing bidirectional CA with single CA (sCA), i.e., only keeping the \(atlas\!\to\!SPC\) pathway without global target-shape descriptor \(\mathbf{Q}_{s}^{g}\) injected in \(\mathbf{Q}\), leads to a large degradation across metrics (CD 3.10 vs., EMD 9.04 vs.) and introduces self-intersections, highlighting the importance of the \(SPC\!\to\!atlas\) pathway for global target-shape context. For the deformation model, our LADD dynamics improves reconstruction over a per-point translation-only parameterization (CD 2.28 vs., EMD 6.54 vs.), supporting that the additional degrees of freedom help express complex cardiac shape details. Removing Laplacian regularization increases geometric errors (CD 2.48 vs., P2F 1.68 vs.) and introduces self-intersections, confirming its role in stabilizing smooth, topology-preserving deformations. Finally, extending the semantics from two to three labels in SA-CD yields no further gain (CD 2.32 vs.), so we adopt the 2-label setting throughout.
| Variant | label | S | B | A | L | CD \(\downarrow\) | EMD \(\downarrow\) | P2F \(\downarrow\) | NC \(\uparrow\) | ENF \(\downarrow\) | SI(\(\times10^{-5}\)) \(\downarrow\) |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Plain CD | 2\(l\) | \(\times\) | \(✔\) | \(✔\) | \(✔\) | 2.32(0.29) | 6.85(1.15) | 1.46(0.34) | 0.66(0.02) | 0.34(0.00) | 4.84(26.20) |
| sCA | 2\(l\) | \(✔\) | \(\times\) | \(✔\) | \(✔\) | 3.10(0.62) | 9.04(1.56) | 2.35(0.70) | 0.60(0.08) | 0.35(0.00) | 2967.20(360.50) |
| Translation | 2\(l\) | \(✔\) | \(✔\) | \(\times\) | \(✔\) | 2.32(0.30) | 6.56(1.09) | 1.46(0.36) | 0.67(0.02) | 0.34(0.00) | 0(0) |
| w/o L | 2\(l\) | \(✔\) | \(✔\) | \(✔\) | \(\times\) | 2.48(0.36) | 6.34(0.96) | 1.68(0.43) | 0.65(0.02) | 0.34(0.00) | 526.94(242.56) |
| Full | 2\(l\) | \(✔\) | \(✔\) | \(✔\) | \(✔\) | 2.28(0.27) | 6.54(1.02) | 1.41(0.34) | 0.66(0.02) | 0.34(0.00) | 0(0) |
| Full, 3-label | 3\(l\) | \(✔\) | \(✔\) | \(✔\) | \(✔\) | 2.32(0.30) | 7.05(1.17) | 1.46(0.38) | 0.65(0.02) | 0.34(0.00) | 25.60(4.15) |
2.2pt
4 Conclusion↩︎
In this work, we design Bi-PT, a novel point transformer architecture based on bidirectional cross-attention, for four-chamber heart mesh reconstruction from sparse point cloud observations. By combing Bi-PT with locally affine diffeomorphic deformation dynamics, semantic-aware chamfer distance, and Laplacian regularization, our model can ensure accurate reconstruction of patient-specific 3D heart geometry from segmentation boundary of clinical sparse CMR views.