1 Introduction↩︎

Accurate cardiac structure modeling is vital for applications like clinical diagnosis and intervention planning [1]–[3]. However, faithfully representing multi-part heart geometry from diverse input configurations remains a significant challenge.

Implicit functions offer significant flexibility and have been used across various fields. But conventional implicit-based shape representations [4]–[7] are typically tailored for single-structure modeling, neglecting the correlations among structures. Recent multi-part approaches [8]–[11] have begun to address this, but they rely on computationally expensive test-time optimization. Moreover, during test time, they require complete surface information to optimize the latent code. However, the observation is often sparse, and sometimes available only for some structures and not others. For instance, only sparse delineations of the four-chamber structure can be obtained from Long-Axis (LAX) views, while Short-Axis (SAX) MRI stacks cover the left and right ventricles, but not the atrial structures. These constraints severely limit their clinical utility.

To address these limitations, we propose VecHeart, a versatile framework for high-quality four-chamber cardiac structure reconstruction and generation given potentially incomplete data. It keeps the fidelity of the individual parts while maintaining overall consistency across the whole heart.

We start from the VecSet [12] latent representation to represent complex surfaces. Given surface points sampled from a 3D shape, VecSet uses a query vector and an encoder to turn them into a set of latent codes that are fed to a set of attention layers and then decoded into a signed distance function. VecHeart uses a similar encoder and decoder, along with a part-specific query vector. This enables us to introduce three new techniques designed to handle the heart’s complex multi-part anatomy even when the data is missing for some of them:

• Hybrid Part Transformer. We sequentially and iteratively apply intra-part attention and inter-part attention to capture the correlations between the parts, as shown in Fig. 1 (a).

• Anatomical Completion Masking. During training, we randomly mask some chambers and force VecHeart to still output the full anatomy from the incomplete data, as shown by Fig. 1 (b). This is key to handling missing data.

• Modality Alignment. To allow reconstruction not only from surface samplings but also from potentially sparse CMR slices as depicted by Fig. 1 (c), we ensure that codes obtained from such slices are similar to those obtained from denser samplings.

As a result, VecHeart provides superior representation ability and a unified solution for robust cardiac modeling across a wide spectrum of input densities, even when some structures are entirely missing. We further show that with minimal effort, we can use flow matching [13], [14] for 3D+t mesh sequence generation.

2 Method↩︎

We aim for a consistent representation of the four cardiac chambers, the Left Ventricle (LV), LV myocardium, Right Ventricle (RV), Left Atrium (LA) and Right Atrium (RA). We seek to obtain this multi-component structure from potentially noisy data from different modalities and sometimes missing altogether.

Among the many existing approaches to modeling complex 3D structures, VecSet [12] is considered one of the most powerful. However, it is designed to model individual objects as opposed to multiple anatomically related objects that must remain consistent with each other. Assigning a separate VecSet model to each part would be a simple solution. However, this would not account for their inter-structure relationships. Instead, we introduce VecHeart, which retains the encoder and decoder of VecSet but uses different latent vectors for each part along with interleaved attention layers to model inter-part relationships.

We briefly review VecSet [12] in Sec. 2.1. In Sec. 2.2 and 2.3, we introduce VecHeart, our proposed framework designed to represent multi-part cardiac structures and handle heterogeneous modalities. Training detail is in 2.4. Finally, we describe how VecHeart can be extended beyond reconstruction to 3D+t shape generation in Sec. 2.5 to showcase the proposed framework’s versatility.

2.1 Preliminary↩︎

VecSet [12] is a transformer-based 3D VAE that encodes surface into compact latent codes. Given a 3D surface \(S\), its pipeline consists of three key steps:

• Surface Sampling: Sample a point cloud \(\mathbf{P}\) of \(N\) points from surface \(\mathbf{S}\).

• Feature Encoding: \(\mathbf{P}\) is encoded into compact latent code \(\mathbf{C}\) using a mechanism analogous to Perceiver [15]. This is written as: \[\mathbf{C}=\operatorname{Enc}(\mathbf{L}, \mathbf{P})=\operatorname{Attention}(\mathbf{L}, \operatorname{PosEmb}(\mathbf{P})) \in \mathbb{R}^{D\times M},\] where \(\operatorname{Attention}(Q, K/V)\) is a standard attention block [16]. \(\mathbf{L} \in \mathbb{R}^{D\times M}\) is a learnable query set with \(M\) the set size and \(D\) the dimension.

• Geometry Decoding: Given query points \(\mathbf{Q}_{space}\) sampled from the space, SDF for \(\mathbf{Q}_{space}\) are predicted using a combination of cross/self-attention: \[SDF^{pred}(\mathbf{Q}_{space})=\operatorname{Attention}(\operatorname{PosEmb}(\mathbf{Q}_{space}), \operatorname{Attention}^L(\mathbf{C}, \mathbf{C})),\] where the superscript \(L\) denotes applying the attention layers \(L\) times.

While achieving superior representation capabilities, VecSet is restricted to modeling single object. This renders it unsuitable for directly representing multi-part structures. Assigning a separate VecSet model to each part seems intuitive. However, this will neglect inter-part correlations, which may degrade the performance when dealing with incomplete or sparse input.

2.2 Structuring Multi-Part Anatomy with VecHeart↩︎

To model individual chambers along with myocardium, we introduce the VecHeart architecture depicted by Fig. 1. It expands on VecSet by incorporating a Hybrid Part Transformer (HPT) along with an Anatomical Completion Masking (ACM) strategy. Together, they enable VecHeart to capture inter-part correlations and maintain anatomical integrity, even when the input is incomplete.

2.2.0.1 Hybrid Part Transformer (HPT).

Given a 4-chamber heart-model \(\{\mathbf{S}_p\}_{p=1}^5\) where \(\mathbf{S}_p\) denotes the surface of the \(p\)-th anatomical part, HPT first uses a shared encoder to map each surface into a part-level latent code \(\mathbf{C}_p\). To maintain class-wise distinctions while facilitating cross-part information exchange, it relies on learnable part queries \(\{\mathbf{L}_p\}_{p=1}^5\) to capture part-specific knowledge: \[\mathbf{C}_p=\operatorname{Enc}(\mathbf{L}_p, \mathbf{P}_p)=\operatorname{Attention}(\mathbf{L}_p, \operatorname{PosEmb}(\mathbf{P}_p)) \in \mathbb{R}^{D\times M}, p=1,\ldots,5 \;,\] where \(\mathbf{P}_p\) is the point cloud sampled from \(\mathbf{S}_p\).

Upon obtaining \(\{\mathbf{C}_p\}_{p=1}^5\), we facilitate information exchange across components using interleaved intra/inter-part attentions. Specifically, the intra-part attention is applied independently to each \(\mathbf{C}_p\) to capture localized features, ensuring that the local anatomical details are preserved and refined. Global inter-part attention is performed over all latent codes \(\{\mathbf{C}_p\}_{p=1}^5\) to model structural dependencies and global interactions. Thus we write \(\operatorname{IntraAttn}(\mathbf{C}_p)=\operatorname{Attention}(\mathbf{C}_p, \mathbf{C}_p)\) and \(\operatorname{InterAttn}(\mathbf{C}_p)=\operatorname{Attention}(\mathbf{C}_p, \{\mathbf{C}_p\}_{p=1}^5)\).

The SDF for each anatomical part is obtained by querying its part-level latent code through a shared decoder comprising \(L\) interleaved attention layers: \[\begin{align} SDF_p^{pred}(\mathbf{Q}_{space})&=\operatorname{Dec}(\mathbf{Q}_{space}, \mathbf{C}_p), \\ \operatorname{Dec}(\mathbf{Q}_{space}, \mathbf{C}_p)&=\operatorname{Attention}(\operatorname{PosEmb}(\mathbf{Q}_{space}), \operatorname{XAttn}^L(\mathbf{C}_p)), \nonumber \end{align}\] where \(\operatorname{XAttn}=\operatorname{InterAttn}\circ\operatorname{IntraAttn}\), and \(\circ\) denotes operation composition.

By using part-specific learnable queries and interleaved attention mechanism, HPT maintains class-wise distinction while facilitating cross-part information exchange. In the following, we detail how HPT can be extended to enable complete four-chamber reconstruction from partial observations when integrated with the Anatomical Completion Masking (ACM) strategy.

2.2.0.2 Anatomical Completion Masking (ACM).

Because data can be missing for some chambers, the ability to perform complete reconstructions from incomplete data is highly desirable. Inspired by Masked Autoencoders (MAE) [17], we introduce ACM. Specifically, we randomly mask \(K\) parts and substitute their extracted latent codes \(\mathbf{C}_p\) with the corresponding part-specific learnable queries \(\mathbf{L}_p\) as decoder inputs. Through the interleaved attention, the model is tasked with predicting the SDF for all anatomical parts. This encourages the model to infer the missing structures by leveraging both learned anatomical priors and the geometric context provided by other components. We formulate this as: \[\begin{align} \mathbf{D}_p=(1-\mathbf{M}_p)\mathbf{C}_p+\mathbf{M}_p\mathbf{L}_p \; , \quad SDF_p^{pred}(\mathbf{Q}_{space})=\operatorname{Dec}(\mathbf{Q}_{space}, \mathbf{D}_p) \; , \end{align}\] where \(\mathbf{M}_p \in \{0, 1\}\) is a binary mask. \(K\) is randomly set to \(1\) or \(2\) during training.

2.3 Modality Alignment (MA)↩︎

The architecture of Fig. 1 (b) described above expects point clouds representing the whole chamber as inputs to the encoder. For modalities such as MRI, data is often only available for a few planar slices and reconstruction from such sparse data is a challenge. We address it using latent representations learned from complete surfaces that serve as priors. To this end, we propose our MA strategy.

Let \(\{\mathbf{P}_{p, s}\}_{p=1}^5\) be points from slices corresponding to surfaces \(\{\mathbf{S}_p\}_{p=1}^5\). As shown in Fig. 1 (c), we introduce an additional slice encoder \(\operatorname{Enc}_{s}\) and learnable slice query sets \(\{\mathbf{L}_{p,s}\}_{p=1}^5\) that are used to encode the slice points \[\mathbf{C}_{p,s}=\operatorname{Enc}_{s}(\mathbf{L}_{p,s}, \mathbf{P}_{p,s}) \in \mathbb{R}^{D\times M}, p=1,\ldots,5 \; . \label{eq:encoder}\tag{1}\] \(\mathbf{C}_{p,s}\) can then be fed to the same frozen decoder as before and the weights of the encoder are learned along the other network weights as described below.

2.4 Training↩︎

The training of VecHeart occurs in two stages. First, we train our model using high-quality complete hearts and minimize the composite loss \[\begin{align} \mathcal{L}_{stageI} &=\mathcal{L}_{sdf}+\mathcal{L}_{inter},~ with \;\mathcal{L}_{sdf}=\frac{1}{P}\sum_{p}\mathbb{E}_q\left[||s_p^{pred}(q)-s_p^{gt}(q)||_1\right],\\ and \; & \mathcal{L}_{inter} =\sum_{q \in \mathcal{C}}\sum_{p \in \mathcal{N}(q)}\operatorname{tanh}(\max(-\sigma \cdot SDF_{p}^{pred}(q),0)), \nonumber \end{align}\] where \(\mathcal{C}\) is the contact regions among parts, \(\mathcal{N}(q)\) are point \(q\)’s contacting parts, and \(\sigma\) a scalar. Minimizing \(\mathcal{L}_{sdf}\) maximizes agreement with the ground-truth while minimizing \(\mathcal{L}_{inter}\) discourages intersections.

Then the model is fine-tuned using SAX and LAX views synthesized as in prior works [18]–[21] by incorporating in-plane motion within SAX stacks and spatial misalignments between SAX and LAX views. These slice displacements are modeled as a Gaussian distribution \(\mathcal{N}(0, \lambda^2\mathbf{I})\), where \(\lambda\) is sampled from a uniform distribution \(\mathcal{U}(0, 6)\) mm. Additionally, we randomly drop LAX slices at a probability \(p=0.5\), encouraging the model to predict complete chambers from SAX only slices points. We train the encoder of Eq. 1 by minimizing \[\mathcal{L}_{stageII}=\mathcal{L}_{stageI}+\lambda_{la}\mathcal{L}_{la},~~\mathcal{L}_{la}=\frac{1}{P}\sum_{p=1}^P ||\mathbf{C}_{p,s}-\mathbf{C}_{p}||_2,\] where \(\mathcal{L}_{la}\) denotes the latent alignment loss. This loss ensures the slice-based encoder inherits rich anatomical priors from the surface-based encoder. Hyperparameters \(\lambda_{la}, \sigma\) are set to \(0.001, 50\), respectively.

2.5 Extension to 3D+t Cardiac Shape Generation↩︎

We demonstrate the versatility of our framework by extending it to generate 3D+t four-chamber mesh sequences. We adopt an architecture similar to Craftsman [22] to jointly generate latent codes \(\{C_p\}\) for all anatomical parts. Inspired by CardiacFlow [18], we incorporate Periodic Gaussian Kernel (PGK) encoding, Learnable Initial Values, and Beta Sampling to enable one-step generative flow. To handle sequences of different lengths, we apply linear interpolation before using PGK encoding. The model is trained using standard \(v\)-prediction loss [14].

3 Experiments↩︎

To train and test VecHeart, we curated a high-quality cardiac mesh dataset comprising 1,060 samples, generated from expert annotations in publicly available datasets, including LAA [23], [24], WHS++ [25], and other benchmarks [26]–[29]. Each four-chamber mesh is rigidly registered to a WHS atlas [30], with a 6:2:2 train/val/test split. For reconstruction from real slices and 3D+t mesh sequence generation, we use a dataset comprises 835 CMR sequences constructed from three publicly available MRI datasets: ACDC, M&Ms, and M&Ms-2 [31]–[33]. As in earlier studies [3], [34]–[36], we register and fit a Statistic Shape Model [37], [38] to segmentation masks predicted by a publicly available method [39] to generate ground-truth cardiac meshes. We only evaluate LV myocardium, LV and RV as this dataset mainly contain SAX stacks. We use standard L2 Chamfer Distance (CD) and IoU score at resolution \(128\) to measure 3D reconstruction quality.

In the VecHeart model of Fig. 1, the decoder comprises \(L=3\) interleaved attention layers. For the part-specific learnable queries \(\mathbf{L}_p \in \mathbb{R}^{D \times M}\), the set size \(M\) is 256 and dimension \(D\) is 32. We train the model for \(1000/500\) epochs for stage I/II using Adam [40]. All experiments are conducted using one V100 GPU.

3.1 Heart Shape Modeling for Complete/Incomplete Data↩︎

In this section, we evaluate the shape modeling ability for complete/incomplete data. We use the same setup as conventional approaches [9], [10] to reconstruct the test (unknown) samples. We compare against three multi-part auto-decoding implicit methods, PartSDF [8], ImHeart [9], SDF4CHD [10], as well as with CardiacFlow [18], an auto-encoding voxel-based method. For fairness, we train CardiacFlow using the same ACM training strategy as us. For completeness, we also evaluate Ours-Sep, that is, fitting a separate VecSet model to each part.

We present qualitative results in Fig. 2 (a)(b) and report quantitative ones in Tab. 1. The gray numbers are CD and IOU for parts which no data was provided and whose shape was therefore inferred from other parts. To obtain these numbers for auto-decoding methods, we compute the loss on available components and minimize it with respect to the latent vectors.

VecHeart outperforms the baselines in both complete and missing data scenarios. ImHeart’s lower accuracy likely stems from its occupancy-based representation. While implicit methods like PartSDF deliver respectable results, they are computationally expensive (e.g., 80s vs. our 0.7s feed-forward inference). While CardiacFlow yields the second-best performance on IoU, it is subject to resolution and discretization constraints. Compared to Ours-Sep, which fail to model inter-part relationships and cannot extend to missing-input scenarios, our full model shows the necessity of modeling inter-part correlation and structural priors. If only HPT is used without the ACM strategy, VecHeart cannot produce any meaningful results. These results validate our HPT and ACM designs.

3.2 Shape Reconstruction from Sparse Slices↩︎

We now turn to sparse and noisy CMR slices. CardiacFlow adopts Label Completion U-Net [19], [41] as in the original work to recover full volumes from sparse slices. We experimented with both synthetic and real data. In the synthetic case, following CardiacFlow, we synthesized 10 different multi-view 2D slices at random motion level \(\lambda\) using the displacement strategy of Sec. 2.4 for each test sample. For real experiments, we use the SAX contours of the CMR dataset.

As shown in Fig. 2 (c)(d), VecHeart produces plausible four-chamber reconstructions from sparse slices, underscoring the practical utility and robustness of our approach. We report quantitative results for synthetic data in the top rows of Tab. 2 and for real data in the bottom. As before, the gray numbers denote results for structures which is absent from the input. Our model consistently outperforms the baselines in all configurations. We also compare against Ours-SliceOnly, which uses the same slice encoder and decoder as the full model but without Stage I training. It works less well, thus confirming the importance of the proposed Modality Alignment strategy.

Note that the results on real-world CMR data were obtained by training on synthetic data and without re-training. They can be further improved by fine-tuning on real data, as shown by the numbers corresponding to Ours-Finetune.

3.3 3D+t Cardiac Shape Generation↩︎

Now we evaluate VecHeart’s capability to generate 3D+t cardiac sequences. As in CardiacFlow, we employ the Volume Fréchet Inception Distance (vFID) and Cycle-CD metrics to assess the diversity and periodic consistency of the generated sequences [42]. Fig. 3, [fig:seq95histo] shows we achieve superior qualitative and quantitative performance, producing diverse and high-fidelity sequences. As a contrast, CardiacFlow suffers from discretization artifacts. The results imply VecHeart’s potential to be integrated with clinical metadata for more controllable and interpretable generation and become a general purpose data augmenter.

4 Conclusion↩︎

We present VecHeart, a unified framework for holistic reconstruction and generation of four-chamber cardiac anatomy. It delivers high-fidelity reconstruction across a wide spectrum of input densities ranging from dense point clouds to sparse and noisy slices. Furthermore, integrating a flow-matching scheme enables the VecHeart to generate realistic 3D+t mesh sequences.

VecHeart’s flexibility makes it possible to use partially labeled data, giving it the potential for cross-modality data fusion and robust clinical diagnostics in heterogeneous real-world scenarios. Moving forward, we aim to use VecHeart to reconstruct cardiac anatomy from more modalities, including raw MRI and US images acquired over time, within a unified framework.

References↩︎