1 Introduction↩︎

Histopathology remains the gold standard for diagnosis and grading because of its ability to distinguish benign from malignant tissue [1]. Learning-based algorithms now enable high-throughput analysis of whole-slide images, supporting automatic annotation and AI-assisted reporting with user-defined quantitative metrics that reduce pathologists’ workload [2], [3]. However, model accuracy depends critically on the quantity and quality of labeled training data, which is expensive and time-consuming to obtain in routine clinical workflows. Synthetic data generation therefore offers a practical solution to alleviate annotation scarcity. While generative adversarial networks (GANs) can learn with weak or unpaired supervision, they often introduce artifacts that compromise diagnostic realism [4], [5]. Diffusion models typically achieve higher fidelity and better conditional control, but most existing approaches require paired supervision for training [6], [7]. Cycle-diffusion methods attempt unpaired translation but unfortunately suffer from noisy-clean domain mismatch, substantially increasing computational cost [8], [9]. Furthermore, standard Gaussian initialization inevitably induces structural misalignment during generation [10], [11].

In this work, we propose CHIS, a training-free framework that guides pretrained diffusion models to generate structure-aligned histopathology images (Fig. 1). Given a diffusion model trained solely on unlabeled histopathology images, our CHISfirst generates a prior mask to regulate the spatial location of generated cells. For the starting point of the reverse sampling process, we leverage the insight that phase information primarily encodes structural contours and refine the initialization by fusing the phase of the prior mask with the magnitude of Gaussian noise in the frequency domain. Subsequently, fine-grained textures progressively interact with the dynamic sampling states through adaptive aggregation of wavelet components. The enhanced diffusion model ultimately delivers synthesized data that are not only stylistically consistent but also structurally controllable. Comprehensive evaluations demonstrate the superiority of CHISin image quality, mask-structure fidelity, and downstream segmentation performance.

2 Method↩︎

In this section, we first introduce the pretrained diffusion model that serves as the backbone of CHIS. We then present an efficient framework for generating prior masks. In the following two subsections, we sequentially describe our strategies for refining the initialization state and guiding the reverse diffusion sampling process.

2.1 Pretrained Diffusion Model with Histopathology Images↩︎

We select the Latent Diffusion Model (LDM) [12] as our generative backbone, which shifts the learning of complex image distributions from the high-dimensional pixel space \(\mathbf{X}\) to a compressed latent space \(\mathbf{Z}\). Given a histopathology image \(x \in \mathbf{X}\), a pretrained VAE encoder \(\mathcal{E}\) maps it to a latent representation \(z_0 = \mathcal{E}(x)\). The forward diffusion process progressively adds Gaussian noise to \(z_0\) over \(T\) timesteps. For any timestep \(t \in [0, T]\), the noisy latent \(z_t\) is defined as: \[z_t = \sqrt{\bar{\alpha}_t} z_0 + \sqrt{1-\bar{\alpha}_t} \epsilon, \quad \epsilon \sim \mathcal{N}(0, \mathbf{I}) \label{eq:forward}\tag{1}\] where \(\bar{\alpha}_t\) follows a predefined noise schedule. The reverse denoising process follows the DDIM [13] formulation: \[z_{t-1} = \sqrt{\bar{\alpha}_{t-1}} \left(\frac{z_t - \sqrt{1-\bar{\alpha}_t}\,\epsilon_\theta(z_t,t)}{\sqrt{\bar{\alpha}_t}}\right) + \sqrt{1-\bar{\alpha}_{t-1}}\,\epsilon_\theta(z_t,t) \label{eq:reverse}\tag{2}\] where the denoising network \(\epsilon_\theta(z_t, t)\) predicts the noise added to \(z_t\), allowing deterministic reconstruction of \(\hat{z}_0\) from \(z_T \sim \mathcal{N}(0, \mathbf{I})\) step-by-step. The learnable model \(\epsilon_\theta\) is pretrained solely on unlabeled histopathology images and remains frozen in our proposed CHIS. Finally, the VAE decoder \(\mathcal{D}\) produces synthesized images by \(\hat{x}_0 = \mathcal{D}(\hat{z}_0)\). In the following sections, we will describe how CHISensures the spatial distribution of cells in \(\hat{x}_0\), starting from a random state \(z_T\), can be precisely controlled by masks \(y\in\mathbf{Y}\) generated in Section 2.2.

2.2 Generation of Histopathology Mask with Cell Aggregation↩︎

To facilitate the generation of histopathology masks, we propose an efficient framework for aggregating single-cell masks with geometric priors ( in Fig. 1). Given a collection of cell masks, the synthesis of a histopathology mask \(y\) can be formulated as an aggregation process constrained by three category-specific metrics: cell count, mask area, and inter-cell distance, denoted as \(\{count, area, dist\}\). These metrics can be either derived from the mask collection or specified directly by pathologists.

To aggregate cell masks into a histopathology mask, we partition \(count\) cells into two groups: clustered cells and isolated cells. We first generate cell clusters by sequentially sampling cells from the collection and positioning them together within the specified \(dist\) threshold. Each cell cluster is then treated as a super-cell and combined with isolated cells from the collection. The distances between all cells and super-cells are maintained above the preset \(dist\) threshold. Additionally, the size of each cell fluctuates around the geometric prior \(area\). This generated mask benefits downstream segmentation models through densely distributed clustered cells while preserving mask fidelity through the inclusion of isolated cells.

2.3 Structural Initialization with Spectral Decomposition↩︎

During the reverse sampling process, the initial state \(z_T\) largely determines the content of the denoised results. Rather than randomly sampling \(z_T\), we incorporate structural features from the mask \(y\)—specifically, the contours of cell masks—into \(z_T\) by decomposing \(y\) using the Fast Fourier Transform (FFT). As described in [14], structural information is primarily preserved in the phase component. We therefore refine \(z_T\) through structure-aligned phase fusion between the mask \(y\) and Gaussian noise \(z_T\) ( in Fig. 1).

For the latent diffusion model, the encoder \(\mathcal{E}\) transforms the input into a latent feature map. Although \(\mathcal{E}\) can directly accept the mask \(y\), we observe a significant distribution shift in \(\mathcal{E}(y)\) when a binary mask \(y\) differs substantially from the pretraining data (i.e., histopathology images). To address this issue, we apply an extract-and-fill procedure to roughly colorize the regions of \(y\). Specifically, we first select a reference histopathology image \(x^\text{ref}\) and apply Otsu thresholding to obtain its coarse segmentation in the optical density domain. The background regions of \(y\) are then filled with the average color of the corresponding regions in \(x^\text{ref}\). To enhance the textural features of foreground cells in \(y\), pixels are colored according to the average color of pixels at the same boundary distance in \(x^\text{ref}\). Finally, we obtain \(\hat{y}\) with a color style similar to that of \(x^\text{ref}\).

Based on the refined input \(\hat{y}\), we obtain the latent feature \(\hat{z}^\text{ref}_0=\mathcal{E}(\hat{y})\). The FFT operation \(\mathcal{F}\) is applied to \(\hat{z}^\text{ref}_0\) and \(z_T\): \[\mathcal{F}(\hat{z}^\text{ref}_0)=A^\text{ref}\exp(j\phi^\text{ref}),\quad\mathcal{F}(z_T)=A^\text{noise}\exp(j\phi^\text{noise})\] where \(A^{\cdot}\) and \(\phi^{\cdot}\) denote the amplitude and phase parts, respectively. These two parts are subsequently mixed with \[A^\text{mix}=A^\text{noise},\quad\phi^\text{mix}=\Theta(r)\odot\phi^\text{ref} +(1-\Theta(r))\odot\phi^\text{noise}\] where \(\Theta(r)\) is a low-pass filter that keeps phase only within a cutoff radius \(r\). The structure-aligned state comes with inverse FFT \(z^\text{mix}_T=\mathcal{F}^{-1}(A^\text{mix}\exp(j\phi^\text{mix}))\). With different \(r\), we can control the degree of structural preservation regarding mask \(y\). Additionally, the style of \(z^\text{mix}_T\) can be easily changed by selecting another \(x^\text{ref}\) even without any annotations.

2.4 Adaptive Textural Modulation with Wavelet Fusion↩︎

Beyond structural alignment, textural similarity also contributes to the fidelity of generated images and further reinforces structural consistency. Here we propose wavelet-based guidance to progressively enhance textural details in \(z^\text{mix}_t\) and improve structure–texture coherence during the reverse diffusion process ( in Fig. 1). We employ the stationary wavelet transform (SWT) \(\mathcal{S}\) to decompose both the evolving latent \(z^\text{mix}_t\) and the prior guidance \(\hat{z}^\text{ref}_t\) into coarse- and fine-grained texture components, yielding \[\mathcal{S}(z^\text{mix}_t)=\{L^\text{mix}_t, H^\text{mix}_t\},\quad \mathcal{S}(\hat{z}^\text{ref}_t)=\{L^\text{ref}_t, H^\text{ref}_t\}\] where the coarse texture \(L^\cdot_t=[LL^\cdot_t]\) and fine-grained texture \(H^\cdot_t=[LH^\cdot_t\mid HL^\cdot_t\mid HH^\cdot_t]\) correspond to low- and high-frequency components in the frequency domain, respectively [15].

Since the coarse texture \(L^\text{mix}_t\) depicts the overall contours in \(z^\text{mix}_t\), we aim to align it with \(L^\text{ref}_t\). Hence, the distribution of \(L^\text{mix}_t\) is shifted using the mean \(\mu(\cdot)\) and standard deviation \(\sigma(\cdot)\) via: \[\tilde{L}^\text{mix}_t = \sigma(L^\text{mix}_t)\,\frac{L^\text{ref}_t - \mu(L^\text{ref}_t)}{\sigma(L^\text{ref}_t)} + \mu(L^\text{mix}_t), \label{eq:adain}\tag{3}\] Then we modulate foreground and background textures based on three rules: i) Relaxed constraints on background textures; ii) Primary reliance on \(\hat{z}^\text{ref}_t\) for low-frequency contour textures; iii) Reduced guidance on detailed high-frequency textures. Three hyperparameters \(\gamma\), \(\chi\) and \(\zeta\) were set accordingly to control the degree of texture modulation. Thereafter, the final weights for modulating low- and high-frequency components are given by \(\omega_L=\delta_{\{y,1\}}\chi+ \delta_{\{y,0\}}\gamma\) and \(\omega_H = \delta_{\{y,1\}}\zeta+\delta_{\{y,0\}}\gamma\), where \(\delta\) represents the Kronecker delta function. These two weights guide the modulation with weighted summation \[\tilde{L}^\text{mix}_t = \omega_L \cdot\hat{L}^\text{mix}_t + (1-\omega_L) \cdot L^\text{mix}_t, \quad \tilde{H}^\text{mix}_t = \omega_H \cdot H^\text{ref}_t + (1-\omega_H) \cdot H^\text{mix}_t, \label{eq:blend}\tag{4}\] Finally, the inverse SWT transforms the components \(\tilde{L}^\text{mix}_t\) and \(\tilde{H}^\text{mix}_t\) back into the spatial domain \(z^\text{mix}_t\), status for the next sampling step. This process repeats during the early sampling stage, until the textures are well consolidated by \(\hat{z}^\text{ref}_t\).

3 Experiments↩︎

3.1 Implementation Details↩︎

While CHISis framework-agnostic across diffusion-based models, we instantiate it on PixCell [16], a backbone pretrained on large-scale histopathology images that synthesizes realistic tissue but leaves cellular structure uncontrolled. We keep PixCell frozen, preserving all parameters and its foundation-feature conditioning; CHISacts only on the sampling trajectory: it replaces the initial noise at \(t{=}T\) with a phase-fused latent, applies textural guidance over the first 40 steps with \(\gamma=0.2\), \(\chi=0.95\), and \(\zeta=0.6\), and lets PixCell run freely in the final 10 steps.

We evaluate CHISon three histopathology datasets: MoNuSAC [17], Kumar [18], and PanNuke [19], with an 80/20 split for training/test data. For comparison, we select various baseline methods representing different approaches, including fully supervised models (NuDiff [7], SDM [6]), GAN-based models (SynDiff [20], CycleDiff [9]), and training-free models (UGDM [21], ADMMDiff [22]). For all methods, the number of synthesized images matches that of the training split in each dataset. The synthesized images are assessed from three perspectives: (i) evaluate their alignment to prior masks using a binary segmentation model; (ii) investigate their benefits to downstream instance segmentation model; (iii) measure their fidelity compared to real datasets. Detailed experimental settings are introduced in the corresponding sections.

3.2 Alignment Between Prior Mask and Synthesized Image↩︎

Inspired by [10], we adapt zero-shot nnU-Net [25] to segment the synthesized images and then evaluate the consistency between the segmentation results and the prior masks. As shown in Table [tab:fs195hd95], our CHISachieves the best performance in preserving the structural information from the guidance masks. Although GAN-based models excel at style transformation with unpaired data, both SynDiff and CycleDiff fail to regulate mask-to-image correspondence. To refine structural details, UGDM and ADMMDiff steer the sampling trajectory by anchoring the reverse diffusion states. However, their inability to effectively modulate coarse- and fine-grained structures leads to performance degradation. Our CHIScarefully initializes the starting point with structural information and progressively modulates the dynamic states at different frequency components. The performance of our CHISapproaches that of the fully supervised model NuDiff and even exceeds it on the MoNuSAC dataset.

3.3 Improvement on Downstream Segmentation Task↩︎

To demonstrate the benefits of CHISfor downstream segmentation tasks, we augment the training dataset with synthesized images, doubling the size of each dataset. The performance of Hover-Net [26] is then compared before and after dataset expansion. In addition to the methods mentioned in Section 3.2, we also include comparisons with on-the-fly augmentation techniques CutOut [23] and CutMix [24]. As shown in Table [tab:dice95aji], CHISachieves the highest Dice and AJI scores among all competitive methods. Compared to the baseline, the segmentation performance gain from our framework is substantial, especially considering that it eliminates the need for data annotation and model training. Moreover, we observe that CHIS’s performance approaches that of fully supervised models SDM and NuDiff, which require substantial annotation and computational resources. The performance difference of NuDiff between Table [tab:fs195hd95] and Table [tab:dice95aji] indicates that synthesized images require not only overall contour alignment but also similarity in textural features. Only by achieving both aspects can we avoid introducing distribution shifts during model testing. This finding indirectly validates CHIS’s ability to balance both global structural integration and local feature preservation.

3.4 Image Quality Assessment of Synthesized Data↩︎

In this section, we compare the synthesized data directly with the real image dataset, where similarity is quantified using the image quality metrics FID [27] and IS [28]. When a style reference image is required for UGDM, ADMMDiff, and CHIS, we randomly select one from the corresponding training set. Quantitative results in Table [tab:fid95is] demonstrate the advantage of CHISin generating high-quality images with respect to visual perception. The improved performance of UGDM, ADMMDiff, and CHISshows that incorporating a reference during the reverse sampling process effectively regulates the style of generated content. In contrast, both supervised and GAN-based models focus on learning general distribution features from heterogeneous data, which inevitably leads to synthetic data exhibiting artifacts with mixed styles. In Fig. 2, we present three example results with their corresponding style references and prior masks. Our CHISachieves superior performance in terms of both cell distribution and style preservation.

4 Conclusion↩︎

In this work, we explore the feasibility of a controllable framework for histopathology image synthesis, addressing the critical challenge of limited data resources in clinical applications. Equipped with our training-free structural initialization and textural modulation modules, pretrained diffusion models can now achieve high mask-to-image faithfulness without requiring labor-intensive data annotation or substantial computational overhead. Experimental results demonstrate that our proposed CHISnot only rivals competitive works in generation quality, but also impressively improves downstream instance segmentation performance. As an efficient and scalable solution that bridges the gap between zero-shot efficiency and domain-specific precision, CHIShas the potential to advance various downstream applications, such as virtual staining and pathologist training.

References↩︎