1 Introduction↩︎

Diffusion MRI (dMRI) tractography enables non-invasive reconstruction of white matter pathways in health and disease [1], [2]. However, it uses indirect measurements of axonal orientations based on water diffusion and has millimeter-scale resolution, orders of magnitude coarser than individual axons. Anatomic tracer studies in non-human primates (NHPs) offer a gold standard for validating tractography [3], [4], as they allow direct visualization of the complete trajectory of axonal projections at single axon resolution [5], [6]. Yet reconstructing fiber pathways from tracer histology requires labor-intensive manual annotation, severely limiting the amount of tracer data available for validation studies.

Machine learning can automate this process, but progress has been constrained by labeled data scarcity and variability across tracers, injection sites, and brains. Methods for axon segmentation in electron microscopy [7], [8] or fluorescence microscopy in rodents and marmosets [9]–[11] do not generalize to macaque tracer data, which are particularly valuable for translational studies due to known human–macaque homologies [12]. Initial efforts to deploy deep learning for segmenting fiber bundles in macaque tracer data [13] demonstrated feasibility but relied on ad hoc post-processing steps to reduce false positives. Recent work employed semi-supervised pre-training to improve accuracy and generalizability to sparse bundles, while obviating the need for post-processing [14], but still required manual annotations from multiple brains.

Synthetic training data, i.e., images generated from ground-truth segmentation labels, offer a promising strategy for reducing the burden of manually labeling real data. Domain randomization, i.e., systematically varying geometry, appearance, and noise during synthesis, has proven effective in encouraging models to learn structural patterns robust to dataset-specific artifacts, and has found applications across biomedical imaging domains [15]–[17]. In this approach, a network trained on synthetic data can learn to segment real data, not because the synthetic data are realistic, but because they are very diverse. To date, this strategy has not been explored for segmenting tracer histology.

Here, we present a synthetic-data augmented framework for fiber bundle segmentation in macaque tracer histology. Our synthesis framework aims to replicate the key feature of real tracer data, where a tracer labels axons projecting from a single injection site in the gray matter, as they leave the injection site, split into bundles of varying densities, and follow different paths through the white matter to reach their destinations. We propose to leverage dMRI tractography as a geometric prior to synthesize image patches with this appearance. Importantly, we do not need the end-to-end trajectory of the tractography streamlines to be anatomically correct, as we only use them to generate image patches. We compose the streamlines with backgrounds from blockface photos, diversified via domain randomization. Combined with limited real annotated sections, this achieves strong cross-brain generalization with substantially reduced annotation effort. This paper has the following contributions: (1) A tractography-driven synthesis pipeline with controllable domain randomization. (2) A mixed real+synthetic training strategy matching state-of-the-art [14] performance with 3\(\times\) less annotated data. (3) Ablations on the amount of real data, the synthetic-to-real data ratio, and randomization strength, showing that even a single annotated section yields competitive performance. The code is publicly available at https://github.com/lincbrain/synthetic-tracer-histology-segmentation.

2 Methods↩︎

2.1 Synthetic Data Generation↩︎

We design an on-the-fly synthesis pipeline that generates realistic image–mask pairs from a single ex vivo macaque brain (M0) for which both whole-brain dMRI and co-registered blockface photographs are available. The dMRI data were acquired on a 4.7T preclinical scanner using a 3D echo-planar sequence (0.5 mm isotropic, 515 gradient directions, \(b_{\max} = 40{,}000\) s/mm\(^2\)), comprising three b-shells (\(b = 4{,}000\), \(8{,}000\), \(12{,}000\) s/mm\(^2\)) with 64, 64, and 128 gradient directions respectively, following the protocol of [18]. We used multi-shell multi-tissue constrained spherical deconvolution [19] implemented in MRtrix3 and seeded probabilistic tractography (iFOD2 algorithm, maximum bending angle \(30^\circ\), minimum FOD amplitude \(0.1\), all other parameters at default) 10,000 times per voxel, repeated for 96 voxels distributed throughout the cortex to mimic different “injection sites”, yielding anatomically diverse fiber configurations that effectively decouple the synthetic training distribution from any single brain’s anatomy.

After sectioning, blockface photographs were captured at 16–21 μm in-plane resolution and the stacked volume was registered deformably to the \(b{=}0\) dMRI volume using ANTs [20]. Let \(B\in\mathbb{R}^{W\times D\times H}\) and \(\mathcal{S}=\{s_1,\ldots,s_N\}\) denote the blockface volume and tractography streamlines transformed into blockface space, respectively. Each synthetic sample \((\hat{I},\hat{M})\) is constructed as follows.

Background. To bridge the domain gap between blockface photography and dark-field microscopy, we sample a slice index \(y^\ast \sim \mathcal{U}\{1,\ldots,D\}\) and extract the coronal section \(B_{y^\ast}(x,z)=B(x,y^\ast,z)\). We apply mild CLAHE, a power-law dark-field transform with \(\gamma \sim \mathcal{U}(0.8,1.2)\) restricted to the brain mask, and a dark-field colormap [21], to emulate the appearance of a brain section. To further bridge the domain gap, the image is processed with augmentation presets [17] sampled with weighted probability, spanning optical transforms, parametric noise models, smooth multiplicative bias fields, and Lab color perturbations. Finally, we sample a random \(P\times P\) crop (\(P=512\)).

Foreground (synthetic tracer signal). Given the slice plane \(y^\ast\), we select streamlines originating from the current “injection site” that pass within distance \(\tau\) of the plane: \[\mathcal{S}_{y^\ast} = \{s_i \in \mathcal{S} : \min_{(p_x, p_y, p_z) \in s_i} |p_y - y^\ast| \le \tau\}, \label{eq:streamline95selection}\tag{1}\] where \((p_x, p_y, p_z) \in s_i\) denotes a 3D point along streamline \(s_i\), \(p_y\) its \(y\)-coordinate (coronal axis), and \(\tau\) is an adaptive thickness parameter scaled with output resolution. Points on \(\mathcal{S}_{y^\ast}\) are densified via linear interpolation at 0.2-voxel spacing and projected onto the coronal plane. We then sample a fraction \(\rho_{\max}\sim \mathcal{U}(0.02,0.05)\) of streamlines, from the densely seeded tractography streamlines, and subsample further by \(r_{\text{viz}}\sim \mathcal{U}(0.5,0.7)\) for rendering. Each streamline is rendered as an anti-aliased curve with stochastic linewidth \(w\sim \mathcal{U}(0.5,1.5)\) px, opacity fading linearly from \(\alpha=0.4\) at \(y^\ast\) to \(0\) at distance \(\delta\), and per-streamline color jitter in the orange–yellow range to mimic tracer appearance. Optionally (\(p_{\mathrm{inj}}=0.3\)), injection-site artifacts are added by rendering short curved streamlets radiating from a random point with saturated orange hues. The background crop and rendered tracer signal are composited to form the synthetic image \(\hat{I}\).

Ground-truth mask. These binary masks mimic the appearance of fiber bundle annotations. They are generated from a dense streamline set, so that \(\hat{M}\) captures the full spatial extent of each bundle even when the synthesized tracer signal is sparse. We accumulate a density map \(D(x,z)\) by rasterizing line segments between consecutive projected streamline points (Bresenham). We then Gaussian-smooth (\(\sigma{=}2.0\) px) and threshold using an absolute count threshold \(\theta{=}0.08\): \[\hat{M}_{\text{raw}}(x,z)=\mathbf{1}\Big[(G_{\sigma} * D)(x,z) \ge \theta\Big].\] Post-processing applies per-component dilation within a disk (to expand bundles while preserving separation) and small-object removal. This decoupling, sparse fibers in \(\hat{I}\) but dense bundles in \(\hat{M}\), encourages the model to detect bundles even when only partially visible, reflecting real tracer histology.

On-the-fly generation and domain randomization. Samples are synthesized dynamically during training: each mini-batch randomly selects a seed voxel, slice index, and patch location, rendering the corresponding sample through the full pipeline and providing large diversity without storage overhead (\({\sim}4\) s/patch on CPU). Synthesis parameters were selected empirically via visual inspection and prior knowledge of tracer appearance. Beyond the in-pipeline transforms, we apply a second augmentation stage during training that includes spatial transforms (flips, rotations \(\pm 20^\circ\), elastic deformations), appearance transforms (color jitter, random gamma, hue–saturation shifts), and stochastic degradations (Gaussian/ISO noise, mild blur).

2.2 Mixed Training Pipeline↩︎

An overview of the pipeline is shown in Figure 1. Each training batch combines real and synthetic samples at ratio \(r_{\text{syn}}\). Real patches use foreground-aware sampling (half centered on fiber pixels, half at random locations) to address class imbalance. At inference, overlapping \(1024{\times}1024\) patches are aggregated by averaging to produce full-section probability maps, followed by connected component filtering to remove small spurious detections smaller than 800 pixels..

3 Experiments↩︎

Datasets. We use high-resolution coronal histological sections from four in-house macaque datasets (M1–M4) with bidirectional tracer injections at multiple cortical sites. Sections (50 μm thick) were digitized at 0.4 μm in-plane resolution, with every 24^th section processed to visualize a specific tracer (1.2 mm effective inter-slice spacing). This sampling strategy follows previous tracer studies in non-human primates [22], [23]. Expert neuroanatomists annotated fiber bundles (i.e. groups of axons traveling in close proximity from injection sites to terminal fields) into three density classes: dense, moderate, and sparse. Approximately 30 sections per brain were annotated, yielding \({\sim}90\) section-level evaluation samples across the three held-out brains (M2–M4), each downsampled \(16{\times}\) to a \({\sim}4\text{k}{\times}4\text{k}\) pixel field of view at 6.4 μm effective resolution. We train using annotations from a single brain (M1). Generalization is evaluated on three held-out brains: M2 and M3 (partially used to train the SOTA, precluding direct comparison on these brains) and M4 (fully held-out, enabling comparison with all methods).

Evaluation Metrics. Standard overlap metrics (Dice, IoU) are ill-suited to this task: severe class imbalance deflates their values, and the exact boundaries of each bundle are less important than its location and extent. We therefore adopt the component-based evaluation of [13]: connected component matching yields True Positive Rate (TPR) per density class (dense/moderate/sparse), False Discovery Rate \(\text{FDR} = \text{FP}/(\text{TP}+\text{FP})\), and average TP/FP per section.

Baselines. SAM2 [24] and MedSAM2 [25] are evaluated as zero-shot baselines, prompted automatically from the white-matter region. Both perform poorly on tracer histology, occasionally detecting dense bundles but mostly missing moderate and sparse bundles, and FDR close to 1.0, underscoring the need for task-specific training. We further compare against: (i) Sundaresan et al. [13], the initial self-supervised deep-learning method for this task, which relies on cross-section post-processing; (ii) Bintsi et al. [14], the current state-of-the-art (SOTA), which uses semi-supervised pre-training and requires annotations from three brains (M1–M3); and (iii) a real-only baseline using the pipeline from [14], trained on only one brain (M1) instead of three, to isolate the effect of synthetic data from the effect of additional annotated brains.

Implementation Details. We use a 2D U-Net [26] with nine encoder–decoder levels, each comprising two \(3{\times}3\) convolutions with instance normalization and LeakyReLU. Training uses AdamW [27] with learning rate \(10^{-4}\), cosine annealing with linear warmup, batch size 8, mixed-precision (FP16), and a combined BCE–Dice loss for 150 epochs. Each batch mixes real and synthetic data at ratio \(r_{\text{syn}} = 0.7\) (default). Real patches (\(1024{\times}1024\)) are sampled with 50% centered on fiber pixels to address class imbalance; synthetic patches (\(512{\times}512\)) are generated on-the-fly and resized to match. Real data receives conservative augmentation (flips, small rotations), while synthetic data undergoes extensive domain randomization (Sec. 2.1). Training runs on a single NVIDIA RTX 6000 GPU.

4 Results↩︎

Main comparison. Table 1 compares all methods on the held-out test brain M4. Our method outperforms the SOTA [14] in overall detection (\(\text{TP}_\text{avg}\) 3.24 vs. 3.00) and achieves similar sparse-bundle sensitivity (0.79 vs. 0.78), while being trained only on one brain instead of three. It substantially outperforms Sundaresan et al. [13] (\(\text{TP}_\text{avg}\) 3.24 vs. 1.76) and improves sparse bundle detection over the real-only baseline (0.79 vs. 0.52), at the cost of a higher FDR (0.36 vs. 0.18).

Figure 2 shows qualitative results on two sections from M4, comparing the SOTA  [14] (left) with our method (right). Both methods recover dense and moderate bundles well, but the SOTA baseline often misses sparse bundles that our model detects, in line with the higher sparse-bundle TPR in Table 1. These additional detections come at the cost of slightly more FPs.

Cross-brain generalization. Since we train on M1 only, we can additionally evaluate on M2 and M3, which the SOTA [14] used for training. Table 2 shows that our mixed real+synthetic training (Proposed) generalizes well across held-out brains, improving over [13] especially on moderate and sparse bundles (e.g., on M2, \(\text{TPR}_\text{M/S}\) 0.95/0.64 vs./0.48). Compared to the real-only baseline, the proposed model substantially increases \(\text{TPR}_\text{S}\) (from 0.35 to 0.64 on M2 and from 0.38 to 0.62 on M3), at the cost of higher FP\(_\text{avg}\) and FDR on M2 (0.40 vs.), reflecting a systematic shift toward higher recall under cross-brain shift. Notably, on M3 the FDR is reduced relative to the real-only baseline (0.37 vs.), suggesting that the elevated FDR on M2 reflects a challenging domain shift rather than a systematic failure mode.

2.2pt

Ablations studies. Real data amount. Table ¿tbl:tab:ablations? (left) reports mean performance across held-out brains M2–M4. A single annotated section paired with synthetic data already achieves competitive sensitivity (\(\text{TPR}_\text{D/M/S}\) 0.88/0.80/0.57) with the lowest FDR (0.32); using a full annotated brain further improves detection, especially for sparse bundles (0.69 vs.), at the cost of a higher FDR (0.38). Performance is non-monotonic so 5 sections do not consistently outperform 1, indicating high variance in the low-shot regime.

Synthetic ratio. Table ¿tbl:tab:ablations? (right) varies the proportion of synthetic images per batch, averaging over M2–M4. Synthetic-only training fails (mean \(\text{TPR}_\text{S}\) 0.26, FDR 0.94), confirming that real supervision is essential. Compared to the real-only setting (\(r_{\text{syn}}{=}0\%\)), mixed real+synthetic training improves dense and moderate TPR, maintains sparse recall, and reduces FDR, achieving a better overall precision–recall trade-off. This holds across a broad range of synthetic ratios (\(r_{\text{syn}}\in\{30\%,50\%,70\%\}\)), indicating robustness to this hyperparameter.

Domain randomization. Across M2–M4 we also compare our default domain randomization to a less aggressive variant. Default DR consistently achieves a better precision–recall trade-off: on all brains it maintains comparable TPR while reducing FDR (e.g., 0.40 vs. on M2 and 0.36 vs. on M4), suggesting that stronger augmentation acts as an effective regularizer across brains.

5 Discussion and Conclusion↩︎

We have presented a mixed real+synthetic training strategy for fiber bundle segmentation in tracer histology that leverages dMRI tractography as a generative prior. Our results demonstrate that synthetic augmentation effectively complements limited real supervision: performance comparable to the SOTA [14] is achieved with annotations from a single brain (vs.three), and even a single annotated section yields competitive detection when paired with synthetic data. Crucially, synthetic data alone is insufficient. Real supervision remains essential to anchor the learned representations but at intermediate mixing ratios, synthesis acts as a regularizer that improves precision relative to real-only training.

The primary limitation of our approach is elevated FPs on some held-out brains, which we attribute to domain shifts in background appearance, tracer characteristics, and injection patterns not fully captured by the current synthesis pipeline; notably, on M3 FPs increase while FDR decreases relative to the real-only baseline, indicating that TP gains outweigh FP increases. We note that this does not reflect circularity from the tractography prior: our model learns only local axon-bundle texture from thin histological sections, not long-range connectivity, so the dominant failure mode of tractography does not propagate into the segmentation task. Mixed training with real annotated histology further grounds the model in biologically valid tracer appearance. Increasing training diversity through additional real brains or improved synthesis would further improve generalization. The non-monotonic behavior in the low-shot regime also suggests that which sections are annotated matters as much as how many, motivating active selection strategies. Future work will focus on improved synthesis to better capture tracer-dependent intensity profiles and staining artifacts via generative models [28], and uncertainty-aware filtering to reduce FPs.

Ultimately, by reducing dependence on annotated data through synthesis, this work enables reliable bundle segmentation not only for similar protocols but also for unseen tracers and tissue preparations, thus making a key step toward high-throughput mesoscale connectivity mapping.

References↩︎