1 Introduction↩︎

Identifying biomarkers in medical images is a critical first step in many biomedical workflows. It involves examining images to locate important markers. Segmentation of biomarkers within medical imagery is a branch of medical image segmentation that faces difficulties such as scarce sample availability which can hinder semantic segmentation efforts. Notable AI models including UNet [1] and its variants [2] improve segmentation performance by employing a nested encoder decoder design with dense skip connections. However these models based on CNN often fail to capture long range dependencies a limitation that has been mitigated by incorporating attention mechanisms [3]. Vision Transformers excel at modeling interactions across distant regions yet often struggle to capture precise local details. New architectures such as SegFormer [4] and UFormer [5] integrate convolutional modules into transformer frameworks to balance both global and local feature extraction. Even with these advances further refinement is needed to fully model pixel level relationships in every spatial dimension.

To achieve better segmentation accuracy and mitigate the shortcomings, we introduce an architecture that can capture contextual features across multiple scales by leveraging the Nested-UNet arrangements and by leveraging an attention mechanism to effectively upsample features. In this study, our primary focus is to increase the segmentation accuracy of biomarkers, which present numerous challenges due to significant differences in size and appearance and often have unclear boundaries, and to compare our approach with existing Nested-UNet architectures. Below we summarize the main contributions of this study:

• We propose a model with a hierarchical arrangement of encoder and decoder modules that efficiently integrates contextual features across multiple scales.

• Additionally, we introduced inner and outer attention units, enabling the network to focus on more relevant features during upsampling.

• To further enhance the quality of segmentation, we integrated channel-wise feature recalibration using Squeeze-and-Excitation modules.

• We conducted comprehensive experiments on three medical image segmentation datasets and found that our proposed enhancements yield clear performance gains over existing Nested-UNet methods.

2 Related Work↩︎

Recent work in Convolutional neural network architectures for semantic segmentation has centered on merging deep features from various levels. Studies [6]–[9] demonstrate that drawing information from multiple layers enhances overall performance. Variants of UNet which belongs to Nested-UNet family such as U2-Net [10] apply multi-scale fusion to capture both local details and broader context. Models like UNet++ [11] and UNet 3+ [12] introduce nested dense and full scale skip connections between encoder and decoder modules and employ deep supervision to strengthen decoder outputs. Despite these improvements, convolutional approaches still find it challenging to model long range dependencies, prompting attention based solutions. For instance MA-UNet [13] integrates multi-scale attention. Even with these advances, convolutional models remain in need of greater flexibility to effectively capture both fine and global information across diverse data sets. Transformer-based architectures represent a significant advancement in computer vision. Initially designed to implement complex attention in natural language processing they have also been adapted for image based tasks. In segmentation TransUNet [14] employs a transformer encoder to learn rich feature representations along with a UNet style CNN-based decoder to upsample those features effectively. Swin-UNet [15] utilises a hierarchical Swin Transformer encoder with shifted windows to gather multi-scale context and a matching decoder with a patch expanding layer to restore spatial detail during upsampling. However most of transformer-based methods depend on large data sets to capture long range dependencies.

3 Method↩︎

3.1 Overview↩︎

This architecture employs a hierarchical nested design in which each Inner-UNet module refines feature representations at a particular resolution by combining convolution blocks with attention gates. These inner modules are embedded within a larger Outer-UNet scaffold that manages the progression of information through successive downsampling and upsampling stages. The outer network draws on the outputs of all inner modules to assemble multi-scale feature maps, ensuring that both local details and global context are maintained throughout the segmentation process. By integrating channel recalibration at every stage, the model delivers strong feature encoding and precise mask generation across diverse medical imaging data sets.

3.2 Squeeze and Excitation (SE) Block:↩︎

As described in [16] and illustrated in Figure 1 part b this block first compresses feature maps using global average pooling then applies dense layers with ReLU and sigmoid activations to compute channel-wise scaling factors. These factors are multiplied with the original feature maps to enable channel-wise attention.

3.3 Convolution Block:↩︎

This block applies depthwise separable convolution layers followed by batch normalization and a LeakyReLU activation to capture and refine feature representations and to enable non linear transformations.

3.4 Attention Gate:↩︎

To prioritize key features from skip connections, we use an attention mechanism following [3]. The attention mechanism filters out irrelevant and noisy signals before combining important activation. This mechanism also regulates gradient flow to focus updates on crucial spatial areas, enhancing model accuracy.

3.5 Atrous Spatial Pyramid Pooling (ASPP):↩︎

As introduced in Deeplab [17], the ASPP block maintains the spatial dimension of feature maps while extracting high-semantic features with a larger field-of-view using the multi-grid method. We have modified it to use depthwise separable convolution.

3.6 Inner-UNet (I-UNet) Block↩︎

The I-UNet block shown in Figure 1 part c is designed to capture long range and multiple scale contextual information through an attention gate. Convolution blocks followed by max pooling in the encoder extract hierarchical features, and a bridge convolution block connects this representation to the decoder. During decoding upsampling layers work together with the attention gate and convolution blocks to merge coarse and fine details via skip connections. A final residual connection then fuses the original local feature maps with the aggregated long range multiple scale information to produce the block’s output. Various configurations of the I-UNet module are embedded within the Outer-UNet encoder and decoder and are designated by a parameter k. The value of k combines the initial convolution block with the total number of convolution blocks in the encoder. In this setup k minus one corresponds to the count of encoder convolution blocks, k minus two indicates both the number of max-pooling layers in the encoder and upsampling layers in the decoder, and twice k gives the total convolution block count inside the I-UNet module. All variants share a symmetric design that incorporates attention gates and skip connections. Figure 1 part c depicts the I-UNet 4 variant.

3.7 Bridge Block↩︎

In the deeper layers, if the spatial dimension becomes significantly small, the network cannot extract semantic information. Therefore, we use dilated convolutions to allow the network to learn dense semantic information. As shown in Figure 1 part d.

3.8 Outer-UNet↩︎

The core network appears in Figure 1 part a and brings together all the components described above. Its aim is to extract feature maps, merge them effectively and then recover spatial resolution. The encoder path is built from four Inner-UNet modules named I-UNet 7, 6, 5, and 4 each followed by a convolution block for spatial feature learning an SE block for channel-wise recalibration and a max pooling layer for downsampling. The bridge block then refines these deep representations and the ASPP block gathers multi-scale context. In the decoder path upsampling layers, attention gates and skip connections reverse the process to fuse encoder features precisely. After each upsampling step one of the Inner-UNet modules is applied. Finally outputs from all decoder stages are resized to the original resolution concatenated and passed through a 1x1 convolution before a sigmoid activation produces the final binary mask. This design enables the model to learn a variety of spatial features and blend them seamlessly to create segmentation masks that cover multiple scales and contexts.

4 Experiments↩︎

4.1 Datasets↩︎

To evaluate our approach we used three publicly available segmentation datasets. These sets include images captured under varying illumination conditions and magnifications and feature different cell types and imaging modalities. Our goal was to assess the model’s ability to generalize across these diverse scenarios. For training we generated overlapping patches of size 256 × 256 pixels with a stride of 128. During testing, we applied the same patch generation procedure to perform predictions.

Data Augmentations: We apply both offline and online augmentation during training. Using offline augmentation on the training sets we generated 8820 samples for MoNuSeg, 13710 samples for DSB, and 9852 samples for the EM dataset. Offline transforms are applied with full certainty and consist of random brightness and contrast adjustments grid distortion image transposition and elastic transformation. During online data loading we perform at each iteration rotations up to 90 degrees width and height shifts of up to 0.3 of the image dimension shear operations of up to 0.5 zoom of up to 0.3 and random horizontal and vertical flips.

The details of the datasets are as follows:

1. The Multi-organ Nucleus Segmentation (MoNuSeg) [18] dataset contains H&E-stained histopathology images. The dataset comprises 44 images, each with dimensions of 1000 by 1000 pixels. The MoNuSeg dataset consists of 30 images allocated for training and an additional 14 images reserved for testing.
   

2. The 2018 Data Science Bowl (DSB) [19] dataset encompasses a diverse collection of segmented cell images acquired under various conditions and from different organisms. The images have dimensions ranging from 128 x 128 to 512 × 512 pixels. We solely relied on the stage 1 training set due to the unavailability of publicly accessible ground truth masks.
   

3. Electron Microscopy (EM) [20] dataset features annotated mitochondria from the CA1 hippocampus region of the brain. The training and testing sets each contain 165 images with corresponding masks, with a size of 768x1024 pixels.

4.2 Evaluation Metrics↩︎

We evaluated both existing deep learning methods and our HistoSeg++ using standard evaluation metrics [21]. Pixel based performance was assessed using IoU, Dice, precision and recall. To capture surface distance characteristics we included the Hausdorff distance at the ninety fifth percentile which enriches our evaluation. Each metric was calculated for every prediction and then averaged across all test samples.

4.3 Experimental Setup↩︎

We trained the network with the Adam optimizer using a learning rate of 0.0001 and a batch size of eight and initialized convolution kernels with the He Uniform scheme. Training was carried out from scratch for two hundred epochs. For MoNuSeg and EM we used their standard test sets while for DSB we adopted a nine to ten split between training and evaluation data. We created a validation set from the training samples, selected the checkpoints that performed best on the validation set, and used them for evaluation. We reproduce all the results for existing methods using their publicly available repositories and use the optimal settings reported in their work. For our work, we employed the edge-aware loss function which improves boundary accuracy in segmentation by first computing cross-entropy between predicted and ground truth masks and then amplifying errors at true object edges. Edges are identified via Sobel filtering followed by a gradient magnitude threshold. Error values at those edge locations are multiplied by a factor greater than one before averaging across all pixels. Edge-aware loss function is defined as: \[L(y, \hat{y}) = \frac{1}{N} \sum_{i=1}^N \left(1 + \left(\text{em}_i \times (w - 1)\right)\right) \times \text{BCE}(y_i, \hat{y}_i)\] where \[\text{edge\_mask (em)} = \frac{1}{D} \sum_{d=1}^D \mathbf{1}\left(\sqrt{s_x^2 + s_y^2} > 0.1\right),\] \[s_x = \text{Sobel}_x(y), \quad s_y = \text{Sobel}_y(y),\] and \(w\) is the edge weight, \(N\) is the number of samples, and \(D\) is the spatial dimensionality of the mask.

4.4 Results↩︎

To evaluate our proposed method and existing methods, we selected baseline models such as UNet [1] and nnU-Net [22], as they are original architectures widely used in segmentation tasks. HoVer-Net [23] is included because it serves as a baseline for biomarker segmentation. Swin-UNet [15] and TransUNet [14] are baseline transformer-based methods for segmentation. UNet++ [11], UNet 3+ [12], and U2-Net [10] are SOTA methods from the Nested-UNet family. Quantitative comparison is presented in Table 1. For the nuclei segmentation, HistoSeg++ achieves the highest accuracy, outperforming transformer-based methods such as Swin-UNet, which exhibit lower segmentation fidelity. Among convolutional variants, UNet++ and nnU-Net deliver solid results but lag in capturing fine nuclear details. In the cell segmentation dataset, characterized by diverse morphologies and imaging conditions, our approach again leads in overall quality and precision. Transformer backbones like TransUNet show strong precision but often miss subtle cell regions, while UNet 3+ performs weakest, struggling with overlap accuracy and boundary localization. For mitochondrial segmentation, the nested skip connections and deep supervision of UNet++ enable strong multi-scale feature extraction. In contrast, the original UNet performs poorly, reflecting its limited capacity for modeling long-range dependencies in complex micrographs. By combining edge-aware loss and nested Inner-UNet modules, our method yields the most complete and precise mitochondrial masks. Qualitative results are shown in Figure 2. Transformer-based methods often over-segment and add artifacts, while Nested-UNet variants yield precise, coherent masks. Our approach produces reliable segmentation with minimal over-segmentation.

4.5 Ablation Studies↩︎

We conducted ablation studies to assess the effectiveness of components in our model, using the MoNuSeg dataset for the experiments, as detailed in Table 2. Our baseline involves nested U-Nets in an encoder-decoder setup. Adding inner (in_A) and outer (out_A) attention improves all metrics, notably enhancing Recall, which aids in better identifying relevant cases. Integrating ASPP, and SE modules boosts IoU and Dice scores for increased segmentation accuracy. These advancements consistently improve segmentation but require more computational resources.

5 Conclusion↩︎

This work presents a novel deep neural framework for precise segmentation in medical imaging. The proposed model combines receptive fields of varying sizes with spatial and channel attention modules to capture contextual cues at multiple scales and improve feature reconstruction during upsampling. We validate its performance on three public datasets, demonstrating consistent gains in segmentation quality. With further refinement the approach could enable real time processing and support multi class segmentation.

Acknowledgments↩︎

This work was supported by the National Research Foundation of Korea(NRF) grant funded by the Korea government(MSIT)(RS-2025-00573160).

References↩︎