1 Introduction↩︎

Accurate segmentation of medical images plays a critical role in computer-aided diagnosis and treatment planning. By delineating anatomical structures and pathological regions, segmentation models enable reliable disease screening, quantitative analysis, and therapy assessment.

Despite recent advances, medical image segmentation still faces a fundamental trade-off between annotation accuracy and efficiency. Fully supervised models, such as U-Net [1] and its variants [2], achieve strong performance with pixel-level annotations, but dense mask labeling is expensive and time-consuming, especially for lesions with irregular or ambiguous boundaries. To alleviate this burden, weakly supervised segmentation methods employ coarse annotations such as points [3], scribbles [4], or bounding boxes [5]. These annotations significantly reduce labeling effort but provide limited geometric constraints. In particular, conventional bounding boxes fail to encode orientation, making them inadequate for elongated or anisotropic lesions.

To bridge the gap between full and weak supervision, we propose an intermediate paradigm, termed Oriented Bounding Box Guided Segmentation (OBBSeg), combining the efficiency of weak supervision with the geometric fidelity of full supervision. Fig. 1 illustrates the proposed Oriented Bounding Box (OBB) annotations, which provide a compact, orientation-aware annotation that better captures lesion geometry than conventional bounding boxes. By jointly encoding spatial extent and orientation, OBBs tightly fit elongated or anisotropic lesions (see row 1), providing stronger geometric constraints and reducing ambiguous regions (see rows 3-5). Moreover, they alleviate the overlap between closely packed lesions that commonly occurs with conventional box annotations (see row 2).

Furthermore, OBB annotation remains lightweight, requiring only four corner points per instance. To evaluate its annotation cost, we conducted a human annotation study using our training set, where three independent annotators labeled 1,616 OBBs across 1,450 images. The average annotation time was 17.8 s per OBB, only slightly longer than standard bounding boxes (13.3 s per image) and substantially shorter than dense mask annotation (91.1 s per image). These results show that OBB annotation provides richer geometric information with only a modest increase in annotation effort, making it an effective compromise between annotation efficiency and geometric precision.

Although OBB annotations capture target extent and orientation, their rectangular geometry may introduce shape bias during segmentation learning. To address this limitation, we propose the Mask-to-OBB (M2O) loss, a supervision strategy that bridges coarse OBB annotations and fine-grained segmentation masks. M2O projects predicted masks into the OBB space through rotation and projection operations, supervising only spatial extent and orientation while excluding explicit boundary information. This design effectively mitigates rectangular shape bias without iterative refinement. Implemented entirely with matrix operations, M2O is end-to-end differentiable, plug-and-play transferable to other segmentation tasks, and incurs no additional inference cost. As shown in Table 4, M2O consistently outperforms direct OBB supervision, demonstrating the effectiveness of the proposed supervision strategy.

Although OBB supervision reduces annotation cost and provides geometric priors, it remains limited in modeling complex lesion appearance and cross-domain variation. To address this, we propose a unified OBBSeg framework that integrates OBB-based geometric constraints with prompt-guided semantic learning. Different from existing prompt-based methods that introduce prompts only at the decoder stage, OBBSeg injects prompt information into the encoder to explicitly guide feature learning with semantic priors.

In this design, OBBs provide structural cues of spatial extent and orientation, while prompts progressively refine semantic representations and enhance boundary awareness during feature extraction. To further strengthen weakly supervised learning, we introduce two complementary modules: Prompt-assisted Foreground Enhancer (PAFE), which highlights lesion regions using prompt-derived masks while suppressing background interference, and the Differential-based Foreground Enhancer (DBFE), which improves feature discriminability by explicitly enhancing foreground–background contrast.

Overall, OBBSeg jointly leverages geometric constraints, prompt-driven semantics, and differential feature enhancement to improve segmentation accuracy and generalization under weak supervision. Our contributions are fourfold:

• We propose OBBSeg, a new segmentation paradigm that leverages oriented bounding boxes to bridge the gap between weakly and fully supervised learning, providing geometry-aware supervision with low annotation cost.

• We introduce the M2O loss, a differentiable and noise-free supervision formulation that removes the rectangular bias of box annotations and enables geometry-consistent learning.

• We integrate prompt-guided learning with OBB supervision and design two complementary modules, PAFE and DBFE, to enhance foreground representation and improve segmentation robustness.

• Extensive experiments on 13 medical datasets across 5 imaging modalities demonstrate that OBBSeg outperforms existing weakly supervised methods and achieves performance approaching fully supervised models.

2 Related Work↩︎

Medical Image Segmentation. Medical image segmentation aims to delineate anatomical and pathological structures from modalities such as X-ray [6], MRI [7], and ultrasound [8]. Fully supervised models, including U-Net [1], U-Net++ [2], and PraNet [9], have achieved strong performance across various medical imaging tasks. However, these methods rely heavily on pixel-level annotations, which are expensive and time-consuming to obtain. To reduce annotation costs, weakly supervised segmentation methods have been increasingly explored. In this work, we adopt oriented bounding boxes as supervision, which provide geometry-aware annotations that better capture object extent and orientation while maintaining low labeling cost.

Weakly Supervised Learning. Weakly supervised segmentation aims to reduce annotation effort by using coarse labels such as points [3], boxes [5], scribbles [4], and circles [10]. [5] proposed a box-supervised approach with Background-Aware Pooling and Noise-Aware Loss, achieving competitive results on the PASCAL VOC 2012 benchmark [11]. WeakPolyp [12] introduced a projection–backprojection strategy together with Scale Consistency Loss to mitigate box-shape bias, significantly improving polyp segmentation performance. Zhang et al. [13] proposed a Class-Driven Scribble Promotion network that integrates scribble annotations with image-level pseudo-labels and uncertainty-aware optimization. Despite these advances, weak supervision often struggles to capture accurate geometric structures due to the coarse nature of the annotations. In contrast, our method introduces the Mask-to-OBB loss to provide geometry-consistent supervision while eliminating the rectangular bias of box annotations.

Prompt-based Image Segmentation. Prompt-based segmentation leverages auxiliary guidance, such as points, boxes, or textual cues, to improve model adaptability and generalization. Representative models such as SAM [14] and SAM2 [15] encode prompts into geometric embeddings to guide segmentation. However, most existing approaches introduce prompts only at the decoder stage, limiting their influence on feature representation. In contrast, our method incorporates prompts into the encoder to enhance foreground representations and suppress background noise, while further utilizing them for supervision, enabling end-to-end prompt-guided segmentation.

3 Proposed Methodology↩︎

Fig. 2 (a) presents the overall architecture of the proposed OBBSeg framework, which leverages prompts as auxiliary cues and oriented bounding boxes as weak supervision. Built upon a ViT [16] backbone, OBBSeg is organized into four stages. In each stage, the PAFE module exploits prompt cues to suppress background noise, while the DBFE module enhances foreground–background contrast to highlight discriminative regions. At the supervision level, OBBSeg adopts a dual-strategy design. Prompt supervision provides hierarchical guidance to refine coarse prompts, while the proposed M2O loss aligns soft mask predictions with OBB annotations to alleviate box-induced shape bias. Through progressive feature refinement and noise-robust supervision, OBBSeg produces accurate segmentation masks with minimal annotation cost.

3.1 Prompt-assisted Foreground Enhancer↩︎

Fig. 2 (b) illustrates the Prompt-assisted Foreground Enhancer (PAFE), designed to suppress background noise under weak supervision. Unlike conventional attention mechanisms that lack explicit guidance, PAFE utilizes prompt cues to guide the network toward lesion-relevant regions, improving semantic consistency and interpretability. Given an input image \(x\), the backbone extracts a hierarchy of feature maps \(\{x_i \mid i \in (0,1,2,3)\}\). For each stage \(i\), a prompt-derived binary mask \(p_i\) is generated and resized to match the spatial resolution of \(x_i\). Foreground-aware filtering is achieved by element-wise multiplication (\(\odot\)) between \(p_i\) and \(x_i\). Since prompts (e.g., points or scribbles) may only partially cover the target region, a residual connection adds the original feature map back to preserve information, producing the enhanced feature representation \(x_e\): \[x_e = x_i \odot p_i + x_i. \label{eq:pafe}\tag{1}\] To obtain the prompt mask \(p_i\), we directly leverage user-provided prompts (e.g., points or scribbles) as partial supervision signals. Supervision is enforced only on confidently labeled pixels within the prompt regions, providing reliable and noise-free guidance. This partial supervision encourages the model to progressively learn object-aware masks aligned with the target’s location and shape, as illustrated in Fig. 2 (d).

3.2 Differential-based Foreground Enhancer↩︎

Medical image segmentation often suffers from high visual similarity between foreground and background, which hinders discriminative feature learning, especially under limited training data. To address this issue, we propose the Differential-based Foreground Enhancer (DBFE), which explicitly amplifies foreground–background contrast to facilitate feature discrimination. The overall architecture is shown in Fig. 2 (c).

Given the prompt-derived binary mask \(p_{i+1}\) highlighting the foreground region, its complement \((1 - p_{i+1})\) represents the background. We first extract background cues by applying the background mask to the input feature \(x_e\) via element-wise multiplication followed by average pooling, producing a background representation \(x_e^{'}\). To enhance feature contrast, a differential factor is computed by subtracting \(x_e^{'}\) from \(x_e\) and applying the foreground mask \(p_{i+1}\) through element-wise multiplication (\(\odot\)), which emphasizes discriminative foreground information. Following a residual design, this differential factor is added back to the original feature to preserve contextual information, yielding the enhanced feature \(x_{i+1}\).

\[\begin{align} x_e^{'} &= \text{pooling}((1 - p_{i+1}) \odot x_e) \\ x_{i+1} &= (x_e - x_e^{'}) \odot p_{i+1} + x_e \end{align} \label{eq:dbfe}\tag{2}\]

3.3 Mask-to-OBB Loss↩︎

Direct supervision with OBBs introduces rectangular shape bias, misleading the model toward box-like predictions. To address this, we propose Mask-to-OBB (M2O) loss, which removes shape bias by aligning predictions with OBBs in a geometry-aware way. Specifically, we extract instance masks, rectify them via affine transformation, and eliminate shape details through axis-wise projection and minimum fusion. The resulting masks retain spatial extent without contour bias, and are inversely transformed back for pixel-wise loss computation. This enables effective learning from coarse OBB annotations without shape-induced errors.

Split. Given the predicted mask \(s\) and a set of OBB masks \(\{b_1, b_2, \dots, b_M\}\), we first decompose \(s\) into instance-level sub-masks, each corresponding to an individual object. Specifically, for each OBB mask \(b_m\), the sub-mask is extracted by element-wise multiplication \(s_m = s \odot b_m\).

Affine Transformation. Given an OBB mask \(b_m\), we extract its rotation angle \(\theta_m\) and construct the affine transformation matrix \(R_m\). This matrix rotates the sub-mask \(s_m\) by \(\theta_m\) clockwise, resulting in the rectified prediction \(s_a\) which is aligned to axis orientation. \[s_a = \text{Warp}(s_m, R_m) \label{eq:affine}\tag{3}\]

Projection. We perform max-projection on \(s_a\) along the horizontal and vertical axes, yielding the maximum probability vectors: \(s_h\) and \(s_v\). This step completely eliminates the shape and contour information of \(s_a\), retaining only its spatial location and dimension information. \[\begin{align} s_h &= \text{max}(s_a, \text{axis}=1)\in [0,1]^{1 \times W} \\ s_w &= \text{max}(s_a, \text{axis}=0)\in [0,1]^{H \times 1} \\ \end{align} \label{eq:projection}\tag{4}\]

Minimum Fusion. To reconstruct the spatial mask, we back-project the horizontal and vertical projection vectors by replicating \(s_h\) across width and \(s_w\) across height, yielding two 2D masks \(s_h \in [0,1]^{H\times W}\) and \(s_w \in [0,1]^{H\times W}\). These masks are then fused via element-wise minimum operation to produce the final aligned prediction \(s_{hw}\). \[\begin{align} s_h &= \text{repeat}(s_h, H, \text{axis}=1) \in [0,1]^{H\times W} \\ s_w &= \text{repeat}(s_w, W, \text{axis}=0) \in [0,1]^{H\times W} \\ s_{hw} &= \min(s_h, s_w) \in [0,1]^{H\times W} \\ \end{align} \label{eq:back-projection}\tag{5}\]

Inverse Affine Transformation. We apply the inverse transformation matrix \(R_m^{-1}\) to \(s_{hw}\), rotating it counterclockwise by \(\theta_m\) back to an axis-aligned state. This produces the transformed predicted mask \(s_o\). \[\begin{align} s_o &= \text{Warp}(s_{hw}, R_m^{-1}) \end{align} \label{eq:inverse:affine}\tag{6}\]

Combination. For each image, we repeat the above transformation process for all OBBs, generating \(M\) rectified prediction masks \({s_o^1, s_o^2, \dots, s_o^m}\). We then aggregate them via pixel-wise maximum fusion to obtain the final transformed mask \(s_t\): \[s_t = \max(s_o^1, s_o^2, \dots, s_o^M) \in [0,1]^{H\times W} \label{eq:combination}\tag{7}\]

Supervision. The transformed prediction \(s_t\) produced by the M2O transformation enables direct computation of Binary Cross-Entropy loss \(\mathcal{L}_{\text{BCE}}\) and Dice loss \(\mathcal{L}_{\text{Dice}}\) against the box-shaped OBB ground truth \(y\). The combined loss \(\mathcal{L}_{\text{M2O}}\) is defined as Eq. 8 . Definitions of \(\mathcal{L}_{\text{BCE}}\) and \(\mathcal{L}_{\text{Dice}}\) are provided in the supplementary material. \[\mathcal{L}_{\text{M2O}} = \mathcal{L}_{\text{BCE}}(s_t, y) + \mathcal{L}_{\text{Dice}}(s_t, y) \label{eq:M2O}\tag{8}\]

3.4 Prompt Supervision Loss↩︎

To compensate for the limited supervision from OBB annotations and better exploit prompt masks, we introduce the Prompt Supervision Loss \(\mathcal{L}_{\text{PS}}\). This loss serves two purposes: (1) guiding early-stage feature learning to refine intermediate predictions \(p_i\), and (2) strengthening supervision to complement OBB-based learning. Specifically, \(\mathcal{L}_{\text{PS}}\) enforces hierarchical consistency between the intermediate predictions and the final prediction with respect to the prompt mask: \[\mathcal{L}_\text{PS} = \mathcal{L}(s, p) + \sum_{i=1}^4 \mathcal{L}(p_i, p).\] To support different prompt types (, box, point, scribble, circle), we adopt adaptive supervision functions \(\mathcal{L}\) tailored to each prompt. This results in \(\mathcal{L}_{\text{PS}}^{\text{box}}\), \(\mathcal{L}_{\text{PS}}^{\text{scribble}}\), \(\mathcal{L}_{\text{PS}}^{\text{point}}\), and \(\mathcal{L}_{\text{PS}}^{\text{circle}}\). Detailed implementations for each prompt type are provided in the supplementary material.

3.5 Total Loss↩︎

Following WeakPolyp, we adopt the Scale Consistency Loss \(\mathcal{L}_{\text{SC}}\) to alleviate the supervision deficiency of OBB annotations by enforcing prediction consistency across different input scales. Combined with the proposed \(\mathcal{L}_{\text{M2O}}\) and \(\mathcal{L}_{\text{PS}}\), the overall loss function is defined as Eq. 9 , where \(\lambda\) is a balancing hyper-parameter. Detailed analysis of \(\lambda\) is provided in the supplementary material. \[\mathcal{L}_{\text{Total}} = \mathcal{L}_{\text{M2O}} + \lambda \mathcal{L}_{\text{PS}} + \mathcal{L}_{\text{SC}} \label{eq:loss:total}\tag{9}\]

4 Experimental Results↩︎

Datasets. We evaluate OBBSeg on 13 public datasets across five imaging modalities: colonoscopy (Kvasir [17], ClinicDB [18], ColonDB [19], ETIS [20], Endo [21], SUN-SEG [22]), dermoscopy (ISIC-2017 [23], ISIC-2018 [24], [25]), MRI (NCI-ISBI [26]), CT (Synapse [27]), and ultrasound (TN3K [8], DDTI [28], BUSI [29]). Detailed descriptions are provided in the supplementary material.

Competing Methods. We compare OBBSeg with 20 representative methods, including U-Net [1], U-Net++ [2], AttnUNet [38], TransUNet [39], PraNet [9], FLA-Net [33], MS-TFAL [34], SANet [30], LGRNet [35], MEGANet [31], CASCADE [32], EMCAD [36], WeakPolyp [12], KnowSAM [40], BoxInst [41], BoxTeacher [42], PointSup [43], AGMM [44], GazeMedSeg [45], and MedSAM2 [37].

Implementation Details. We use two backbones: Res2Net50 [46] and the SAM2 encoder [15], [47]. When using SAM2, we initialize the encoder with its pretrained weights and keep them frozen during training, and optimize only the OBBSeg-specific modules. Input images are resized to \(352\times352\). Following WeakPolyp [12], OBB annotations are generated from ground truth masks via random rotations and translations (Table 9 and Table 10) to simulate real annotations. In the promptable setting, prompts are also simulated from ground truth masks following the standard protocol in promptable segmentation. These masks are used only offline to derive sparse or coarse prompt cues, and are not provided as dense training targets or loss supervision.

Training is performed on two RTX 4090 GPUs using AdamW (lr=\(1e^{-4}\), weight decay=\(5e^{-4}\)) with a batch size of 16 for 16 epochs. For datasets without official splits, we adopt an 80/20 train–test split. During inference, OBBSeg can operate with prompts (SAM2+OBBSeg) or in a prompt-free manner, enabling both interactive and automatic segmentation.

Evaluation Metrics. Model performance is evaluated using the Dice coefficient and the 95th percentile Hausdorff Distance (HD95). Dice measures overlap, while HD95 measures boundary accuracy. Higher Dice and lower HD95 indicate better performance. See the supplementary material for detailed definitions.

Quantitative Comparison. Tables 1 and 2 report OBBSeg results on diverse segmentation tasks, using the weighted average Dice score (“Avg.”) based on dataset sizes. Two baselines, WeakPolyp and SAM2, are considered. WeakPolyp uses Res2Net [46] as the backbone and does not require prompts. By simply adding the M2O loss without other modifications, we achieve a consistent 1%–2% improvement across all datasets, demonstrating the effectiveness of M2O. For SAM2, we integrate the full OBBSeg framework (M2O + Prompt) and evaluate different prompt types. This leads to substantial performance gains, approaching, and on several datasets exceeding, the performance of fully supervised methods across datasets. These results demonstrate the effectiveness of combining weak supervision with prompt guidance, highlighting its potential for large-scale applications. Table 3 further compares OBBSeg with existing weakly supervised methods. WeakPolyp+M2O already outperforms prior approaches by a large margin. Building upon this, OBBSeg with prompt guidance further improves performance, achieving state-of-the-art results on both the polyp (Kvasir) and prostate (NCI-ISBI) datasets.

Multi-organ Segmentation. We evaluate OBBSeg on the 8-class Synapse dataset [27], achieving average Dice of 83.8%, 86.5%, 86.9%, 88.2% across Point, Box, Circle, Scribble prompts in Table 5, respectively, demonstrating its effectiveness for multi-class segmentation.

Visualization Comparison. Fig. 4 shows qualitative comparisons across three modalities. Despite using only coarse OBB supervision, our model (row 3) accurately captures lesion boundaries. Especially for elongated or tilted structures (column 1, 2, 3), OBBSeg not only outperforms the weakly supervised WeakPolyp but also achieves visually competitive results with fully supervised models(row 5, 6, 7), highlighting OBBSeg’s consistent superiority across diverse modalities.

Ablation Study. Table 4 and Fig. 5 (a) present the contribution of each module. M2O provides clear gains and establishes robust supervision with reduced box bias. When prompts are available, the prompt supervision loss yields the largest improvement, with circle prompts achieving the best performance. DBFE and PAFE are complementary modules that benefit from accurate foreground cues provided by prompts. DBFE relies on foreground–background contrast estimation; however, noisy early predictions limit its effectiveness without prompt guidance. Prompt supervision supplies reliable foreground cues, enabling DBFE to better distinguish features and further improve performance.

Boundary Segmentation Analysis. Table 6 reports boundary segmentation performance (HD95) under box and OBB annotations. OBB supervision achieves notably lower HD95 values, indicating more precise and smoother boundaries, confirming that OBBs provide stronger geometric constraints and better boundary localization.

Performance under Equivalent Annotation Budget. To further validate the efficiency of Oriented Bounding Box (OBB) supervision, we conduct a comparative analysis under a fixed annotation budget in Table 7. Based on the average annotation time per instance, we subsample the OBB training set such that its cumulative annotation cost aligns with that of the full axis-aligned bounding box (AABB) training set.

Elongated Lesion Segmentation. Table 8 presents segmentation results for elongated lesions. Incorporating OBB and M2O loss improves performance, with particularly strong gains for slender lesions (+5.3%), demonstrating that OBBs better capture the shape of elongated structures.

OBB Sensitivity Analysis. We evaluated the sensitivity of OBB annotation to rotational and positional errors. Table 9 shows that varying OBB angle by ±30° causes only a minor performance drop (\(<\)​2%). Table 10 shows that shifting the padding by up to 15 pixels has negligible effect, demonstrating strong tolerance to annotation errors.

We further evaluate the robustness of human OBB annotations introduced in Sec. 1. As shown in Table 11, human OBBs have a deviation of \(21.20\%\), which is comparable to the strongest Box perturbation in Table 10 (Padding \(0{\sim}20\), deviation \(19.74\)). Under a similar perturbation level, WeakPolyp suffers a substantial drop of \(12.2\%\) (\(78.5 \rightarrow 66.3\)), whereas WeakPolyp+M2O with human OBBs decreases by only \(1.5\%\) (\(79.0 \rightarrow 77.5 \pm 0.32\)). This suggests that the proposed OBB supervision remains stable under realistic annotation variations.

Model Size and Inference Time. OBBSeg can be integrated with different backbones, and the model size and inference time therefore depend on the selected architecture. Using SAM2 as the backbone, the model size is approximately 900 MB, with per-image inference times of 8.4 ms (Point), 12.3 ms (Scribble), 12.1 ms (Box), and 8.2 ms (Circle).

Failure Cases and Limitations. Due to the coarse nature of weak supervision, the annotations lack fine-grained boundary details. In addition, the projection mechanism in the M2O loss provides limited supervision within the OBB region. As shown in Fig. 5 (b), the method may produce suboptimal segmentation for lesions with complex contours or indistinct boundaries. Furthermore, the projection-based design may struggle with highly complex tree-like anatomical structures (, vascular networks) where topology plays an important role. The current framework is also designed for static 2D images and does not explicitly model temporal dependencies in medical videos. Extending the framework to native 3D medical volumes remains an important direction for future work.

5 Conclusion↩︎

We introduced OBBSeg, a new supervision paradigm that combines oriented bounding boxes with prompt-guided learning to provide stronger geometric constraints than conventional weak labels. The proposed Mask-to-OBB loss and hierarchical prompt supervision alleviate box-induced bias and significantly improve segmentation performance. Extensive experiments show that OBBSeg achieves accuracy approaching fully supervised methods across diverse datasets. Nevertheless, segmentation quality is still limited by the coarse nature of OBB annotations. Future work will explore adaptive label refinement and hybrid supervision to further enhance scalability and generalization.

Acknowledgments↩︎

This work was supported in part by ICFCRT (W2441020), Guangdong Basic and Applied Basic Research Foundation (2023B1515120026), Shenzhen Peacock Program (KQTD20210811090044003), Young Faculty Startup Fund of Shenzhen University, and Scientific Development Fund from Guangdong Provincial Key Laboratory of Visual Media and Multidimensional Intelligence.

References↩︎