1 Introduction↩︎

Ultrasound imaging is widely used in routine clinical practice because it is non-invasive, real-time, portable, and cost-effective[1], [2]. It supports screening, follow-up, and point-of-care assessment across fetal, breast, and other organ systems[1]–[5]. However, ultrasound images often suffer from speckle noise, low contrast, operator dependence, and large appearance variations across devices and acquisition settings, motivating extensive deep learning-based analysis[6], [7]. These factors make reliable interpretation difficult even for experienced clinicians and raise concerns about model generalization under distribution shift[8]. Therefore, accurate ultrasound image classification is important for computer-aided diagnosis, including fetal standard-plane recognition and breast lesion assessment[1], [2], [9].

Existing ultrasound classification methods mainly follow two paradigms. Conventional representation learning methods use transfer learning or self-supervised pretraining to extract discriminative features[3]–[5], [8], [10]–[13]. Although effective on benchmark datasets, these “black-box” models may overfit superficial textures rather than clinically meaningful attributes such as echo patterns, boundary cues, and internal structures. This can lead to misclassifications on hard cases (Fig. 1) and limits clinical trust because the prediction is difficult to verify. Interpretable and attribute-guided models, including Concept Bottleneck Models and Vision-Language Models, introduce semantic concepts to align model reasoning with clinical cognition[14]–[22]. Yet their dependence on dense annotations, task-specific concept labels, or complex prompt-tuning[23]–[25] limits lightweight adaptation and increases computational cost. Thus, improving both performance and interpretability without heavy annotation burdens remains challenging.

Motivated by this, we aim to develop a plug-and-play module for existing architectures[9] that improves diagnostic accuracy and transparency while preserving their original training pipelines. Two issues are central: how to transform domain-agnostic medical priors into a structured representation that can guide feature learning, and how to adaptively combine global semantic predictions with attribute-based cues across diverse clinical cases. A practical solution should provide interpretable evidence without requiring dense attribute annotation for every image.

In this paper, we propose AttrGuide, an attribute-guided dual-branch framework that can be integrated into existing classifiers with negligible overhead. The original architecture serves as the baseline branch, while a parallel attribute-guided branch injects clinical priors to bridge visual features and medical knowledge. It constructs a lightweight semantic pathway from predefined medical attributes and reuses the backbone features for attribute matching. An adaptive fusion module then combines baseline logits with prior-informed attribute cues in a data-dependent manner, enabling low-cost self-correction and improving both generalization and interpretability. Since the branch reuses backbone features and performs decision-level fusion, the framework requires only minimal structural modification.

To summarize, the main contributions of this paper are:

• Plug-and-play framework: We propose an attribute-guided dual-branch framework that can be seamlessly integrated into existing models to enhance both robustness and interpretability.

• Clinical-prior injection: We design a dedicated attribute-guided branch that injects domain-agnostic medical priors to provide human-interpretable diagnostic evidence.

• Adaptive decision fusion: We introduce an adaptive fusion module that dynamically reconciles global and attribute-based decisions to yield superior classification performance.

Extensive experiments show that AttrGuide can be integrated into various SOTA backbones, consistently improving performance and interpretability with negligible computational overhead.

2 Method↩︎

We consider a labeled ultrasound dataset \(\mathcal{D}=\{(x_i,y_i)\}_{i=1}^N\), where \(x_i\) is an image and \(y_i \in \{1,\dots,C\}\) is the class label. We collect English attribute words \(\mathcal{A}=\{a_k\}_{k=1}^{K}\) and build a fixed class–attribute matrix \(M\in\{0,1\}^{C\times K}\), where \(M_{c,k}=1\) if class \(c\) is expected to exhibit attribute \(k\). Our goal is to learn a mapping \(f_\theta: x \mapsto z_{\text{fus}} \in \mathbb{R}^C\), where \(\theta\) denotes all learnable parameters and \(z_{\text{fus}}\) are fused logits that integrate a conventional baseline branch with an attribute-guided branch before softmax. The overall architecture (Figure 2) treats any existing encoder+FC classifier as the baseline branch producing logits \(z_{\text{cls}}\), reuses its encoder features to drive an additional attribute-guided branch that produces attribute-based class logits \(z_{\text{attr}}\) via \(M\), and finally combines \(z_{\text{cls}}\) and \(z_{\text{attr}}\) through a lightweight fusion block at the decision level.

2.1 Building the Medical Attribute Semantic Space↩︎

We first construct an attribute semantic space on the text side. Let the pre-trained CLIP text encoder be \(T(\cdot)\)[26]. For each attribute word \(a_k\), we build an ultrasound-specific prompt and encode it as \(e_k = T(\texttt{prompt}(a_k)) \in \mathbb{R}^{d}\), stacking all attribute embeddings into \(E = [e_1;\dots;e_K] \in \mathbb{R}^{K\times d}\).

The matrix \(E\) is precomputed and frozen. On the image side, the backbone \(B(\cdot)\) outputs local features \(v_i\) (ViT tokens or ResNet patches), which together with \(\{e_k\}\) are projected into a shared space and \(\ell_2\)-normalized as \(\tilde{v}_i=\mathrm{norm}(W_v v_i)\) and \(\tilde{e}_k=\mathrm{norm}(W_a e_k)\).

In this common space, we measure the correlation between each patch and each attribute via cosine similarity and aggregate patches to obtain per-attribute logits: \[s_{k,i} = \cos(\tilde{v}_i,\tilde{e}_k),\quad w_{k,i} = \mathrm{softmax}_i(\gamma\cdot s_{k,i}),\quad \ell_k = \sum_{i=1}^{N} w_{k,i}\, s_{k,i}. \label{eq:attr-logits}\tag{1}\] where \(\gamma\) is a scaling factor and \(\ell=[\ell_1,\dots,\ell_K]\) is the attribute prediction vector. Given the class–attribute matrix \(M\), we derive for each class \(y\) a binary target vector \(m_y\in\{0,1\}^{K}\) that encodes which attributes are expected to be present, and supervise attribute prediction with: \[\mathcal{L}_{\text{attr-pred}}=\mathrm{BCEWithLogits}(\ell, m_{y}). \label{eq:attr-loss}\tag{2}\] Beyond this attribute-prediction loss, we introduce a regularization term that further pulls the predicted attribute activations towards the same target vector \(m_y\) using both an element-wise penalty and a directional constraint in the semantic space: \[\mathcal{L}_{\text{reg}} = \mathrm{MSE}(\sigma(\ell), m_y) + \bigl(1-\cos(\sigma(\ell), m_y)\bigr). \label{eq:reg-loss}\tag{3}\]

2.2 Dual-Branch Adaptive Fusion Module↩︎

Baseline classification branch. The global feature \(g\) from the frozen or fine-tuned backbone is passed through the existing classifier head to obtain logits \(z_{\text{cls}}=W_{\text{cls}} g\), which coincide with the baseline prediction (cls_logits).

Attribute-guided branch. The attribute logits \(\ell\) are passed through a sigmoid to obtain scores \(p=\sigma(\ell)\in[0,1]^K\), which we interpret as attribute activations for the current image. These activations are then aggregated into the class space using the fixed matrix \(M\). For class \(c\), we compute a weighted average over its attribute set: \[s_c=\frac{\sum_{k} M_{c,k}\, p_k\cdot p_k}{\sum_{k} M_{c,k}\, p_k+\epsilon}, \label{eq:class-score}\tag{4}\]

where \(s=[s_1,\dots,s_C]\) are class scores that are subsequently scaled to logits, yielding the attribute-guided output \(z_{\text{attr}}\).

Adaptive fusion (low-cost self-correction). To combine the two decisions, we use a learnable fusion weight \(\alpha\) and temperature \(\tau\) that rescale and interpolate the two logits. The final fused logits are \[z_{\text{fus}}=\alpha\cdot (z_{\text{cls}}/\tau) + (1-\alpha)\cdot (z_{\text{attr}}/\tau). \label{eq:fusion}\tag{5}\]

During training, we always use the fused output \(z_{\text{fus}}\) as the sole classification head, computing the cross-entropy loss only on \(z_{\text{fus}}\); the attribute prediction and regularization terms act as auxiliary supervision to shape the shared representation and fusion weights, as validated by the ablations in Table 3.

2.3 Joint Optimization Objective and Training Strategy↩︎

The training objective combines a cross-entropy loss on the fused logits, an attribute prediction loss, and a regularization term that aligns attribute predictions with the class–attribute matrix: \[\mathcal{L}= \lambda_{\text{fus}}\,\mathcal{L}_{\text{fus}} + \lambda_{\text{reg}}\,\mathcal{L}_{\text{reg}} + \lambda_{\text{attr-pred}}\,\mathcal{L}_{\text{attr-pred}}. \label{eq:overall-loss}\tag{6}\]

where \(\mathcal{L}_{\text{fus}}\) is the cross-entropy loss between the fused logits \(z_{\text{fus}}\) and the true class label, \(\mathcal{L}_{\text{attr-pred}}\) is the BCEWithLogits loss on attribute logits \(\ell\) against the binary target vector \(m_y\), and \(\mathcal{L}_{\text{reg}}\) is the regularization term in Eq. 3 that combines an MSE penalty and a cosine-similarity term between the predicted attribute activations (after sigmoid) and the same target vector \(m_y\) derived from \(M\). Using both terms encourages the model to match each attribute probability numerically while also aligning the overall attribute direction in the semantic space. All modules—including the backbone, attribute branch and fusion weight—are optimized jointly using a single optimizer, while the primary supervision for classification is always applied to \(z_{\text{fus}}\).

3 Experiments↩︎

3.1 Datasets and Setup↩︎

Datasets. We consider both public and private ultrasound datasets. The breast cancer three-class task is based on BUSI[2] (normal/benign/malignant), while the fetal plane 7-class task uses an internal multi-center dataset of standard views (abdomen, brain, femur, etc.).

Models and Training. We adopt ImageNet-pretrained ResNet50, ViT-B and Vim-s-16 (BU-Mamba[27]) backbones and simply attach our attribute-guided branch and fusion head, denoting the resulting model as AttrGuide. All models are trained with Adam under the same hyper-parameters (learning rate \(1\times10^{-4}\), batch size 32), and the attribute tables are designed by ultrasound experts from public guidelines for BUSI and the fetal views.

3.2 Results and Analysis↩︎

We first evaluate AttrGuide as a plug-in to an existing SOTA framework. On BUSI, we attach our module to the BU-Mamba Vim-s-16 encoder[27] and compare it with ViT and VMamba baselines to verify that it works across diverse backbones without re-designing the architecture. We further test whether the same module remains effective in a multi-task setting where BUSI classification is trained jointly with segmentation or auxiliary tasks [28]. Table ¿tbl:tab:busi95encoders? reports BUSI test accuracy, including BU-Mamba with and without AttrGuide, and the accompanying multi-task plot shows how attribute guidance influences ACC and Macro F1.

4pt

4pt

4pt

Generality. AttrGuide consistently improves performance across datasets and backbones, and remains effective when plugged into a strong baseline (BU-Mamba), improving BUSI test accuracy from 87.86% to 88.72% (Tables ¿tbl:tab:busi95encoders? and 1).

Low cost. The plug-in attribute-guided branch adds only marginal training-time and parameter overhead on BUSI (Table 2), while delivering consistent gains across backbones (Table 1).

6pt

Interpretability. Figure 3 shows that the attribute-guided branch predicts clinically meaningful attributes with high accuracy (87.56% fetal, 83.29% BUSI), providing human-interpretable cues for verification.

Ablation. Table 3 verifies that learnable fusion is crucial: the full model achieves the best accuracy (+5.8 over Cls-only) and outperforms naive averaging, showing that both branches are needed.

4 Conclusion↩︎

In this paper, we introduce an attribute-guided dual-branch framework to enhance ultrasound classification by bridging visual features with clinical priors. We demonstrate that our approach consistently boosts the performance of SOTA backbones—for instance, improving BU-Mamba’s accuracy on the BUSI dataset from 87.86% to 88.72%—while remaining highly efficient with less than 5% additional training time. From the results, we mainly conclude that: (1) the injection of domain-agnostic attributes provides essential evidence that bolsters both model robustness and clinical interpretability; and (2) the adaptive fusion module is critical for synergistically integrating global and attribute-based semantics. In the future, we intend to extend this low-cost, plug-and-play enhancement to a broader range of medical imaging tasks.

References↩︎