Image classification is a classic computer vision task that aims to predict a category label for each image. Recent years have witnessed the tremendous success of image classification due to the rapid development of deep learning [1], especially for deep classification networks [2]–[4]. Conventionally, a classification network is trained with cross-entropy loss on one-hot labels, such that it predicts almost orthogonal category labels for images in different classes. Such a scheme focuses on the differences of categories, yet it may encounter challenges in effectively capturing inter-class similarity [5], [6]. Hierarchical classification, contrast to classic image classification, attempts to classify the object into a hierarchy of categories, with each hierarchy represents one specific level of concept abstraction. For example, birds can be categorized into a three-level hierarchy of order, family and species according to their taxonomic characters, as shown in Fig. 1. In hierarchical classification learning, the labels can be organized according to the domain knowledge (e.g, WordNet database [7]) or automatically generated by algorithms [8]–[12].

Compared to traditional multi-class classification, hierarchical classification pays more attention to exploit rich semantic correlation among categories [13], in order to improve the performance of traditional classification tasks [14], [15] or mitigate the severity of prediction mistakes [16], [17], i.e., making the incorrectly classified samples fall within semantically related categories. Take the example of making diagnosis for one malignant tumor, mistaking it as another malignant one in the same subtype should be much less serious than mistaking it as one benign tumor, as such a mistake would have crucial implications in terms of the follow-up diagnosis and treatment.

Hierarchical classification has been proved effective in many visual scenarios, which can be traced back to traditional linear classifiers [19]–[22]. Recently, hierarchical classification has attracted much attention in deep learning [23]. Most of the existing works [16], [24]–[27] commonly fall in a multi-task learning manner, which employs a hierarchy-agnostic feature extractor to learn universal representations across hierarchies and parameterizes a set of hierarchy-specific heads to encode private representations for each hierarchy. For example, a deep network was proposed in [24] to classify coarse- and fine-level categories through two different branches. In [16], a granular network was proposed for food image classification, where a single network backbone was shared by multiple fully-connected layers with each one responsible for the label prediction within one hierarchy. Chen et al. [26] likewise employed a multi-head architecture to output hierarchy-specific prediction, but instead introduced an attention mechanism to guide label prediction of fine-level hierarchies through that of the coarse-level hierarchies. Furthermore, Alsallakh et al. [25] used a standard deep architecture to fit the finest-level labels, and added branches at the intermediate layers to fit the coarser-level labels. Recently, Chang et al. [27] pointed out that fine-level features benefit coarse-grained classifier learning, yet coarse-level label prediction is harmful to fine-level feature learning. They in turn proposed to disentangle coarse- and fine-level features by leveraging hierarchy-specific classification heads, and further force finer-level features (with stopped gradient) to participate in coarser-level label predictions. Such a simple method has achieved significant improvement in fine-level image recognition.

Despite significant progress, these recent methods predict category labels for each hierarchy using independent classification heads. This implies that all class hierarchies are statistically independent (to be elaborated in 2), such that category correlations across hierarchies are encoded solely by a shared feature extractor. Such a multi-task learning strategy is often sub-optimal, as it may lead to the common issue of under-transfer as knowledge can not be sufficiently transferred across different classification heads [28]. Moreover, these recent methods primarily focus on a restrictive setting where labels for all hierarchies are provided during the training phase. In contrast, the more challenging yet realistic setting, where samples are labeled at only partial hierarchies, remains under-explored [29].

To address the aforementioned issues, this paper aims to develop a deep hierarchical classification method capable of explicitly capturing category correlations embedded in class hierarchies. Unlike prior work that implicitly encodes category correlations through a universal feature extractor, our approach explicitly connects category predictions between adjacent hierarchies. We achieve this by using transition probabilities of category labels from one hierarchy to the next, based on the input image or features. Concretely, we propose Label Hierarchy Transition (LHT), a unified probabilistic approach, to recursively predict the labels from fine- to coarse-level class hierarchies. The LHT model contains two components: (1) an arbitrary existing classification network, which is used to predict the labels for the finest-level hierarchy; (2) a transition network, which generates a label hierarchy transition matrix for each hierarchy (except for the finest-level one). Each element of the matrix denotes the transition probability of one class at the corresponding hierarchy given the other class of the adjacent finer-level hierarchy. For each coarser-level hierarchy, the category prediction can be formulated as the product of its corresponding transition matrix and the prediction score of its adjacent finer-level hierarchy. The two components are jointly trained in an end-to-end manner, and the learned transition matrices for coarser-level hierarchies theoretically provide supervision for the finer-level ones. This enables our proposed LHT framework significantly improve the performance of hierarchical classification, especially in scenarios where a large number of samples have labels only at partial hierarchies.

Moreover, the hierarchical cross-entropy loss used in prior works based on multi-task learning tends to produce orthogonal predictions at each hierarchy and is not conducive to the learning of correlations among different categories. To address this challenge, we further propose a confusion loss to explicitly regularize the label hierarchy transition matrices, which facilitates the network to learn category correlations from all class hierarchies during the training phase.

In summary, the main contributions are four-fold:

• We propose a unified deep classification framework to address the hierarchical classification problem. Contrast to predicting the labels for each hierarchy, we propose to learn label hierarchy transition matrices, which encode the likelihood of similarity between classes across different label hierarchies. To the best of our knowledge, we are the first to utilize deep networks to explicitly learn transition probabilities across class hierarchies.

• We provide grounded theoretical understanding that explains how our method potentially works in real-world scenarios. Specifically, we prove that the proposed method facilitates bi-directional information flow between fine- and coarse-level hierarchies, enabling both granular detail enhancement and broader contextual integration. Furthermore, the learned transition matrices at coarse-level hierarchies serve as supervisory signals for fine-level ones, offering a practical advantage in missing-label scenarios.

• We propose a confusion loss to directly regularize the negative entropy of the column space of label hierarchy transition matrices, which further encourages the network to learn correlations among class hierarchies, and alleviates the issue of over-confident predictions arising from cross-entropy loss.

• Our LHT method can be readily applied to any existing classification network, and the resulting model can be trained in an end-to-end manner. Extensive experiments on a series of hierarchical classification tasks demonstrate the superiority of our approach beyond the prior state-of-the-arts.

The rest of this paper is organized as follows. Section 1 revisits related works. Section 2 describes the formulation of the proposed LHT framework as well as its training algorithm. Section 3 presents experimental results and analysis in detail. Section 4 provides a realistic application of our method in computer-aided diagnosis. Section 5 summarizes the final conclusion.

1 Related Work↩︎

Hierarchical classification [30]–[32], exploiting a hierarchical structure of labels for classification task, has been extensively explored in natural language processing [33]–[35], computer vision and computer-aided diagnosis [36]–[38]. In the existing literature, hierarchical classification is mainly related to three aspects of research: hierarchical architecture, hierarchical loss function, and label embedding based methods. Next, we discuss related studies with respect to these three aspects.

1.1 Hierarchical architecture↩︎

A common way of designing a hierarchical architecture is to alleviate the computational burden of traditional linear classifiers when the number of recognition categories become huge [22], [39]–[42]. The core idea was to create a tree-like classifier whose leaf nodes were used for the prediction of the given categories and internal nodes for the cluster of categories.

Most recently, deep learning based methods are increasingly used to address hierarchical classification [16], [24]–[27], [29], [43]. These methods commonly fall into a multi-task learning manner, and adopt shared-specific feature representation learning to fit labels across different levels of hierarchy. Such a flexible architecture is advantageous in learning low-level features on one hand, while on the other hand, it can easily incorporates canonical inclusion relations between the parent node (super-class) and its child nodes (sub-classes) [27], [43] or exclusion relations [29] between the sibling nodes at the same hierarchy into the network design. Contrary to these works, our proposed LHT framework learns the conditional probabilities between two arbitrary nodes within adjacent hierarchies to encode the affinity relations.

1.2 Hierarchical loss function↩︎

A range of literature on hierarchical classification resorts to incorporating the class hierarchies in the loss function. In [44], the metrics associated with each node of the taxonomy tree were learned based on a probabilistic nearest-neighbour classification framework. In [45], the tree-structure hierarchies were used to impose prior on the parameters of the classification layer for knowledge transfer between classes, and similar strategy was proposed by Zhao et al. [21] in the traditional linear classifier. In [23], various label relations (e.g., mutual exclusion and subsumption) were encoded as conditional random field to pair-wisely capture the correlation between classes. In [46], max constraint loss was proposed to produce predictions coherent with the hierarchy constraint for multi-label hierarchical classification. Recently, Bertinetto et al. [17] proposed a hierarchical loss that could be viewed as a weighted cross-entropy loss, and the weights were defined according to the depth of the label tree. In parallel to our method, chen et al. [29] proposed a probabilistic loss to encode the label relation within each hierarchy. Unlike these methods [47], [48], we combine cross-entropy with a novel confusion loss to regularize the learned transition matrices, thereby fully exploiting class correlations across hierarchies. Besides, in hierarchical segmentation [47]–[49], prior works such as HSSN [47], [48] incorporates class hierarchies by directly constraining the output space through a tree-min loss to produce hierarchy-coherent predictions. In contrast, our LHT takes a probabilistic approach by learning transition matrices to model inter-class similarities, which not only enforces hierarchy coherence, but also leverages cross-level correlations more flexibly, especially under missing-label scenarios.

1.3 Label embedding↩︎

Label embedding aims to encode the correlation information among classes into the traditional one-hot labels, which has been widely explored in multi-class classification [5], [6], [50], [51]. For example, Label Smoothing presented in [5], [6] softens the one-hot label by assigning a slightly lower probability to the target class and distributes the remaining probability evenly across the non-target classes. In hierarchical classification, label embedding techniques typically map one-hot labels to vectors, where the relative positions of these vectors encode the structural information embedded in the class hierarchy. Specifically, Bertinetto et al. [17] proposed Soft Labels, an embedding-based method that softens one-hot labels using a metric derived from the lowest common ancestor (LCA) height within a given label tree. Dhall et al. [52] employed entailment cones to learn order-preserving label embeddings and subsequently integrated them with image embeddings for hierarchical classification by using the transitivity of the embeddings across levels in the hierarchy. Although label embedding is a potential strategy to capture the correlation across class hierarchies, in this paper, we attempt to develop a unified deep classification framework for explicitly encoding correlation of hierarchical classification.

2 The Proposed Method↩︎

In this section, we first introduce the problem formulation of hierarchical classification and the preliminaries of our work. We then provide an overview of our proposed LHT framework that explicitly encodes category correlations across distinct hierarchies. Next, we present the architecture of the proposed transition network. Finally, we introduce the proposed confusion loss to encourage correlation learning across hierarchies.

2.1 Problem Formulation and Preliminaries↩︎

In hierarchical classification tasks, a given image \(\mathbf{x}\) is typically annotated with a sequence of \(K\) labels \((y^1, y^2, \ldots, y^K)\), where \(K\) represents the number of label hierarchies. Each \(y^k \in \{1, 2, \ldots, C_k\}\) is the label within the \(k\)-th hierarchy, with \(C_k\) denoting the number of categories of this hierarchy. We denote \(\mathbf{y}^k\) is the corresponding one-hot vector of \(y^k\). Without loss of generality, we define \(k=1\) as the finest-level hierarchy, and assume the granularity becomes coarser as \(k\) increases. Accordingly, the number of classes decreases with larger \(k\), i.e., \(C_k > C_{k+1}\). The goal of hierarchical classification is to learn a classifier to predict hierarchical labels for each unknown \(\mathbf{x}\).

The multi-task approach to addressing hierarchical classification commonly involves two steps: ) it first employs a hierarchy-agnostic feature extractor \(f_\omega\) parameterized by \(\omega\) to learn a universal representation \(\hat{\mathbf{x}}=f_\omega(\mathbf{x})\) across hierarchies, and ) for the \(k\)-th hierarchy, it uses a hierarchy-specific classification head \(g^k_{\theta^k}\) parameterized by \(\theta^k\) to predict the corresponding classification distribution \(\mathbf{p}^k=g^k_{\theta^k}(\hat{\mathbf{x}})\). Note that here, with some abuse of notation, we define \(\mathbf{p}^k=[p(Y^k=1|\mathbf{x}),\cdots,p(Y^k=C_k|\mathbf{x})]^T\), where \(Y^k\) denotes a random variable representing the label of the \(k\)-th hierarchy. The involved parameters are jointly optimized using a hierarchical cross-entropy loss \[\mathcal{L}_{{\rm CE}}=\sum_{n=1}^N\sum_{k=1}^K H(\mathbf{p}^k_n, \mathbf{y}^k_n), \label{eq:hierarchical95ce}\tag{1}\] where \(\{\mathbf{x}_n,(\mathbf{y}_n^1,\cdots,\mathbf{y}_n^K)\}_{n=1}^N\) represents a batch of \(N\) training samples, and \(H(\cdot,\cdot)\) is the cross-entropy loss defined as \(H(\mathbf{p}, \mathbf{y})=\sum_i \mathbf{y}[i]\log \mathbf{p}[i]\), with \(\mathbf{p}[i]\) denoting the \(i\)-th entry of \(\mathbf{p}\).

1 is equivalent to solve the negative log-likelihood for \(p(y^1,\cdots,y^K|\mathcal{I})=\prod_{k=1}^K p(y^k|\mathcal{I})\). The detailed derivation can be found in Appendix A.1. This implies that all class hierarchies are considered statistically independent of each other, such that category correlations across hierarchies are encoded solely by the shared feature extractor \(f_\omega(\cdot)\). Such a multi-task learning strategy is often sub-optimal due to the following two reasons: ) It fails to explicitly model the hierarchy/task relationships, which may lead to the common issue of under-transfer as knowledge can hardly be transferred across different classification heads [28]. ) In missing-label scenarios where most samples are labeled at only partial hierarchies, there is a risk that the classification head associated with the finest-level hierarchy might overfit, as it is trained on fewer samples compared to the coarser-level hierarchies.

In light of the aforementioned reasons, a natural question is how to explicitly model hierarchical relationships within a multi-task deep learning framework to facilitate information transfer across different classification heads, especially in challenging scenarios with missing labels.

2.2 Label Hierarchy Transition↩︎

The goal of this paper is to propose a unified probabilistic framework to explicitly encode category correlations across hierarchies into the existing multi-task learning framework. Unlike previous works based on the assumption of independent distributions across different hierarchies, we mathematically consider that the label \(Y^{k}\) of the \(k\)-th hierarchy is statistically dependent on \(Y^{k-1}\) and the sample \(\mathbf{x}\) itself. It is natural to encode such correlations between adjacent hierarchies through the conditional distribution \(p(Y^k|Y^{k-1},\mathbf{x})\). Subsequently, we can formulate the distribution \(p(Y^k|\mathbf{x})\) by marginalizing over the variable \(Y^{k-1}\): \[p(Y^k|\mathbf{x})\!=\!\sum_{i=1}^{C_{k-1}}p(Y^k|Y^{k-1}=i,\mathbf{x})\cdot p(Y^{k-1}=i|\mathbf{x}), \label{eq:transition}\tag{2}\] where \(p(Y^{k-1}|\mathbf{x})\) is the label distribution of the \((k\!-\!1)\)-th hierarchy with \(k=2,\cdots,K\).

For simplicity, here we define \[t^k_{ij}\triangleq p(Y^k=i|Y^{k-1}=j,\mathbf{x}), \label{eq:probability}\tag{3}\] and then we have a \(C_k\times C_{k\!-\!1}\) matrix \[\mathbf{T}^k=\begin{bmatrix} \begin{array}{cccc} t^k_{11} & t^k_{12} & \cdots & t^k_{1C_{k-1}} \\ t^k_{21} & t^k_{22} & \cdots & t^k_{2C_{k-1}} \\ \vdots & \vdots & \ddots & \vdots \\ t^k_{C_k1} & t^k_{C_k2} & \cdots & t^k_{C_kC_{k-1}} \\ \end{array} \end{bmatrix}.\] We call this matrix Label Hierarchy Transition Matrix (or abbreviated as Transition Matrix), since it defines the probabilities of transitions from fine-grained labels to coarse-grained labels and we can get a glimpse of the transition status between the adjacent class hierarchies. Note that the elements of \(\mathbf{T}^k\) are positive and each column sums to 1, i.e., \(\sum_{i=1}^{C_{k}}t^k_{ij}=1\) for each \(j\in\{1,\cdots,C_{k-1}\}\).

2.2.1 Model Formulation↩︎

In this paper, the core of our Label Hierarchy Transition model is to predict the label distribution \(p(Y^k|\mathbf{x})\) for each hierarchy, by jointly learning the two kinds of conditional probability involved in Eq. 2 . Notably, 2 inherently defines a fine-to-coarse learning paradigm, as illustrated in Fig. 2. Within this paradigm, the finest-level label distribution is first fitted, and the coarser-level ones are subsequently learned, conditioned on the input image and its adjacent finer-level label distribution. Specifically:

• For the finest-level hierarchy, we use a vanilla Classification Network \(f_\omega\) with parameters \(\omega\) to predict its label distribution \(\mathbf{p}^1\) as follows: \[\mathbf{p}^1=\sigma(\mathbf{z})=\sigma(f_\omega(\mathbf{x})), \label{eq:fine95level95prediction}\tag{4}\] where \(\mathbf{p}^1\!=\![p(Y^1\!=\!1|\mathbf{x}), \cdots, p(Y^1\!=\!C_1|\mathbf{x})]^T\), \(\mathbf{z}\) denotes the logits of \(\mathbf{x}\), and \(\sigma\) is the softmax function.

• For the \(k\)-th level hierarchy with \(k=2,\cdots,K\), we design a light-weight Transition Network \(g^k_{\theta^k}\) parameterized by \(\theta^k\) to predict its transition matrix \(\mathbf{T}^k\), such that \(\mathbf{T}^k=g^k_{\theta^k}(\mathbf{x})\). Thus, the label distribution \(\mathbf{p}^k\) can be formulated as \[{\mathbf{p}}^k=\mathbf{T}^k{\mathbf{p}}^{k-1}. \label{eq:coarse95level95prediction}\tag{5}\] The architectural designs of \(g^k_{\theta^k}\) will be elaborated in the next subsection.

From 4 5 , we can get the label distributions of all hierarchies. This allows for the joint optimization of the parameters involved in both the Classification Network and the Transition Networks, i.e., \(\{\omega, \theta^2,\cdots,\theta^K\}\), in an end-to-end manner. Given a batch of \(N\) training samples \(\{\mathbf{x}_n, (\mathbf{y}_n^1, \mathbf{y}_n^2,\cdots,\mathbf{y}_n^K)\}_{n=1}^N\), the hierarchical cross-entropy loss can be formulated as follows: \[\begin{align} \mathcal{L}_{{\rm CE}}&=\sum_{n=1}^N\sum_{k=1}^KH({\mathbf{p}}_n^k, \mathbf{y}^k_n) \\ &=\sum_{n=1}^N\Bigl(H\bigl({\mathbf{p}}^1_n,\mathbf{y}^1_n\bigr) \!+\!\sum_{k=2}^K H\bigl(\mathbf{T}^k_n{\mathbf{p}}_n^{k-1}, \mathbf{y}^k_n\bigr)\Bigr). \end{align} \label{eq:hierarchical95ce95lht}\tag{6}\] In addition to 6 , we propose a novel confusion loss to enhance the learning of category correlations across all class hierarchies during the training phase. The details will be discussed in 2.4.

2.2.2 Theoretical Analysis.↩︎

Our LHT framework defined by 4 5 offers the following theoretical guarantees: ) LHT facilitates the transition of information from the finest-level hierarchy to the coarser-level ones via forward propagation; ) LHT enables the transition of true label information from coarser-level hierarchies to the finer-level ones through backpropagation; ) The learned transition matrices for coarser-level hierarchies theoretically provide supervised signals for the finer-level hierarchies, thereby benefiting the training process for missing-label scenarios.

For ease of notation, we only consider the single-sample case (\(N=1\)) for 6 , but the derivation for the multi-sample case is straightforward. We first introduce the definition of the transition matrix from the \(1\)-st hierarchy to the \(k\)-th hierarchy.

Given the definition of \({\mathbf{T}}^{k\leftarrow1}\), we show that our proposed LHT facilitates the transition of information from the finest-level hierarchy directly to any coarser-level ones as in the following 1.

The proposition can be proved by verifying that each \((i,j)\)-th element of \({\mathbf{T}}^{k\leftarrow1}\) represents the transition probability \(p(Y^k=i|Y^1=j,\mathbf{x})\). We provide the proof in Appendix A.2. 1 indicates that the label prediction for the \(k\)-th hierarchy is conditioned on the information derived from all the preceding \(k-1\) hierarchies. This, on the one hand, demonstrates that our LHT framework facilitates the transition of information from fine-level hierarchies to coarse-level ones through the forward propagation of the framework. On the other hand, since LHT optimizes all parameters end-to-end via 6 , it implies that LHT also benefits the transition of true label information from coarser-level hierarchies to the finest-level one through gradient backpropagation, as illustrated in the following proposition.

The proof of 2 is provided in Appendix A.3. ?? demonstrates that the coarse-level hierarchies influence the finest-level hierarchy by adjusting its logit \(\mathbf{z}\) using the true label \(y_k\) from the \(k\)-the hierarchy in the end-to-end training phase. Intuitively, if \(\mathcal{G}_j^k < 0\), reducing the loss \(\mathcal{L}^k\) by updating parameters along the negative gradient direction (\(-\mathcal{G}_j^k\)) will effectively increase \(\mathbf{z}[j]\); conversely, if \(\mathcal{G}_j^k > 0\) the same process will lead to a decrease in \(\mathbf{z}[j]\). In fact, the Transition Networks can rectify inaccurate predictions by adjusting the logits of the finest-level hierarchy during training. This can be formalized in the following corollary.

This corollary indicates that reducing \(\mathcal{L}^k\) along the direction (\(-\mathcal{G}^k_{j^*}\)) can guide the logits \(\mathbf{z}\) of the finest-level hierarchy towards the value that yield higher probability for the subclass most visually similar to \(y^k\) during training.

Intrinsically, in 2, \(\mathcal{G}^k_j\) measures the distance between two distributions. By the definition of transition probability in 3 , \(\mathcal{G}^k_j\) can be rewritten as: \[\label{eq:bayes} \begin{align} \mathcal{G}^k_j & =\bigl(1-\frac{t^{k\leftarrow{1}}_{y^k j}}{\mathbf{p}^k[y^k]}\bigr)\mathbf{p}^1[j] \\ & = p(Y^1=j|\mathbf{x})-\frac{p(Y^k=y^k|Y^1=j,\mathbf{x})p(Y^1=j|\mathbf{x})}{p(Y^k=y^k)} \\ & = p(Y^1=j|\mathbf{x})-p(Y^1=j|Y^k=y^k, \mathbf{x}), \end{align}\tag{7}\] where the third equal sign holds due to Bayes’ rule. ?? 7 further demonstrate that the \(k\)-th hierarchy can facilitate logit adjustment for the finest-level one by modifying the distance between \(p(Y^1|\mathbf{x})\) and \(p(Y^1|Y^k=y^k,\mathbf{x})\) during the end-to-end training phase. Especially in missing-label scenarios, ?? 7 indicate that the coarser-level hierarchies can provide supervision for the finest-level hierarchy. In fact, the adjustment induced by the \(k\)-th hierarchy positively impacts the label prediction of the finest-level hierarchy. This can be summarized in the following theorem:

We present the proof of 1 in Appendix A.5. Since \(p(Y^1|Y^k=y^k,\mathbf{x};\omega,\theta)\) represents the predicted label distribution at the finest-level hierarchy given that the \(k\)-th hierarchy’s prediction is the ground-truth label, this theorem implies that our LHT framework aims to encourage the prediction of the Classification Network \(p(Y^1|\mathbf{x};\omega)\) to better match the conditional distribution \(p(Y^1|Y^k=y^k,\mathbf{x};\omega,\theta)\). In summary, it concludes that:

• The coarse-level hierarchies can provide supervision for the finest-level hierarchy by implicitly minimizing the distance between the two probability distributions \(p(Y^1|\mathbf{x};\omega)\) and \(p(Y^1|Y^k=y^k,\mathbf{x};\omega,\theta)\).

• The Classification Network tends to predict the labels for the finest-level hierarchy while maintaining the accuracy of the \(k\)-th hierarchy’s prediction during the end-to-end training phase.

2.3 Architectural Designs↩︎

As aforementioned, the proposed LHT framework mainly consists of the Classification Network and the Transition Networks.

Firstly, we emphasize that for the Classification Network as formulated in 4 , we employ a well-designed existing network, such as ResNet-50 [4], to predict category labels for the finest-level hierarchy. On one hand, this aims to insure a fair comparison of classification performance at the finest-level hierarchy with traditional hierarchical classification methods based on the multi-task learning paradigm, while on the other hand, the target is to demonstrate that our proposed LHT is a plug-in for enhancing conventional classification networks through hierarchical supervision, especially in missing-label scenarios.

Next, we elaborate the architectural designs for the Transition Networks. As shown in 2 (b), for hierarchy \(k\), the network \(g_{\theta^k}^k\) is responsible for producing the transition matrix \(\mathbf{T}^k\). A natural way to implement the network is to use a single fully-connected (FC) layer, which can be formulated as: \[\label{eq:vanilla95transition95net} \mathbf{T}^k = g^k_{\theta^k}(\mathbf{x}):=\texttt{softmax}\bigl(\texttt{reshape}(\mathbf{W}^k\hat{\mathbf{x}}+\mathbf{b}^k)\bigr),\tag{8}\] where \(\hat{\mathbf{x}}\in\mathbb{R}^d\) is a high-level embedding of \(\mathbf{x}\) branched from the Classification Network, \(\theta^k=\{\mathbf{W}^k,\mathbf{b}^k\}\) are the parameters of \(g^k_{\theta^k}\), with \(\mathbf{W}^k\in\mathbb{R}^{(C_{k}\times C_{k-1})\times d}\) and \(\mathbf{b}^k\in\mathbb{R}^{(C_{k}\times C_{k-1})}\) denoting the weight matrix and bias term of the FC layer, respectively. The operator \(\texttt{reshape}(\cdot)\) rearranges the elements of a \((C_{k}\times C_{k-1})\) vector into a \(C_{k}\times C_{k-1}\) matrix, and \(\texttt{softmax}(\cdot)\) normalizes the columns of the input matrix using the softmax function for satisfying \(\sum_{i=1}^{C_k}\mathbf{T}_{ij}^k=1\).

From 8 , we can observe that the Transition Networks capture sample-specific information through the weight matrix \(\mathbf{W}^k\) and sample-agnostic information through the bias term \(\mathbf{b}^k\). This indicates that the learned transition probabilities are conditioned on both the input sample and categorical knowledge. Therefore, the architectural design of the Transition Network complies with 3 .

Remark 1: From the given hierarchical labels, we can obtain a deterministic transition matrix \(\mathbf{S}^k\), where only one element is 1 in each column. This indicates that \(\mathbf{S}^k_{i,j}=1\) if category \(j\) at hierarchy \(k\!-\!1\) is a subclass of category \(i\) at hierarchy \(k\), and \(\mathbf{S}^k_{i,j}=0\) otherwise. Instead, in Eq. 8 , we aims to model \(p(y^k|y^{k-1},\mathbf{x})\), which can capture data-driven correlation from image context and hierarchy prior. In fact, if we replace \(g_{\theta^k}^k(\hat{\mathbf{x}})\) with \(\mathbf{S}^k\) in Eq. 6 (referred to as LHT_naive), it will result in a significant performance degradation, as the hierarchical relationships derived from a label tree are not always visually grounded. For example, the Aircraft dataset [53], whose hierarchical relationships are based on a maker-family-model organization, is defined more by semantics than by visual similarity. As shown in 3, LTH_naive achieves notably lower performance than our LHT model. To further demonstrate the importance of adaptively learning visual correlations, we constructed a new dataset, Perturbed-Aircraft, which contains a label hierarchy that is less visually consistent than the original dataset Aircraft ². Under this challenging scenario, our LHT model surpasses LHT_naive by 5.1%, further demonstrating the effectiveness of learning data-driven transition matrices to capture visual category correlations.

A lightweight design for \(g_{\theta^k}^k\): The architecture of \(g_{\theta^k}^k\) designed in 8 requires a large number of parameters. To prevent an explosion of parameters, we propose a simple design for \(g_{\theta^k}^{k}\) as shown in 4.

Firstly, we replace the original FC layer with two smaller ones, which means we use a low-rank approximation to estimate \(\mathbf{W}^k\hat{\mathbf{x}}\) in 8 . This can be formulated as: \[\label{eq:low95rank95approx} \texttt{reshape}(\mathbf{W}^k\hat{\mathbf{x}}) :\approx \underbrace{\texttt{reshape}(\mathbf{W}^k_1\hat{\mathbf{x}})}_{\mathbb{R}^{C_k\times s}}\otimes\underbrace{\texttt{reshape}(\mathbf{W}^k_2\hat{\mathbf{x}})}_{\mathbb{R}^{C_{k-1}\times s}},\tag{9}\] where \(\mathbf{W}^k_1\in\mathbb{R}^{(C_k\times s)\times d}\) and \(\mathbf{W}^k_2\in\mathbb{R}^{(C_{k-1}\times s)\times d}\) are parameters of the two \(\texttt{FC}\) layers, with \(s\) denoting a hyper-parameter in order to control the rank of the above approximation. The operation \(\mathbf{A}\otimes \mathbf{B}\) denotes the matrix multiplication between \(\mathbf{A}\) and \(\mathbf{B}^\top\) (i.e., the transpose of \(\mathbf{B}\)). 9 indicates that \(\mathbf{W}\hat{\mathbf{x}}\) is approximated with a rank \(s\) matrix, which reduces its parameter complexity from \(\mathcal{O}\bigl(d(C_k\times C_{k-1})\bigr)\) to \(\mathcal{O}\bigl(ds(C_k + C_{k-1})\bigr)\). We initialize \(s\) as 2, which ensures efficiency and good performance throughout our experiments. The ablation study on \(s\) can be found in 3.1.3.

Next, we aim to design a lightweight bias term for replacing \(\texttt{reshape}(\mathbf{b}^k)\) in 8 as it involves a relatively large parameter complexity \(\mathcal{O}(C_k\times C_{k-1})\). As previously discussed, this bias term plays a crucial role in capturing sample-agnostic information and encoding categorical knowledge to satisfy 3 . Interestingly, the deterministic transition matrix \(\mathbf{S}^k\), derived from the given hierarchical labels, can serve as a valuable prior for encoding categorical granularity information. Therefore, we replace \(\texttt{reshape}(\mathbf{b}^k)\) with the following simplified formulation: \[\label{eq:bias95term} \texttt{reshape}(\mathbf{b}^k):\approx \gamma \mathbf{S}^k + \beta,\tag{10}\] where \(\gamma\) is a scaling parameter to ensure consistency in magnitude between \(\gamma\mathbf{S}^k\) and \(\mathbf{W}^k\hat{\mathbf{x}}\), and \(\beta\) is a shifting parameter. From 10 , we can see that this new bias term introduces negligible parameters compared to the original version.

By substituting 9 10 into 8 , we obtain a lightweight formulation of the Transition Network: \(\label{eq:light95transition95net} g_{\theta^k}^k(\mathbf{x})=\texttt{softmax}\bigl(\texttt{reshape}(\mathbf{W}^k\hat{\mathbf{x}})+\texttt{reshape}(\mathbf{b}^k)\bigr)\), with the parameters \(\theta^k=\{\mathbf{W}_1^k,\mathbf{W}_2^k,\gamma,\beta\}\). We emphasize that our basic principle in designing the transition network is to integrate both sample-specific and sample-agnostic information, as investigated in 3 .

2.4 Confusion Loss↩︎

As aforementioned in 2.2, the parameters involved in the Classification Network and the Transition Networks, i.e., \(w\) and \(\theta=\{\theta^k\}_{k=2}^K\), can be jointly optimized end-to-end using the loss 6 . However, the network training by minimizing this objective function remains the following challenges: () The manually annotated labels within each hierarchy are one-hot, and such a cross-entropy loss encourages the network to move all categories toward orthogonality at each hierarchy, counteracting the inter-class similarity introduced by class hierarchies; () From the perspective of numerical optimization, there are infinite combinations of \(\mathbf{T}^k\) and \({\mathbf{p}}^{k-1}\), which easily make the learned transition matrices to collapse to a trivial solution.

To address the aforementioned challenges, we further introduce an additional regularization loss on transition matrices \(\{\mathbf{T}^k\}_{k=2}^K\), which we call the confusion loss \(\mathcal{L}_{{\rm Conf}}\). We herein revisit the label hierarchy transition matrices involved in our LHT model. The column vector (e.g., the \(j\)-th one) of transition matrix \(\mathbf{T}^k\) represents the label distribution of the \(k\)-th hierarchy when the sample is labeled as class \(j\) at the \((k\!-\!1)\)-th hierarchy. Each element of this vector denotes the likelihood of similarity, or confusion, between two classes across two adjacent hierarchies. The proposed confusion loss aims to encourage such class confusion to counter the over-confident prediction induced by the cross-entropy loss. Mathematically, for a batch of \(N\) training data \(\{\mathbf{x}_n, (y_n^1, y_n^2,\cdots,y_n^K)\}_{n=1}^N\), the confusion loss is defined as the negative entropy of the column space spanned by the transition matrices, i.e., \[\mathcal{L}_{\rm Conf}=\sum_{n=1}^N\sum_{k=2}^K\sum_{j=1}^{C_{k-1}}\frac{1}{C_{k-1}} \Bigl\langle\mathbf{T}_n^k[:,j],\log \mathbf{T}_n^k[:,j]\Bigr\rangle, \label{eq:loss95conf}\tag{11}\] where \(\mathbf{T}_n^k\) is the predicted transition matrix for \(\mathbf{x}_n\) and \(\mathbf{T}_n^k[:,j]\) denotes its \(j\)-th column, \(\log(\cdot)\) is element-wise logarithm operation for vectors, and \(\langle \cdot, \cdot\rangle\) is the inner product operation of vectors. Through minimization of Eq. 11 , the conditional distribution represented by each column of \(\mathbf{T}_n^k\) is encouraged to have a higher entropy (i.e., this distribution inclines to approach uniform distribution). In other words, this introduces confusion in fine-to-coarse label transitions during training, thereby encouraging the Transition Networks to encode sample-specific information into transition matrices and alleviating the over-confident prediction induced by the hierarchical cross-entropy loss. Furthermore, since \(\mathbf{T}^{k\leftarrow 1}\) can be factorized as \(\mathbf{T}^{k\leftarrow 1}=\mathbf{T}^k \cdots \mathbf{T}^2\) (Proposition 1), applying confusion loss to each transition matrix may implicitly facilitate the modeling of inter-class similarities between hierarchy 1 and \(k\).

Combined Eq. 6 and Eq. 11 , the overall training loss can be formulated as \[\mathcal{L}=\mathcal{L}_{{\rm CE}}+\lambda \mathcal{L}_{{\rm Conf}}, \label{eq:loss}\tag{12}\] where \(\lambda\) is hyper-parameter for trade-off between the cross-entropy term \(\mathcal{L}_{{\rm CE}}\) and the confusion regularization term \(\mathcal{L}_{{\rm Conf}}\). All the parameters involved in our LHT can be trained in an end-to-end manner. The training algorithm is summarized in Algorithm 5.

3 Experiments on Natural Images↩︎

In this section, we first demonstrate the effectiveness of the proposed LHT in supervised/semi-supervised hierarchical classification tasks following [29]. Specifically, the labels of training samples are given at all levels of the class hierarchy under the supervised setting. However, under the semi-supervised setting, the labels are observed at different levels of the class hierarchy, indicating that part of samples are only observed at internal levels of the whole hierarchies³. Next, we demonstrate the applicability of the proposed LHT on large-scale low-shot learning where transferable visual features are learned from a class hierarchy to encode the semantic relations between source and target classes. We also provide thorough ablation studies to gain insight into our method by showing the efficacy of each key component.

3.1 Supervised/Semi-supervised Hierarchical Learning↩︎

3.1.1 Experimental Setups↩︎

Datasets. In our experiments, we evaluate our method on three public benchmark datasets: CUB-200-2011 [18], Aircraft [53] and Stanford Cars [54], where the label hierarchies are publicly available or constructed by Chang et al. [27] according to their lexical relationships in Wikipedia. The details of these datasets are as follows:

• CUB-200-2011⁴ is a dataset of bird images mainly introduced for the problem of fine-grained visual categorisation methods. It contains 11,877 images belonging to 200 bird species. For hierarchical classification evaluation, the labels are re-organized into three-level hierarchy with 13 orders, 38 families and 200 species by tracing their biological taxonomy [27]. For example, the original label “Brewer Blackbird" is spanned into (”Passeriformes”, “Icteridae”, “Brewer Blackbird”) in the order of (Order, Family, Species).

• Aircraft⁵ consists of 10,000 images across 100 airplane classes, with a data ratio of approximately 2:1 between the training and test sets. The dataset includes a three-level label hierarchy with 30 makers (e.g., “Boeing”), 70 families (e.g., “Boeing 767”) and 100 models (e.g., “767-200”). As a result, this dataset can be readily used in our experiments without any modification.

• Stanford Cars⁶ contains 8,144 training and 8,041 test images across 196 car classes. It is re-organized into two-level label hierarchy, i.e., 9 car types and 196 car classes as introduced in [27]. For example, the class label “BMW X6 SUV 2012” are spanned into (“SUV”, “BMW X6 SUV 2012”). We use the standard public train/test splits, and not any bounding box/part annotations are involved in all our experiments.

For the supervised hierarchical classification setting, we train the model using the complete set of labels at all hierarchical levels for each training sample. For the semi-supervised hierarchical classification setting, in order to simulate a lack of domain knowledge, we randomly select 30%, 50%, 70%, and 90% samples (denoted as \(\mathcal{R}=90\%\) for example) to mask out their labels at the finest-level hierarchy, such that their intermediate parent classes are relabeled as the leaf classes. When employing this approach in semi-supervised scenarios, in our model training process, we simply neglect the loss imposed on the finest-level hierarchy for the samples that have been relabeled to their parent classes.

Evaluation Metrics: We adopt the same evaluation metrics as Chang et al. [27] to evaluate our method. Specifically, ACC (i.e., accuracy, the percentage of images whose labels are correctly classified) is reported at each class hierarchy, and then mACC (the mean of ACC across all class hierarchies) is calculated to measure the classification performance at a hierarchical level. Note that, unless otherwise specified, the ACC results are reported as the mean over three random seeds of each experiment.

Implementation Details: All our experiments are implemented with the Pytorch platform [56] running on NVIDIA Gefore RTX 3090Ti GPUs. Following [27], we employ ResNet-50 pre-trained on ImageNet [57] as our network backbone, followed by a fully-connected layer with 600 hidden units. For fair comparison, the features of fully-connected layer are uniformly split into \(K\) parts (\(K=2\) for Standford Cars and \(K=3\) for other datasets), with each used for predicting the labels at each hierarchy. During training, the models are trained with Momentum SGD with a momentum of 0.9, a weight decay of \(5\times10^{-4}\), a batch-size of 8, and an initial learning rate of 0.0002 for backbone layers and 0.002 for other layers. The training epoch is set to be 200 and the learning rate is adjusted by the cosine annealing schedule. We initially set \(\lambda=0.1\) for Cub-200-2011 and \(\lambda=2.0\) for the other two datasets, except for \(\lambda=0.01\) for all the three datasets under a relabeled ratio of \(\mathcal{R}=90\%\). The same augmentation strategies as described in [29] are adopted, i.e., each image is resized to \(448\times448\), and then the resulting image is randomly cropped (random cropping for training and center cropping for testing) and randomly horizontally flipped.

3.1.2 Main Results↩︎

We compare the proposed method with state-of-the-art methods on this task, including: () HD-CNN [24], which aims to learn a hierarchical deep convolutional neural network where a single network backbone was shared by multiple fully-connected layers with each one responsible for the label prediction within one hierarchy; () HMCN [55], which designs a specific network architecture with multiple local outputs for label prediction per hierarchical level and one global output layer for capturing semantic correlation in the hierarchy as a whole; () C-HMCNN [46], which exploits the hierarchy information by imposing the parent-child constraint on the output of the coarser-level hierarchies, so as to ensure that no hierarchy violation happens (i.e., when the network predicts a sample belonging to a class, this sample also belongs to its parent classes); () Soft Labels [17], an embedding-based method that softens the finest-level labels according to the metric derived by the label tree during training, and during testing, we backtrack through the label tree to obtain other coarse-level labels; () FGoN [27], one of the current state-of-the-arts which equips with hierarchy-specific classification head to disentangle and reinforce the features of different hierarchies, that is, only forces finer-level features to participate in coarser-level label predictions; () HRN [29], which incorporates hierarchical feature interaction into model design and proposes a hierarchical residual network, where hierarchy-specific features from parent level are added to features of children levels through residual connections. For fair comparison, we conduct all experiments under a unified codebase that includes: the public source codes of C-HMCNN [46]⁷, Soft Labels [17]⁸, FGoN [27]⁹ and HRN [29]¹⁰, and the reproduced codes of HD-CNN [24] and HMCN [55] since no source code is available for the original papers. When adapting these methods to semi-supervised settings, we neglect the loss over the leaf class if a sample has been relabeled to its parent class.

In 1, 2 and 3, we report ACC of each hierarchical level and mACC results on test sets of CUB-200-2011, Aircraft, and Stanford Cars, respectively, obtained by all competing methods. For supervised hierarchical classification setups (i.e., \(\mathcal{R}=0\%\)), we can observe that our LHT model consistently achieves superior or comparable performance in terms of ACC at each hierarchy, and it consistently outperforms all compared methods over mACC on all three datasets. For semi-supervised hierarchical classification, we select four setups (i.e., \(\mathcal{R}=30\%, 50\%, 70\%, 90\%\)) to imitate the lack of domain knowledge. Note that the dataset with \(\mathcal{R}=30\%\) means that we randomly select 30% samples from the training set and mask out their labels at the finest-level hierarchy during training. It can be observed that: 1) Our LHT achieves the best or second best ACC and mACC performance across all the hierarchies on all the three datasets. Particularly, in cases where limited fine-grained label information (\(\mathcal{R}=90\%\)) is available, our LHT surpasses all compared methods at all hierarchies by an evident margin, resulting in absolute mACC improvements of 5.0%, 2.6%, and 13.6% compared to the second best results on all three datasets, respectively. 2) The performance of all compared methods degrade with an increasing number of relabeled samples (\(\mathcal{R}=0\%\rightarrow90\%\)), but our LHT model consistently outperforms current state-of-the-art methods. This indicates that LHT can better utilize hierarchical label information to learn a more discriminative classifier. 3) To elucidate the origins of performance improvements, we visually represent the confusion matrices obtained from the test set of Cub-200-2011 dataset, employing a relabeled ratio of \(\mathcal{R}=90\%\). It should be noted that the diagonal vector of a confusion matrix signifies per-class recall. As shown in Fig. 6, our LHT model exhibits a notably superior improvement compared to the baseline ResNet-50 and the current state-of-the-art method FGoN [27]. Specifically, LHT achieves a mean-class-recall (defined as the arithmetic mean of recalls across all 200 classes) of 70.0%, surpassing FGoN by 7.8%. These results substantiate the superiority of our LHT in effectively capturing hierarchical knowledge during the training process.

3.1.3 Ablation Study↩︎

In order to gain a deeper understanding of the effectiveness underlying the proposed LHT model, We perform extensive ablation study to evaluate and understand the contribution of each critical component in the proposed LHT model. Unless mentioned otherwise, the experiments in this subsection are all performed on Cub-200-2011 dataset with relabeled ratio \(\mathcal{R}=90\%\).

First, we investigate the impact of label hierarchy to demonstrate that hierarchical label information is beneficial for performance enhancement, but it requires effective learning mechanisms to exploit its underlying semantics. To this end, we append a branch for each hierarchy level on ResNet-50 and use the hierarchical cross-entropy loss (denoted as ResNet-50 + HCL) to fit the labels at each hierarchy, and the results are listed in [tbl:tab:ablation]. It can be observed that the performance only exhibits a marginal improvement of about 2.4%, indicating a limited boost in comparison to the advancements achieved by our proposed LHT model.

Next, we validate the superiority of the proposed transition network, which aims to effectively encodes visual correlations embedded in class hierarchies, as delineated in Sec. 2.3. To substantiate this, on one hand, we quantify the performance of the model by integrating our proposed Transition Networks into the baseline model (ResNet-50 + HCL). As shown in [tbl:tab:ablation], the model achieves an accuracy of 66.7%, which yields an absolute gain of 8.4% over the baseline model. This substantiates the efficacy of our proposed transition network. We further investigate the hyper-parameter \(s\) introduced by the Transition Networks in Fig. 7 (a). We can see that the performance remains relatively stable as the value of \(s\) varies. As such, we fix it as 2 across all our experiments for a better trade-off between computation efficiency and model performance. On the other hand, if we directly treat the transition matrices as trainable parameters (LHT_simplified) instead of learning them through the Transition Networks, model performance degrades from 68.8% to 58.1%, further validating the effectiveness of the Transition Networks.

Lastly, we conduct ablation analysis on the proposed confusion loss. As shown in [tbl:tab:ablation], when the confusion loss is integrated into the model (ResNet-50 + HCL + Transition Networks), the resulting LHT model achieves a further 2.1% performance improvement. This verifies the effectiveness of the proposed confusion loss on encouraging the transition network to capture the semantic correlation within class hierarchies. In fact, the hyper-parameter \(\lambda\) in 12 plays an important role in trading off between the hierarchical cross-entropy loss and the confusion loss during training, and we thus conduct a detailed ablation analysis on \(\lambda\) in Fig. 7 (b). It can be observed that the initial performance gradually improves as \(\lambda\) increase from 0 to 0.01, which suggests that a smaller \(\lambda\) facilitates the transfer network to output meaningful transfer matrices; however, a much larger \(\lambda\) excessively forces the output transfer matrices to be more smooth, which may amplify biases introduced by inaccurate predictions, especially in semi-supervised scenarios.

3.1.4 Discussion and Analysis↩︎

We provide a comprehensive discussion from multiple perspectives to enrich the understanding of our approach by addressing the following questions.

How does LHT generalize across various architectures? In addition to ResNet-50, we further evaluate our method on ConvNeXt-Base and Swin-Base, which are representative modern CNN and ViT architectures. Compared to ResNet-50, these models have substantially larger parameter capacities, extending the evaluation to a much broader capacity range. We keep all other experimental settings consistent with those used for ResNet-50. As shown in Tab. 5, our proposed LHT consistently outperforms the baseline HD-CNN [24] that employs independent classification heads for each hierarchy under varying missing-label ratios. These results demonstrate that our approach generalizes well to both advanced CNNs and ViT-based architectures, confirming its robustness across diverse architecture designs.

How does LHT perform under sparser annotation settings? As aforementioned, our LHT presents an effective learning mechanism to exploit hierarchical label information, especially in cases where a significant portion of labels are missing at the finest-level hierarchy. Herein, we further validate the efficacy of our method in a more challenging semi-supervised scenario, where we randomly select a fraction of samples and mask out their labels at the two finest-level hierarchies (i.e., Species and Family) on Cub-200-2011 dataset. As shown in 6, Our proposed LHT model consistently improves the baseline HD-CNN [24] across different amounts of relabeled data. For example, in an extremely sparse annotation scenario (\(R^2=90\%\)), LHT significantly improves the baseline model by around 9.4% ACC at the finest-level hierarchy and 4.3% mACC across all hierarchies. This demonstrates that our LHT achieves higher performance with less label information, thus confirming its superiority in effectively utilizing class hierarchies.

Does the learned transition matrix encode semantic correlations? To answer this question, we visualize the transition matrices output by the Transition Network on Cub-200-2011 in 8. Specifically, we show the learned transition matrix associated with the two finest-level hierarchies, denoted as \(\mathbf{T}^1 (species\rightarrow family)\), and also visualize the corresponding naive transition matrices \(\mathbf{S}^1 (species\rightarrow family)\) directly inferred from hierarchical labels (as mentioned in Remark 1) for better reference. We can see that the naive transition matrix can only encode the deterministic state transition from the finer- to the coarser-level hierarchies since only one element is 1 in each column, whereas the ones learned from our proposed LHT model are indeed conditional label distributions considering the uncertainty from image context and hierarchy prior. This is mainly because, on the one hand, the learned matrices largely reflect the transition states in the naive transfer matrix, while on the other hand, LHT produces distinct transition matrices for different input samples. We also visualize the evolution of the weight \(\gamma\) during training in Appendix E to clarify the impact of the learned transition matrices.

Does the learned attention map encode semantic correlations? In Fig. 9, we further employ Grad-Cam [58] to visualize the attention regions of each hierarchical branch by propagating their respective gradients back to feature maps for four typical examples. These objects belong to the same order (i.e., “Charadriiformes"), two families (i.e., ”Laridae" and “Alcidae"), and four species (i.e., ”Black Tern", “Ivory Gull",”Least Auklet" and “Horned Puffin"). It can be observed that 1) our proposed LHT model exhibits a preference for activating more global attention regions in the coarser-level hierarchy compared to the finer-level hierarchy, and the visualization results demonstrate notable difference between the attention patterns of the finer-level categories and that of the coarser-level categories; 2) our LHT model effectively directs attention towards discriminative regions when classifying examples from different species within the same family. For example, when distinguishing between the species ”Least Auklet" and “Horned Puffin", both belonging to the family ”Alcidae", the model focuses intensely on the head, particularly the eye and beak regions, which provide essential discriminative information. This further validates the capability of our LHT model in capturing semantic correlations across different hierarchies to some extent.

3.2 Low-shot Learning with Class Hierarchy↩︎

Large-scale low-shot learning (LSL) involves a large number of source classes with abundant labeled data and target classes with only a few labeled examples. The goal is to recognize unlabeled target samples, which is challenging due to their limited supervision and the difficulty of directly learning discriminative classifiers.

3.2.1 Problem Formulation↩︎

Formally, let \(S_{source}\) denote the set of source classes and \(S_{target}\) denote the set of target classes. The two class sets satisfy the following properties: 1) \(S_{source}\) and \(S_{target}\) are disjoint, i.e., \(S_{source}\cap S_{target}=\emptyset\); 2) the semantic correlation of \(S_{source}\) and \(S_{target}\) are explicitly encoded into a tree-structured class hierarchy, where the leaf nodes cover all source and target classes and the parent nodes group semantically related leaf classes. We are given a large-scale source set \(\mathcal{D}_{source}\) from source classes \(S_{source}\), a few-shot support set \(\mathcal{D}_{target}\) from target classes \(S_{target}\), and a test set \(\mathcal{D}_{test}\) from target classes \(S_{target}\). In this setting, \(\mathcal{D}_{source}\) contains sufficient labeled samples for each source class, while \(\mathcal{D}_{target}\) provides only a few (e.g., 5) labeled samples per target class. The objective is to predict labels for \(\mathcal{D}_{test}\).

Following [59], we address the challenge of LSL by introducing hierarchical classification. The key idea is that, although source and target classes are disjoint, they often share higher-level categories in a tree-structured hierarchy. For example, certain source and target classes may share the broader category of “vehicles", with the former covering various vehicle types and the latter including specific categories such as cars. This shared structure provides a natural bridge between the source and target classes. By performing a hierarchical classification on the source data, the network learns transferable features that capture such semantic relationships. These representations serve as informative priors from semantically related source classes and are particularly beneficial for recognizing target classes.

Specifically, the method follows a two-stage process: ) Training stage, in which only source class samples are used to train the hierarchical network. Since the source classes and target classes share certain common super-classes, the network facilitates the learning of transferable features for LSL on the target classes. ) Test stage, in which visual features of the few-shot support samples and the test samples (both from target classes) are extracted through the learned hierarchical classification network. The test samples are then recognized by a simple nearest neighbor search using the visual features of the support samples as references. Specifically, when adapting the proposed LHT method to large-scale LSL, we train the hierarchical classification network with the source data in the training stage and employ the features from global average pooling as input for the nearest-neighbor classifier in the test stage.

3.2.2 Experimental Setups↩︎

A benchmark derived from ILSVRC2012/2010 [57] is used for performance evaluation [59]. This dataset consists of three parts: a support set with 1000 source classes from ILSVRC2012, a few-shot support set with 360 target classes from ILSVRC2010 (not included in ILSVRC2012), and a test set containing target class samples excluding those in the support set. Following [59], we construct a class hierarchy with three levels according to: 1) we first cluster the word vectors¹¹ of source/target class names using k-means to create a coarser-level hierarchy, and the super-class word vectors are obtained by averaging the word vectors of their child classes; 2) we then acquire upper-layer nodes by clustering the word vectors of the lower-layer nodes. These three hierarchies have 1000, 100, and 10 classes, respectively. We refer to [59] and employ Momentum SGD with a momentum of 0.9, a weight decay of \(5\times10^{-4}\), a batch-size of 128, and an initial learning rate of 0.001 for backbone layers and 0.01 for other layers to train the model for 5 epochs. The top-5 accuracy on \(\mathcal{D}_{test}\) is adopted as the evaluation metric.

3.2.3 Experimental Results↩︎

We compare our model with five large-scale LSL models: () NN, a baseline model employing ResNet50 pre-trained on \(\mathcal{D}_{source}\) as the nearest neighbor (NN) classifier; () SGM [61], a LSL model using the squared gradient magnitude (SGM) loss [7]; () PPA [62], the parameter prediction with activation (PPA) model; () LSD [63], a LSL model with large-scale diffusion (LSD); () KT [59], the state-of-the-art LSL model via knowledge transfer (KT) with class hierarchy. The results in 7 show that our proposed LHT method achieves the best result among all the comparison methods, and the performance gain is significant compared to KT. While KT also introduces hierarchical classification by feeding fine-grained predictions into coarser-level classifiers, it does not explicitly model statistical dependencies across hierarchies. Instead, our LHT makes these dependencies explicit by formulating inter-level correlations as conditional probabilities, which facilitates the capture of hierarchical relationships during training. This design yields more transferable features and explains the consistent improvements compared to KT.

4 Experiments on Medical Images↩︎

In this section, we empirically shed light on the potential of the proposed LHT applied in the domain of computer-aided diagnosis. Concretely, we reformulate the lesion diagnosis problem into a hierarchical classification task, as we can readily obtain a hierarchical categorization of lesions based on pathological priors [36], [64].

6pt

4.1 Experimental Setups↩︎

Dataset. The proposed model is evaluated on the publicly available ISIC2018 dataset [65] from the ISIC2018 challenge on skin lesion diagnosis¹². It is a challenging dataset due to the class-imbalanced issue. The dataset contains totally 10015 dermoscopic images for seven skin diseases, namely Melanocytic Nevi (NV), Melanoma (MEL), Benign Keratosis (BKL), Basal Cell Carcinoma (BCC), Actinic Keratosis / Bowens disease (AKIEC), Vascular Lesion (VASC), and Dermatofibroma (DF). We organize these seven skin lesions into three hierarchies according to their origin (melanocytic or non-melanocytic), degree of malignancy (benign or malignant), and finally their diagnosis (e.g., melanoma, basal cell carcinoma, vascular lesion, etc.) following [64], [66]. In ¿tbl:tab:skin?, we list the statistics and class hierarchies of the dataset. Since the ground truth of official validation and testing set was not released, we randomly split the entire training set to 70% for training, 10% for validation and 20% for testing. To verify the proposed method can notably improve the performance of skin lesion diagnosis and concurrently reduce the annotation cost of training samples, we conduct experiments in a semi-supervised learning regime, i.e., randomly selecting 80% of training samples and masking out their labels at the finest-level hierarchy (\(\mathcal{R}=80\%\)) or the two finest-level hierarchies (\(R^2=80\%\)).

Evaluation Metrics. To quantitatively evaluate the proposed method, we adopt five metrics, including Precision, Recall, F1-score, AUC¹³ and Accuracy for evaluation. We report the averaged results from five independent runs with different random seeds.

Implementation details. We employ ResNet-50 pre-trained on ImageNet [57] as our network backbone, and train the models with Momentum SGD with a momentum of 0.9, a weight decay of \(5\times10^{-4}\), a batch-size of 8, and an initial learning rate of 0.0002 for backbone layers and 0.002 for other layers. The training epoch is set as 200 and the learning rate is decayed by the cosine annealing schedule. The hyper-parameter \(\lambda\) in Eq. 12 is set as 0.01. In all our experiments, we resize input images to \(448 \times 448\) (originally around 600 × 450 pixels) and simply apply random horizontal flipping and random cropping (random cropping for training and center cropping for testing) for data augmentation.

4.2 Experimental Results↩︎

To understand the effectiveness of the proposed LHT applied to skin lesion diagnosis, we herein answer the following research questions under both \(\mathcal{R}=80\%\) and \(R^2=80\%\) settings: 1) Does LHT contribute to the diagnosis of skin lesions by using class hierarchy? 2) How does LHT perform compared with prior hierarchical classification methods when used in skin lesion diagnosis?

To answer the first question, we report the per-class recall results for all the seven classes and compare it with a vanilla model (ResNet-50 without using hierarchical label information to train the model) in 8. It can be observed that: 1) Our proposed LHT method consistently improves the recall of each lesion category and improve the average recall by \(67.2\%\rightarrow 82.4\%\) under \(\mathcal{R}=80\%\) and \(67.2\%\rightarrow 75.8\%\) under \(R^2=80\%\), respectively. This demonstrates that our approach effectively leverages semantic correlations within the label hierarchy to enhance the performance of skin lesion diagnosis. 2) In the setting of \(R^2=80\%\), it only requires coarse-level annotations (just identifying whether the lesion is melanocytic or non-melanocytic) for 80% of training data, and our LHT model results in an absolute performance gain of up to 8.6%. This highlights the significant potential of our method to reduce the annotation cost for computer-aided diagnosis systems.

In 9, we further compare the proposed LHT with state-of-the-art methods for skin lesion diagnosis on ISIC2018 dataset. The comparison methods include HD-CNN [24], HMCN [55], C-HMCNN [46], FGoN [27], and HRN [29], which are described in detail in 3.1.2. The results show that our LHT model consistently outperforms all comparison methods over all five evaluation metrics. For example, we surpass the second best result by \(79.9\%\rightarrow 82.4\%\) under \(\mathcal{R}=80\%\) and \(73.7\%\rightarrow 75.8\%\) under \(R^2=80\%\) on recall, highlighting that the proposed LHT method indeed helps to exploit the coarsely annotated hierarchical labels for skin lesion diagnosis. This reveals that our method can effectively leverage the training data annotated at different hierarchical levels to train a computer-aided diagnosis system. As such, the annotators may only need to provide partial labels with the highest confidence based on their own expertise, rather than annotating every sample into the finest-level label, which largely alleviates the conformation bias arising from noisy labels during training.

5 Conclusion and Discussion↩︎

In this work we have proposed a probabilistic approach to address the hierarchical classification problem. Our key insight is to directly encode the correlation between two adjacent class hierarchies as conditional probabilities, such that the category labels across different hierarchies can be recursively predicted in a fine-to-coarse order. Beyond that, we have designed a confusion loss to further encourage the deep classifiers to learn the hierarchical correlation. We demonstrate the superiority of our method in hierarchical classification on a series of benchmark datasets and further shed light on the potential of the proposed method in computer-aided diagnosis.

Despite these promising results, our method has certain limitations. First, it relies on predefined hierarchical structures, which may not always be available in real-world applications. Second, the current framework focuses primarily on single-label classification, and its scalability to multi-label settings or to more general graph-structured hierarchies (e.g., cases where certain nodes have multiple parent categories) remains to be fully investigated. These limitations point to several promising directions for future research, including incorporating automatically discovered or partially observed hierarchies and extending the framework to multi-label and graph-structured scenarios.

6 Theoretical Proofs in Section 3↩︎

6.1 Derivation of Eq. (1) Equivalent to Solving a Negative Log-Likelihood↩︎

In this section, we provide a detailed derivation that establishes the equivalence between the hierarchical cross-entropy loss \(\mathcal{L}_{{\rm CE}}=\sum_{n=1}^N\sum_{k=1}^K H(\mathbf{p}^k_n, \mathbf{y}^k_n)\) presented in Eq. (1) of the main text and the negative log-likelihood \(p(y^1,\cdots,y^K|\mathcal{I})=\prod_{k=1}^K p(y^k|\mathcal{I})\).

Derivation. We first denote \(\mathbf{y}^k\) as the one-hot vector corresponding to \(y^k\), \(\mathbf{p}^k\) as the associated prediction for \(y^k\), and \(\mathbf{v}[i]\) as the \(i\)-th entry of the vector \(\mathbf{v}\). We then have

\[\begin{align} -\log p(y^1,\cdots,y^K|\mathbf{x}) = &-\log\prod_{k=1}^K p(y^k|\mathbf{x}) \\ =&-\sum_{k=1}^K \log p(y^k|\mathbf{x}) \\ =&-\sum_{k=1}^K \log \prod_{i=1}^{C_k}(\mathbf{p}^k[i])^{\mathbf{y}^k[i]} \\ =&-\sum_{k=1}^K \sum_{i=1}^{C_k}\mathbf{y}^k[i]\log \mathbf{p}^k[i] \\ =&\sum_{k=1}^K H(\mathbf{y}^k, \mathbf{p}^k). \end{align}\] Note that to the right of the third equal sign, it implies that the prediction at each hierarchy follows a categorical distribution.

6.2 Proof of Proposition 1↩︎

6.3 Proof of Proposition 2↩︎

6.4 Proof of Corollary 1↩︎

6.5 Proof of Theorem 1↩︎

In this section, we present a detailed proof of Theorem 1, which relies solely some mild and practical assumptions.

7 Why Learn Data-driven Transition Matrices?↩︎

While the naive transition matrix in Eq. (10) can capture the pre-defined parent-child relationships, we would like to emphasize that additional predictions based on data-driven correlations, as employed in our work, are essential for effectively handling visual recognition tasks with hierarchical labels.

The main motivation for learning data-driven transition matrices is that the hierarchical relationships derived from a label hierarchy tree are not always visually grounded. In certain cases, such as for the Aircraft dataset, these relationships are defined more by semantics than by visual similarity. Specifically, the hierarchy in the Aircraft dataset is structured based on a maker-family-model organization (e.g., “Boeing"–”Boeing 767"–“767-200"), but this semantic-based hierarchy does not always reflect visual similarity. For example:

• Boeing 737 (Boeing family) vs. Boeing 747 (Boeing family): Although both aircraft are produced by Boeing, the 737 is a narrow-body aircraft, whereas the 747 is a wide-body jumbo jet. The visual distinction between these models is quite significant, despite their shared maker.

• Boeing 737 (Boeing family) vs. Airbus A320 (Airbus family): Although from different makers, both the 737 and A320 are narrow-body, twin-engine jets, making their visual similarities more evident than the comparison between the Boeing 737 and Boeing 747.

These examples illustrate that visual similarity does not always align with the hierarchical structure based on maker and family, underscoring the need for additional predictions to capture these visual correlations.

To verify this, we compare the performance of the LTH_naive model, which directly uses naive transition matrices, with our LHT model that employs learnable transition matrices, on the Aircraft dataset. As shown in 3, for the finest-level hierarchy, LTH_naive achieves an ACC of 91.2, which is notably lower than the 92.6 achieved by our LHT model. To further demonstrate the importance of adaptively learning visual correlations, we constructed a new dataset, Perturbed-Aircraft, based on the Aircraft dataset. In this modified version, we retain the original three-level hierarchy but randomly perturb 40% of the category indices at the \(2\)-nd hierarchy. This introduces significant disorder and misalignment in the category relationships, resulting in a label hierarchy that is less visually consistent than the original dataset Aircraft. Under this challenging scenario, the ACC of LHT_naive at the finest-level hierarchy drops by about 4%. In contrast, our LHT model surpasses LHT_naive by 5.1%, further demonstrating the effectiveness of learning data-driven transition matrices to capture visual category correlations.

The script and code for reproducing the experimental results on Perturbed-Aircraft have also been made publicly available at https://github.com/renzhenwang/label-hierarchy-transition.

8 Why not Learn the Transition Matrices form a Coarse-to-fine Manner?↩︎

Technically, our LHT can learn the transition matrices in a fine-to-coarse manner, i.e., we first make predictions at the finest level, and the rest coarser-level labels can be determined according to the label hierarchy.We address the advantages of the proposed fine-to-coarse learning paradigm from the following three key aspects:

• Motivationally, we employ an off-the-shelf classification network to predict labels of the finest-level hierarchy. Such a learning paradigm is advantageous in demonstrating that the Transition Network can be a plug-in to enhance conventional classification networks through hierarchical supervision, particularly in missing label scenarios where a large number of samples have labels only at partial coarser-level hierarchies.

• Theoretically, unlike the coarse-to-fine learning paradigm, the fine-to-coarse learning paradigm enables coarse-level hierarchies to provide supervision for the finest-level hierarchy via gradient back-propagation. This property is particularly important in missing label scenarios, as it allows for an effective utilization of partially labeled samples. To illustrate this, we take a two-level hierarchical classification as an example. As shown in Fig. 11 (a), for a sample \(\mathbf{x}\) labeled only at the second hierarchy \(\mathbf{y}^2\), the fine-to-coarse learning paradigm can provide supervision to the finest-level hierarchy through \(\mathbf{y}^2\). In contrast, this flexibility is not available in the coarse-to-fine paradigm, as shown in Fig. 11 (b), where the gradient from the coarser-level label \(\mathbf{y}^2\) cannot flow to the finest-level, thus failing to provide supervision for the finest-level hierarchy.

• Experimentally, we conducted experiments on the CUB-200-2011 dataset to compare the performance of the fine-to-coarse learning paradigm (LHT_f2c) with that of the coarse-to-fine learning paradigm (LHT_c2f). As shown in Table ¿tbl:tab:c2f95vs95f2c?, the results demonstrate that LHT_f2c consistently outperforms LHT_c2f across various settings, with particularly significant improvements in missing label scenarios. This provides empirical evidence of the practical advantages of the fine-to-coarse paradigm in enhancing hierarchical classification.

9 Delve into the hierarchical structure of the experimental benchmarks↩︎

To clarify why CUB-200-2011, FGVC-Aircraft, and Stanford Cars are appropriate benchmarks for hierarchical classification, we provide qualitative and quantitative evidence that these datasets show clear inter-class similarity within hierarchies, making them well-suited for hierarchical learning.

• Qualitative evidence supports the suitability of these datasets for hierarchical classification. As shown in 12 and 13, the label hierarchies of these datasets align well with the underlying visual structures. In CUB, the taxonomy is grounded in biological knowledge, where families and orders naturally reflect visual traits such as plumage color and body shape. For example, the three auklet species within the Alcidae family are highly similar, whereas families such as Alcidae and Passerellidae exhibit obvious distinctions. In FGVC-Aircraft, the hierarchy is artificially defined by a Maker–Family–Model structure, yet it still encodes meaningful visual correlations. For example, models such as Airbus A318, A319, and A320 are visually similar, whereas cross-family differences (e.g., Airbus vs. Boeing) are more obvious. Although Stanford Cars provides only a two-level hierarchy, it still follows the same principle as FGVC-Aircraft, where the models within a maker are highly similar. Overall, these datasets demonstrate that their label trees effectively encode hierarchical visual correlations, making them well-suited for the evaluation of hierarchical classification methods.

• Quantitative evidence supports the suitability of these datasets for hierarchical classification. To substantiate the above claims, we further conduct a quantitative analysis on CUB using its rich attribute annotations. Each species in CUB is associated with a 312-dimensional attribute vector that encodes detailed properties of color, shape, pattern, and specific body parts (e.g., wing color, bill shape, underpart pattern, et al.). We compute the pairwise cosine similarity between these vectors by \(s_{ij} = \frac{\langle \mathbf{a}_i, \mathbf{a}_j \rangle}{\|\mathbf{a}_i\|\|\mathbf{a}_j\|}\), where \(\mathbf{a}_i\) denotes the attribute vector of the species \(c_i\). The resulting similarity matrix is shown in 14, and it should be noted that the axes correspond to the species indices, consistent with those in 12. The matrix exhibits clear block-diagonal structures aligned with taxonomic structures. In particular, species within the same family (e.g., Alcidae or Icteridae) exhibit substantially higher similarities than those across different families. These results indicate that the given label tree closely corresponds to actual semantic and visual similarities between species, providing a solid foundation for evaluating hierarchical classification.

10 Visualization of the value of the learned weight \(\gamma\).↩︎

To provide a clearer understanding of the impact of the learned transition matrices, we here visualize the evolution of the weight \(\gamma\) throughout the training stage on the CUB-200-2011 dataset with \(\mathcal{R}=0\%\) (corresponding to the setup of Fig. 6 in the original manuscript) in 15. It can be observed that: (1) By the end of training, \(\gamma\) deviates significantly from its initial value of 1.0 and exhibits distinct values across different hierarchies, indicating that the learned components differ substantially from the naive transition matrices; (2) Throughout the training process, \(\gamma\) shows dynamic changes, which supports our initial motivation that \(\gamma\) helps maintain the values of \(\gamma\mathbf{S}\) and the transition network outputs on the same scale. Such an operation is necessary because the elements of \(\mathbf{S}\) are constrained within the range of 0 to 1, while the outputs of the transition network are unconstrained. Scaling \(\mathbf{S}\) by a learnable \(\gamma\) helps ensure numerical stability during network training.

11 How does LHT perform when meeting low-resolution images?↩︎

As investigated in [27], low-resolution images may potentially degrade the performance of a hierarchical classification system, and the proposed FGoN [27] shows evident superiority to take advantage of hierarchical label information, thus mitigating the harm brought about by the image quality. To validate the effectiveness of the proposed method in low-resolution images settings, we directly compare with FGoN following the same training protocols. Concretely, we reduce the image resolution to \(224\times224\), and keep other settings unchanged for fair comparison. As shown in 10, our proposed LHT model consistently surpasses FGoN, and achieves absolute mACC gains of 1.8%, 2.1% and 0.6% on Cub-200-2011, Aircraft and Stanford Cars datasets, respectively. This verifies that our method has fine potential to infer category labels from class hierarchies, so as to alleviate the practical limitations arising from image quality.

Acknowledgment↩︎

References↩︎

---

1. This assumption is practical, as we can introduce a small positive bias term (e.g, \(\epsilon=10^{-8}\)) or directly apply a sigmoid activation before the linear classification layer to ensure the input is greater than 0.↩︎

2. To construct this dataset based on Aircraft, we retain the original three-level hierarchy but randomly perturb 40% of the category indices at the second hierarchy, which introduces significant disorder and misalignment in the category relationships compared to Aircraft.↩︎

3. In this paper, we stem the term of “semi-supervised learning” from [29], in the sense of “partially supervised with incomplete hierarchical labels for a part of training samples", in contrast to the classical term that assumes most training samples are given without any annotation.↩︎

4. http://www.vision.caltech.edu/visipedia/CUB-200-2011.html↩︎

5. https://www.robots.ox.ac.uk/ vgg/data/fgvc-aircraft↩︎

6. https://www.kaggle.com/datasets/jessicali9530/stanford-cars-dataset↩︎

7. https://github.com/EGiunchiglia/C-HMCNN↩︎

8. github.com/fiveai/making-better-mistakes↩︎

9. https://github.com/PRIS-CV/Fine-Grained-or-Not↩︎

10. https://github.com/MonsterZhZh/HRN↩︎

11. The word vectors are obtained by a skip-gram text model [60] in [59] and available at https://github.com/tiangeluo/fsl-hierarchy.↩︎

12. https://challenge2018.isic-archive.com↩︎

13. Each metric is calculated as the average value across all classes.↩︎

14. This assumption is practical, as we can introduce a small positive bias term (e.g, \(\epsilon=10^{-8}\)) or directly apply a sigmoid activation before the linear classification layer to ensure the input is greater than 0.↩︎