1 Introduction↩︎

Deep learning methods are driving progress in a wide span of computer vision tasks, particularly when large labeled datasets are easily accessible for training. Obtaining such large datasets is a cumbersome process, which is often a limiting factor impeding the scalability of these models. To alleviate this limitation, semi-supervised learning (SSL) has emerged as an appealing alternative, which leverages both labeled and unlabeled data to boost the performance of deep models. Despite recent success, SSL approaches work under the closed-set assumption, in which the categories in the labeled and unlabeled subsets share the same underlying class label space. Nevertheless, this assumption rarely holds in real scenarios, where novel categories may emerge in conjunction with known classes, which typically results in significant drops in the performances of standard supervised deep learning models. Thus, the ability to detect whether the input of a deep learning model belongs or not to a set of known classes seen during training is essential for robust deployment in a breadth of critical application areas, such as, medicine, security, finance, agriculture, marketing, and engineering [1], [2]. Thus, devising novel learning models that can address the realistic open-set scenario is of paramount importance.

Novel category discovery (NCD) [3], [4] tackles this problem by exploiting the knowledge learned from a set of relevant known classes to improve the clustering of the unknown categories. Nevertheless, NCD assumes the two sets of classes to be disjoint, which means that the unlabeled dataset contains only instances belonging to the set of novel categories. Generalized category discovery (GCD) [5] considers a more general scenario, where unlabeled data contain instances from both seen and novel classes. This scenario is particularly challenging, as learning is performed under class distribution mismatch, and the unlabeled data may contain categories never encountered in the available labeled set.

1.0.0.1 Contributions:

In this work, we address the generalized category discovery task from an information-theoretic perspective. Our contributions are summarized as follows:

• We introduce a Parametric Information Maximization (PIM) model for GCD. Specifically, we propose a bi-level optimization formulation, which explores a parameterized family of objective functions, each evaluating a weighted mutual information between the features and the latent labels, subject to supervision constraints from the labeled samples. Our formulation mitigates the class-balance bias encoded in standard information maximization, deals effectively with both short-tailed and long-tailed data sets, and is model-agnostic (i.e., could be used in conjunction with any feature extractor).

• We report extensive experiments and comparisons demonstrating that PIM consistently sets new state-of-the-art performances across six different datasets, with larger gaps on the more challenging fine-grained benchmarks. It outperforms both specialized GCD methods and standard information-maximization approaches.

2 Related work↩︎

Semi-supervised learning (SSL) has been widely explored in the machine learning and computer vision community. This learning paradigm aims at leveraging large unlabeled datasets that contain the same set of classes as the labeled samples. Due to their satisfactory performance, consistency-based approaches have gained popularity recently, such as Mean-Teacher [6], MixMatch [7], UDA [8] or FixMatch [9]. An interesting alternative is self-training, which relies on the generation of pseudo-labels from a small amount of labeled data [10], [11], or in solving surrogate classification tasks [12], [13]. Nevertheless, the main limitation is that most existing SSL models rely on the closed-set assumption, as they do not consider unlabeled data points sampled from novel semantic categories.

Novel Class Discovery (NCD), which was formalized in [3], relaxes the closed-set assumption, as it focuses on discovering new categories in the unlabeled set by leveraging the knowledge learned from the labeled set. AutoNovel [14] (also referred to as RankStats) resorts to ranking statistics as an efficient approach for NCD. First, a good embedding is learned in a self-supervised manner for learning the early feature representation layers, which is followed by a supervised fine-tuning step with labeled samples for learning high level feature representations. Finally, to determine whether two instances from the unlabeled set are from the same category, a robust ranking statistics approach is introduced. A dual ranking statistics method coupled with mutual knowledge distillation is further proposed in [15]. OpenMix [16] showed that mixing up both labeled and unlabeled data can prevent the representation learning model from overfitting the labeled categories. Some other methods [17], [18] adopt contrastive learning for the novel category discovery task. UNO [19] unifies a cross-entropy loss to jointly train the model with both the labeled and unlabeled data. Despite the good performance in discovering new categories, these methods assume that the test dataset only contains instances from the novel classes. A recent work by [20] presented a method based on a mutual-information measure, which is different from the discriminative and constrained mutual information we introduce in this work. The mutual information in [20] evaluates the relation between the old and novel categories in the label space, arguing that maximizing such a measure promotes transferring semantic knowledge. In our case, we introduce a parametric, bi-level optimization of the mutual information between the feature and label spaces, on both labeled and unlabeled samples.

Generalized Category Discovery (GCD) extends NCD by allowing both old and new classes to coexist in the unlabeled dataset, which we tackle in this work. This pragmatic yet challenging scenario was recently introduced in [5] and triggered several other recent studies of the GCD problem. In [5], the authors proposed to fine-tune a pre-trained DINO ViT [21] with one supervised and one self-supervised contrastive term. Then, they used a semi-supervised clustering for label assignment. Note that, while UNO [19] and RankStats [14] are originally investigated for the NCD task, they are adapted for GCD in the recent study in [5], yielding UNO+ and RankStats+, respectively. Another recent approach, referred to as ORCA [22], addressed a similar problem, naming it open world semi-supervised learning. ORCA consists of controlling the intra-class variance of the seen classes to align and reduce the learning gap w.r.t. novel categories.

Maximizing the mutual information. Our discriminative partitioning approach (PIM) is built on the general and well-known InfoMax principle [23], which prescribes maximizing the mutual information (MI) between the inputs and outputs of a system. Several variants of this general principle have been recently used in machine learning and computer vision tasks, including deep clustering [24]–[26], few-shot learning [27], representation learning [28]–[31], deep metric learning [32] and domain adaptation [33]. To the best of our knowledge, addressing the GCD problem from an information-theoretic perspective remains unexplored.

The pioneering discriminative clustering model in [25] and the recent transductive few-shot method in [27] are closely related to our work, as they both maximize the mutual information between the features and the latent labels. However, as we shall see in our experiments, the direct application of information maximization [25], [27] to GCD may not be highly competitive. First, the standard mutual-information objective has a strong encoded bias towards balanced partitions, via its marginal-entropy term, which might be detrimental to performances. In this work, we introduce a parametric family of mutual-information objectives, which we tackle with a bi-level optimization formulation, thereby estimating automatically the weight of the marginal-entropy term. Our parametric information maximization effectively deals with both short-tailed and long-tailed data sets, mitigating the class-balance bias. Secondly, the InfoMax models in [25], [27] were designed in the scenario where the unlabeled set contains examples from the classes seen in the available labeled set. Finally, in [25], [27], the mutual-information objective is defined over the set of unlabeled samples. In contrast, we propose a constrained mutual-information formulation defined over both labeled and unlabeled samples, thereby capturing the distribution of the whole data set in the context of GCD. As we will see in our experiments, our parametric, bi-level information maximization substantially outperforms [25], [27] in the GCD scenario.

3 Generalized Category Discovery problem↩︎

Problem definition. Assume we are given a dataset \({\mathcal{D}}\) composed of two subsets so that \({\mathcal{D}}= {\mathcal{D}}_L \cup {\mathcal{D}}_U\). First, \({\mathcal{D}}_L=\{(\boldsymbol{x}_i,\boldsymbol{y}_i)\}_{i=1}^{N}\) refers to a labeled subset containing \(N\) images from a set of known classes in \({\mathcal{Y}}_L\). For each image \(\boldsymbol{x}_i\) in \({\mathcal{D}}_L\), we have access to its corresponding one-hot vector label \(\boldsymbol{y}_i= (y_{i,k})_{1 \leq k\leq K^{\text{old}}}\), where \(K^{\text{old}}=|{\mathcal{Y}}_L|\) is the number of classes in \({\mathcal{Y}}_L\). \(y_{i,k} = 1\) if \(\boldsymbol{x}_i\) belongs to class \(k\), and \(0\) otherwise. Now, let \({\mathcal{D}}_U=\{\boldsymbol{x}_i\}_{i=1}^{M}\) denote the unlabeled subset, which contains \(M\) images from a set of classes \({\mathcal{Y}}_U\) composed of known classes, as well as novel classes, i.e., \({\mathcal{Y}}_L \subset {\mathcal{Y}}_U\). Note that, during inference, \(K=|{\mathcal{Y}}_U|\) is the total number of classes, which contains both known and novel categories. Given this setting, the Generalized Category Discovery (GCD) task introduced in [5] consists in partitioning the images in the unlabeled set into separate clusters at test time. Each obtained cluster is supposed to represent a separate known or novel category. In other words, the GCD problem amounts to jointly solving (i) a semi-supervised classification task for the known classes; and (ii) a clustering task for the novel classes.

Notation. Let us denote \(g_{\boldsymbol{\theta}}: {\mathcal{D}}\rightarrow {\mathcal{Z}}\subset \mathbb{R}^D\) as the trained encoder responsible for mapping an input image \(\boldsymbol{x}_i\) into a feature vector \(\boldsymbol{z}_i\) of dimension \(D\), with \(\theta\) the set of trainable parameters and \({\mathcal{Z}}\) the set of all embedded features, for both the labeled and unlabeled samples. We now define a soft partitioning model \(f_{\boldsymbol{W}}: {\mathcal{Z}}\rightarrow [0,1]^K\), which is parameterized by weight matrix \(\boldsymbol{W}=(\boldsymbol{w}_k)_{1 \leq k \leq K}\), where \(\boldsymbol{w}_k=(w_{k,n})_{1 \leq n \leq D}\) denote its trainable parameters. For each input feature map \(\boldsymbol{z}_i\), \(f_{\boldsymbol{W}}\) outputs a softmax prediction vector \(\boldsymbol{p}_i=(p_{i,k})_{1 \leq k \leq K}\) of dimension \(K\), defined on the standard (\(K-1\))-probability simplex domain \(\Delta^{K-1}= \{\boldsymbol{p}_i \in [0,1]^K~|~ \boldsymbol{p}_i^T \boldsymbol{1} = 1\}\). Note that, similarly to the prior work in [5], we assume the number of clusters during the partitioning task to be known.

Let \(Z \in \mathbb{R}^{D}\) denote a random variable representing the feature map. \(Z\) follows \(\mathbb{P}(Z)\), which denotes the distribution of the set of embedded features \({\mathcal{Z}}\). Hence, each feature map data point \(\boldsymbol{z}_i\) is a realization of \(Z\). Furthermore, let \(Y\in {\mathcal{Y}}=\{1,\dots,K\}\) be the random variable following the dataset label distribution \(\mathbb{P}(Y)\).

4 Background on information maximization↩︎

4.0.0.1 Marginal distributions.

Let \(\boldsymbol{\pi}=(\pi_k)_{1 \leq k \leq K}\), where \(\pi_k=\mathbb{P}(Y=k;\boldsymbol{W})\) denote the marginal distributions that one can approximate by the soft² proportion of points within each cluster, via Monte-Carlo estimation, as follows:

\[\begin{align} \pi_k &= \int_{\boldsymbol{z}} \mathbb{P}(Z=\boldsymbol{z}) \mathbb{P}(Y=k|Z=\boldsymbol{z};\boldsymbol{W})\text{d}\boldsymbol{z} \\ & \approx \frac{1}{|\mathcal{Z}|} \sum_{i \in {\mathcal{Z}}} \mathbb{P}(Y=k|Z=\boldsymbol{z}_i;\boldsymbol{W})= \frac{1}{|\mathcal{Z}|} \sum_{i \in {\mathcal{Z}}} p_{i,k} \end{align} \label{eq95marg95prob}\tag{1}\]

4.0.0.2 Mutual Information.

The mutual information between the labels and the features maps can be written as follows: \[\label{eq95mutual95info} I(Y,Z) = \mathcal{H}(Y) - \mathcal{H}(Y|Z),\tag{2}\] with \(\mathcal{H}(Y)\) referring to the entropy of the marginal distributions \(\mathbb{P}(Y=k;\boldsymbol{W})\), and \(\mathcal{H}(Y|Z)\) referring to the entropy of the conditional probability distribution \(\mathbb{P}(Y|Z; \boldsymbol{W})\).

4.0.0.3 Marginal entropy.

The marginal entropy term \(\mathcal{H}(Y)\) in 2 can be estimated, w.r.t. the soft marginal distributions approximation in 1 , as follows:

\[\label{eq95mutual95ent} \begin{align} \mathcal{H}(Y) &= -\sum_{k=1}^K \mathbb{P}(Y=k;\boldsymbol{W}) \log \mathbb{P}(Y=k;\boldsymbol{W}) \\ &= -\sum_{k=1}^K \pi_k \log \pi_k \end{align}\tag{3}\]

Weakness of InfoMax based approaches [25], [27]. It is common in the literature to maximize the unsupervised mutual information in Eq. 2 , which is often defined over unlabeled samples. This is the case, for instance, of the transductive few-shot inference in [27] (TIM) and the discriminative clustering in [25] (RIM). A closer look at the marginal-entropy term in 3 enables to write it, up to a constant, as a Kullback-Leibler (KL) divergence between the marginal probabilities of predictions and the uniform distribution:

\[\begin{align} \mathcal{H}(Y) & \stackrel{\mathclap{\normalfontc}}{=}- \mathcal{D}_{KL}(Y||\mathcal{U}_K), \label{eq:link95entropy95kl} \end{align}\tag{4}\] where \(\stackrel{\mathclap{\normalfontc}}{=}\) stands for equality up to additive and/or non-negative multiplicative constant, and \(\mathcal{U}_K\) is the uniform distribution over K classes. Thus, the term \(\mathcal{H}(Y)\) pushes the marginal distribution towards the uniform distribution, as made explicit by the previous equation, thereby encoding a strong bias towards balanced partitions. Note that this standard mutual information objective lacks a mechanism to explicitly control the weight of the marginal entropy. Therefore, this term has the potential to harm the performance in the case of imbalanced scenarios, where the underlying class distribution is no longer uniform. Based on the above-identified limitation of the mutual information, we introduce a parametric family of mutual-information objectives, which we tackle with a bi-level optimization formulation, thereby estimating the relative weight of the marginal-entropy term.

5 Proposed bi-level and constrained InfoMax↩︎

5.0.0.1 Constrained mutual information

We propose to maximize a constrained version of the mutual information presented in 2 , integrating supervision constraints on the conditional probabilities \(\boldsymbol{p}_i\) of the samples within the labeled set. Our constrained information maximization reads: \[\max_{\boldsymbol{W}} \, \mathcal{H}(Y) - \mathcal{H}(Y | Z) \quad s.t. \quad \boldsymbol{y}_i=\boldsymbol{p}_i \quad \forall \boldsymbol{z}_i \in {\mathcal{Z}}_L \label{eq95constrained95mutual95info}\tag{5}\] where \({\mathcal{Z}}_L\) denotes the set of embedded features for the labeled samples. It is straightforward to notice that by plugging the equality constraints in 5 into the mutual information, one could write the objective as follows: \[\begin{align} \min_{\boldsymbol{W}} \, \sum_{k=1}^K \pi_k \log \pi_k - \frac{1}{|{\mathcal{Z}}|} \sum_{i \in {\mathcal{Z}}} \sum_{k=1}^K h_{i,k} \log p_{i,k}, \end{align} \label{eq95auxiliary}\tag{6}\] where \(h_{i,k}=y_{i,k}\) if \(\boldsymbol{z}_i \in \mathcal{Z}_L\) and \(h_{i,k} = p_{i,k}\) otherwise. That is, for \(\boldsymbol{y}_i=\boldsymbol{p}_i \quad \forall \boldsymbol{z}_i \in {\mathcal{Z}}_L\), the objectives in 5 and 6 are equal to each other. Interestingly, the terms corresponding to \(h_{i,k}=y_{i,k}\) in 6 yield the standard cross-entropy (CE) loss for the labeled samples. This CE loss could be viewed as a penalty function for imposing constraints \(\boldsymbol{y}_i=\boldsymbol{p}_i \quad \forall \boldsymbol{z}_i \in {\mathcal{Z}}_L\), as it reaches its minimum when these constraints are satisfied. Therefore, we do not need to impose explicitly the equality constraints in 5 . Notice that, for the labeled samples, CE in 6 replaced the conditional entropy term in the mutual information. This is reasonable as CE enables to jointly impose the supervision constraints while encouraging implicitly confident predictions, as it pushes them toward one vertex of the simplex. Both CE and conditional entropy reach their minima at the vertices of the simplex.

5.0.0.2 Bi-level optimization

To mitigate the bias of the mutual information towards balanced partitions, we propose to explore a family of weighted versions of the objective in 6 , which we parameterize with a variable parameter \(\lambda\) and tackle as a bi-level optimization problem: \[\begin{align} F(\boldsymbol{W}, \lambda) = \underbrace{\sum_{k=1}^K \pi_k \log \pi_k}_{ {\mathcal{H}}(Y)} \underbrace{-\frac{1}{|{\mathcal{Z}}_L|} \sum_{i \in {\mathcal{Z}}_L} \sum_{k=1}^K y_{i,k} \log p_{i,k}}_{{\tiny CE}} \\ - \underbrace{\frac{\lambda}{|{\mathcal{Z}}_U|} \sum_{i \in {\mathcal{Z}}_U} \sum_{k=1}^K p_{i,k} \log p_{i,k}}_{\propto {\mathcal{H}}(Y | Z)} \end{align} \label{eq95classifier95loss}\tag{7}\] where \({\mathcal{Z}}_U\) denotes the set of embedded features for the unlabeled samples (i.e., \({\mathcal{Z}}= {\mathcal{Z}}_U \cup {\mathcal{Z}}_L\)). Variable \(\lambda \in (0,1]\) controls the effect of the unsupervised loss terms in Eq. 7 , i.e., confidence vs. class balance. Therefore, as we will see in our experiments, learning \(\lambda\) from the labeled set, via a bi-level optimization, yields highly competitive performances in the GCD setting, more so when dealing with long-tailed (imbalanced) data sets. Our bi-level formulation reads: \[\label{bi-level-formulation} \min_{\boldsymbol{W}} F(\boldsymbol{W}, \lambda) \quad s.t \quad \lambda \in \mathop{\mathrm{arg\,max}}_{\lambda \in (0,1]} A_L (\lambda),\tag{8}\] where \(F\) is the upper-level objective and \(A_L\) is the lower-level objective defined by the clustering accuracy³ on the set of labeled samples: \[A_L(\lambda) = \frac{1}{|{\mathcal{Z}}_L|} \sum_{i=1}^{|{\mathcal{Z}}_L|} \mathbb{1}_{\{\hat{\boldsymbol{y}}_i (\lambda) = \boldsymbol{y}_i\}},\] and \(\hat{\boldsymbol{y}}_i (\lambda)\) are the one-hot vector predictions on labeled samples maximizing parametric mutual information: \[\label{mutual-info-lambda} G(\boldsymbol{W}, \lambda) = \sum_{k=1}^K \pi_k \log \pi_k - \frac{\lambda}{|{\mathcal{Z}}|} \sum_{i \in {\mathcal{Z}}} \sum_{k=1}^K p_{i,k} \log p_{i,k}\tag{9}\] To tackle our problem, we explore a finite set⁴ \(\boldsymbol{\lambda}\) of uniformly-spaced values of variable \(\lambda\) in \((0, 1]\). For each of these values of \(\lambda\), we optimize \(G(\boldsymbol{W}, \lambda)\) in Eq. 9 w.r.t to linear-classifier parameters \(\boldsymbol{W}\) via standard gradient steps, thereby obtaining predictions \(\hat{\boldsymbol{y}}_i (\lambda)\). Note that, although we explore several values of \(\lambda\), this remains computationally efficient as the feature encoder parameters are fixed and only classifier parameters \(\boldsymbol{W}\) are updated. For initializing \(\boldsymbol{W}\), which could be viewed as class prototypes, we use the K-means++ algorithm. This process yields a prediction of the optimal \(\lambda\) as: \(\lambda_{opt} = \mathop{\mathrm{arg\,max}}_{\lambda \in \boldsymbol{\lambda}} A_L (\lambda)\). Finally, the partitioning solution of our GCD problem in Eq. 8 is obtained by optimizing the upper-level objective \(F(\boldsymbol{W}, \lambda_{opt})\) via gradient steps.

6 Experiments↩︎

6.1 Experiments setting↩︎

6.1.0.1 Datasets.

We evaluate and compare our approach to GCD state-of-the-art approaches across six different natural image datasets. More concretely, this includes three well-known generic object recognition datasets CIFAR10 [34], CIFAR100 [34], ImageNet-100 [35], as well as the recent semantic shift benchmark suite (SSB) [36] which is composed of the three fine-grained datasets CUB [37], Stanford Cars [38] and Herbarium19 [39]. Note that the datasets on SSB bring an additional challenge to the performance of the baselines. CUB and Stanford Cars contain fine-grained categories, which are arguably harder to distinguish than generic object classes. Herbarium19 is a long-tailed dataset, which reflects a real-world use case with large class imbalance, and large intra-class and low inter-class variations.

We follow the original GCD setting [5] to split the original training set of each dataset into labeled and unlabeled subsets. More concretely, half of the image samples corresponding to the \(K^{\text{old}}\) known classes are assigned to the labeled subset, whereas the remaining half are assigned to the unlabeled subset. The latter also contains all the image samples from the remaining classes present in the original dataset, which we consider as the novel classes. In this way, the unlabeled subset is composed of instances from \(K\) different classes. Table ¿tbl:tab95GCD95datasets95details? details for each dataset the selected number of classes, as well as the number of corresponding examples present in each subset, w.r.t. the generalized category discovery setting introduced in [5].

6.1.0.2 Evaluation protocol.

To evaluate our model we follow the evaluation protocol presented in GCD [5]. In particular, for the partitioning task, we first employ the Hungarian algorithm to solve the optimal cluster-to-class assignment jointly on both known and novel class examples, i.e. for all the predicted clusters at once. Note that our partial semi-supervised training already enables to correctly align beforehand the clusters corresponding to the known classes with real-class labels. Then, we use this optimal label-assignment solution to estimate the overall partitioning accuracy (ACC) for all classes (All), for known classes (Old), and for novel classes (New). In the evaluation, we also report the estimated number of classes \(\hat{K}\) in the unlabeled set and the corresponding error \(Err=\frac{|\hat{K}-K|}{K}\), with \(K\) the real number of classes for each dataset.

6.1.0.3 Implementation details:

• Encoder \(g_{\boldsymbol{\theta}}\): As in [5], we employ the vision transformer ViT-B-16 [40] as our backbone encoder \(g_{\boldsymbol{\theta}}\) (i.e. the feature extractor). It is first pre-trained on the unlabeled dataset ImageNet [35] with DINO [21] self-supervision. Then, it is fine-tuned on each GCD dataset of interest with a semi-supervised contrastive loss composed of an unsupervised noise contrastive term [41] and a supervised contrastive term [42]. It is empirically demonstrated in [5] that this pre-training procedure achieves robust feature representations. The resulting feature dimension is 768 per input image.

• Partitioning model \(f_{\boldsymbol{W}}\): Our partitioning model follows the architecture of a standard linear classifier. We first initialize \(f_{\boldsymbol{W}}\) prototypes \(\boldsymbol{W}\) with the centroids produced by the semi-supervised k-means (ssKM) clustering model⁵ on the entire feature map set \({\mathcal{Z}}\). The maximum number of clustering iterations for ssKM is set to \(100\). Then, we train \(f_{\boldsymbol{W}}\) with the standard Adam optimizer [43], with a learning rate of \(0.001\) and a weight decay of \(0.01\), for \(1000\) epochs during the partitioning task, but only for \(500\) epochs during the search of \(\hat{K}\) in order to reduce the computational cost. We set the training batch size equal to the size of the dataset which is quite feasible in terms of memory and computation since our approach only requires the pre-computed feature maps.

• Conditional entropy weight \(\lambda\): During the search of the number of classes, we simply set \(\lambda=1\) for the unsupervised discriminative clustering step. However, during the partitioning task where the number of classes is fixed, i.e. with \(K\) assumed to be known or to be equal to \(\hat{K}\), we automatically select the optimal value for \(\lambda\) in the interval \((0,1]\), as previously detailed in Sec. 5.

6.2 Main results↩︎

In this section, we perform a comprehensive empirical evaluation of our method and compare it to GCD [5], as well as several adapted state-of-the-art approaches. In particular, RankStats+ and UNO+ are the adapted versions from RankStats [14] and UNO [19], which were originally developed for the NCD task. Furthermore, results when applying simply k-means [44] on the raw extracted features from DINO are also reported. Scores for k-means, RankStats+, UNO+ and GCD [5] are reported from [5]. We have also evaluated ORCA with its original ResNet architecture [45] by using authors code⁶, as well as with the more competitive ViT-B-16 architecture [40], as both GCD [5] and our method resort to this architecture. In addition, in order to better highlight the superior performance of the proposed approach compared to previous mutual information strategies, we have also adapted the InfoMax approaches RIM [25] and TIM [27], previously discussed in Sec. 2, to this novel setting. Specifically, TIM and the semi-supervised version of RIM were originally designed to deal with semi-labeled datasets, where both the labeled and unlabeled sets contain examples from the same classes. Thus, we have expanded the number of prototypes in RIM and TIM, and their resulting prediction output vector from the \(K^{\text{old}}\) to \(K\). For the sake of fairness, we use the same training hyper-parameters and initialization prototypes as in our approach, as detailed in Sec. 6.1. Furthermore, we also applied RIM and TIM on top of the same fixed feature extractor, similarly to ssKM in GCD [5] and our approach.

We first focus on the partitioning task (¿tbl:tab95accv295all95dsets?), which evaluates the label assignment performance ACC on the unlabeled samples. Following the original GCD setting [5], we assume the real number of classes \(K\) to be known.

Comparisons with the GCD state-of-the-art. Overall, we can observe that our approach PIM significantly outperforms the state-of-the-art methods UNO+, RankStat+ and GCD [5], with a consistent performance improvement ranging from 3% on CIFAR10 up to 11% on CUB, when considering \(\mathtt{ALL}\) categories. Compared to ORCA, one can notice that improvement gain brought by our method is particularly significant on the fine-grained datasets, where it ranges from 9.3% on Stanford Cars, up to 24.7% on CUB. In contrast, the performance differences are less remarkable across general datasets. In particular, PIM does not outperform ORCA-ViTB16 in the simpler CIFAR-10 dataset, whereas it brings 8-9% in performance gains in CIFAR-100 and ImageNet-100, arguably more complex datasets. These results suggest that our approach is more suitable in scenarios where the total number of classes is relatively large, and potentially presenting a higher degree of class imbalance.

Comparison with adapted RIM [25] and TIM [27]. The results obtained by the adapted semi-supervised RIM, TIM, and our approach PIM show the overall superiority of mutual information based methods compared to GCD [5]. In addition, one can observe that while RIM and TIM present very similar results. This behaviour could be explained by the fact that these two adaptations to the GCD problem amount to use the same fixed loss function, and performance differences may be due to the classifier choice for the conditional model. Indeed, RIM uses a multiclass logistic regression, whereas the soft-classifier of TIM measures the l2 norm between prototypes (i.e. classifier weights) and L2-normalized embedded features. Last, and more importantly, we can observe that methods based on the standard mutual information yield suboptimal results compared to the proposed approach. We hypothesize that the reason for these differences is two-fold: (i) RIM and TIM compute the marginal entropy term exclusively over the unlabeled data, while we compute it over the entire distribution. In other words, PIM maximizes the mutual information over the whole dataset, which enables to better capture the entire data distribution; (ii) thanks to the proposed bi-level optimization process, the optimal lambda parameter for each dataset can be estimated automatically (as detailed in Section 5). We stress that our hypothesis is supported by empirical evidence in Section 6.3.

6.3 Ablation studies↩︎

Along these ablation studies, we focus the attention on the challenging fine-grained datasets, as the interest for each of our technical choices can be better observed.

6.3.1 Automatic finding of optimal \(\lambda\): handling both short-tailed and long-tailed datasets↩︎

We now motivate the interest of estimating the most appropriate \(\lambda\) value for each dataset by using the proposed automatic finding strategy presented in 5. \(\lambda_{GT}\) in the Figures (a), (b), (c) is the \(\lambda\) value obtained when using the ground-truth labels, i.e. the value which provides the maximum performance on the unlabeled set, and \(\hat{\lambda}\) in Figures (d), (e), (f) is the estimated optimal value. Interestingly, Figures 1 (a), (b), (c) first show how much the performance \({\mathtt{ALL}}\) ACC can significantly vary depending on the selected \(\lambda\), hence motivating the interest of finding the optimal \(\lambda\) value for each dataset. Second, Figures 1 (d), (e), (f) validate our hypothesis that selecting the \(\lambda\) value that maximizes the Lab ACC on the labeled data in our unconstrained conditional version, also maximizes the ACC on all the unlabeled data (\({\mathtt{ALL}}\) ACC) when using the constrained version instead, showing the correlation between both. Third, w.r.t the classes frequencies observed on each dataset (See Figures 1 (g), (h), (i)), it is also interesting to note that a small \(\lambda\) value provides better results on a dataset with uniform class distributions such as CUB, whereas a higher \(\lambda\) value is more appropriate on the imbalanced dataset Herbarium19. Indeed, \(\lambda\) controls the relative effect of the marginal entropy term in 7 . Thus, these results show that automatically selecting \(\lambda\) can mitigate the encoded bias of the standard mutual information towards balanced partitions by giving more importance to the conditional entropy term in long-tailed (imbalanced) scenarios.

6.3.2 Effect of each loss term↩︎

We now evaluate the contribution of each term in our learning objective 7 . In particular, ¿tbl:tab95loss95terms95effect95fine95grained95dsets? highlights the corresponding effect of the conditional entropy, the marginal entropy, and the proposed penalty constraint (i.e. replacing conditional entropy with CE on labeled points). From these results, we can draw three different observations: 1) Only minimizing the conditional entropy term produces degenerated solutions, as expected. 2) Maximizing the marginal entropy term, as well as enforcing the proposed constraint, prevents these undesired degenerated solutions. 3) Maximizing the marginal entropy term on the entire dataset, and hence maximizing as well the mutual information on the entire dataset, as we propose, further enhances the performance.

6.4 Towards a practical setting↩︎

6.4.1 Estimating the number of classes↩︎

In order to find the number of classes, we follow the strategy proposed in GCD [5] (which we refer to as Max-ACC (GCD)), but we replace the k-means clustering stage with the unconstrained (i.e. unsupervised) version of our method PIM, referred to as Max-ACC (PIM). The results from these methods are reported in ¿tbl:tab95estimate95hat95K?. Overall, the proposed combination Max-ACC (PIM) is more appropriate than Max-ACC (GCD) [5] on both generic and fine-grained datasets, except on CIFAR-100 where Max-ACC (GCD) [5] finds the real number of classes.

6.4.2 Performance when the number of classes is unknown↩︎

While we followed the standard practices for the partitioning task in the experiments of Section 6.2, we argue that having access to the number of expected classes is an unrealistic assumption. Thus, we now relax this assumption by repeating the partitioning experiments with the estimated value of \(\hat{K}\) (See ¿tbl:tab95estimate95hat95K?) instead of the real value \(K\). For the GCD method [5], we used the authors code⁷, except on CIFAR-100 where we directly reported scores from Tab. ¿tbl:tab95accv295all95dsets? because \(\hat{K}=K\). These results, which are reported on ¿tbl:tab95accv295using95hat95k?, demonstrate the superiority of our method even in this more challenging scenario. This suggests that our formulation serves as a more robust solution in the absence of prior knowledge about \(K\) for the GCD task.

            \vskip\caption{\textbf{Realistic GCD partitioning.} ACC scores are obtained by assuming $\hat{K}$ (See Tab. \ref{tbl:tab95estimate95hat95K}) as the number of expected classes.}\label{tab95accv295using95hat95k} -0.15in

\end{table}

7 Conclusion↩︎

In this work, we propose a simple yet effective alternative for Generalized Category Discovery. In particular, we introduce a parametric family of mutual information objectives, which we tackle with a bi-level optimization formulation. Our solution allows to estimate the relative weight of the marginal-entropy term automatically, which mitigates the class-balance bias inherent in standard information maximization. Our empirical validation demonstrates that by learning the optimal weight that controls the relative effect of the marginal-entropy, our model deals effectively with both short-tailed and long-tailed datasets. Indeed, our presented formulation achieves new state-of-the-art results in GCD tasks, outperforming existing solutions across the different benchmarks by a significant margin. Moreover, in contrast to prior work, we relax the assumption that the number of classes in the unlabeled set is given, which facilitates the robustness of the proposed model in this more challenging scenario, and its scalability to more realistic settings. Furthermore, it is noteworthy to mention that our formulation is flexible and could, therefore, be coupled with any trained feature extractor. Thus, we hope that the proposed framework will be useful for future research and development to solve the GCD problem for real-world applications.

Limitations. A common limitation in the current GCD learning paradigm stems from the fact that the models require access to the entire target unlabeled dataset at test time. Needless to say, this strong assumption might hinder the scalability of these approaches when the target set is composed of a small number of images, or when samples appear sequentially. As in [5], we assume that optimal values for finding \(\lambda\) and \(K\) are obtained when the ACC is maximized on the available labeled points. Despite that our extensive experiments empirically confirm that this assumption is promising in practice, it could be interesting to find a metric that could simultaneously consider the novel classes.

8 Appendix↩︎

8.1 Semi-supervised k-means (ssKM)↩︎

Parameters initialization for ssKM. We denote the centroid parameters of ssKM as \(\boldsymbol{W} = (\boldsymbol{W}^{old}, \boldsymbol{W}^{new})\), where \[\left\{ \begin{array}{ll} \boldsymbol{W}^{old}=(\boldsymbol{w}_k^{old})_{1 \leq k \leq K^{old}} \\ \boldsymbol{W}^{new}= (\boldsymbol{w}_k^{new})_{K^{old}+1 \leq k \leq K^{old}+K^{new}}. \end{array} \right.\] Here, \(\boldsymbol{W}^{old}\) and \(\boldsymbol{W}^{new}\) are the centroids for the known and novel classes, respectively. We initialize \(\boldsymbol{W}\) by using both the labeled set and the unlabeled set jointly. As in [5], we first produce the centroids for the known classes using the labeled set, such that: \(\boldsymbol{w}_k^{old} = (\sum_{\boldsymbol{z}_i \in \mathcal{Z}_L} y_{i,k} \boldsymbol{z}_i) / \sum_{\boldsymbol{z}_i \in \mathcal{Z}_L} y_{i,k}.\) Then, we initialize the centroids corresponding to the novel classes with kmeans++ initialization. ssKM objective and clustering process. As in [5], we update the cluster centroids of ssKM algorithm, with the following objective: \[\begin{align} L_{\text{ssKM}}({\mathbf{Y}};{\mathbf{U}};\boldsymbol{W}) &= \left( \sum_{k=1}^K \sum_{\boldsymbol{z}_i \in \mathcal{Z}_L} y_{i,k} \|\boldsymbol{z}_i- \boldsymbol{w}_{k}\|_2 \right) \\ &+ \left(\sum_{k=1}^K \sum_{\boldsymbol{z}_i \in \mathcal{Z}_U} u_{i,k} \|\boldsymbol{z}_i- \boldsymbol{w}_{k}\|_2 \right), \end{align} \label{eq95ssKM}\tag{10}\] where \(\|.\|_2\) denotes the Euclidean distance, and \(\boldsymbol{u}_i = (u_{i,k})_{1 \leq k \leq K}\) denotes the latent binary vector assigning point \(\boldsymbol{z}_i\) to cluster \(k\). \(\mathbf{U} \in \{0, 1\}^{NK}\) denotes the latent assignment matrix composed of the latent binary vectors. ssKM algorithm proceeds with the same block-coordinate descent approach as the standard unsupervised k-means [44]. Thus, in order to minimize \(L_{\text{ssKM}}\), it proceeds with the following cluster assignment and centroid updates cycle:

• U-update: Do the label assignment of the unlabeled points such that, \[u_{i,k} = \left\{ \begin{array}{ll} 1 & if\mathop{\mathrm{\arg\min}}_{k} \|\boldsymbol{z}_i- \boldsymbol{w}_{k}\|_2 = k \\ 0 & otherwise. \end{array} \right.\]

• \(\boldsymbol{W}\)-update: Find \(\mathop{\mathrm{arg\,min}}_{\boldsymbol{W}} L_{\text{ssKM}}({\mathbf{Y}};{\mathbf{U}};\boldsymbol{W})\)

Note that the latent label assignment update step is not applied on the labeled points, for which we simply keep using the available ground-truth labels in the first term in 10 , all along the clustering process.

8.2 Supplementary results↩︎

Effect of prototypes initialization. All along our article experiments, we found empirically consistent to use ssKM (detailed in Sec. 8.1) centroids to initialize the prototypes of the proposed partitioning model PIM. In this section, in order to endorse this choice, we empirically emphasize across Tab. ¿tbl:tab95ctrds95init95fine95grained95dsets? the effect of prototypes initialization on PIM prediction performances. We thus compare the following three different possible initializations (init): -ssRDM init consists of using labeled points for prototypes of known classes, and random points for prototypes of novel classes; -ssKM++ init consists of using labeled points for prototypes of known classes, and kmeans++ points for prototypes of novel classes (w.r.t. known class prototypes); -ssKM init consists of using ssKM centroids as init prototypes. The results observed on Tab. ¿tbl:tab95ctrds95init95fine95grained95dsets? show that the proposed approach is overall almost insensitive to prototypes initialization. One can also note that PIM performances are overall slightly improved when using directly ssKM++ init.

            \vskip\caption{\textbf{Prototypes initialization effect} on PIM ACC performances on fine-grained and generic datasets.}\label{tab95ctrds95init95fine95grained95dsets} -0.05in

\end{table}

Effect of PIM objective components on generic datasets. Tab. ¿tbl:tab95loss95terms95effect95generic95dsets? shows the effect of PIM objective components on the generic datasets, in particular on CIFAR100 and ImageNet-100.

            \vskip\caption{\textbf{PIM objective components effects} on generic datasets in terms of ACC scores.}\label{tab95loss95terms95effect95generic95dsets} -0.05in

\end{table}

Standard deviation (\(\pm\)) of PIM for the generalized category partitioning is shown on Tab.¿tbl:tab95std95all95dsets?.

            \vskip\caption{\textbf{Standard deviation ($\pm$)} for ACC scores of PIM (Tab. \ref{tbl:tab95accv295all95dsets}) over 5 trainings, across fine-grained and generic datasets.                }\label{tab95std95all95dsets} -0.05in

\end{table}

References↩︎