1 Introduction↩︎

The intriguing ability that humans can learn new knowledge with few examples and then generalize it to novel environments [1], [2] has led to a community-wide enthusiasm in artificial intelligence towards few-shot learning (FSL)¹, especially in the applications of visual recognition [3], [4], machine translation [5], [6], reinforcement learning [7], [8], etc.

Specifically for visual recognition, FSL is generally studied in a two-stage setting [9]–[11]. At first, the few-shot learner — a meta-model that aims to generate a classifier given few-shot examples — is provided with a large number of training examples collected from a set of base classes. After learning from these data, the few-shot learner then receives a small number of training examples for a set of novel classes that it needs to generate a classifier to recognize. The quality of the few-shot learner is measured by the generated classifier’s accuracy on recognizing these novel classes. By default, the two sets of classes are disjoint but related (e.g., both about animal species). Thus, to perform well in FSL, the few-shot learner must efficiently re-use what it learned (e.g., features) from the base classes for the novel classes.

Meta-learning, or learning to learn, is arguably the most popular approach to tackle FSL [7], [13]–[16]. The core idea is to simulate the few-shot scenario of learning with novel classes using the base class data. That is, instead of using the base class data in a conventional way just like training a standard multi-class classifier [17], meta-learning approaches sample multiple few-shot tasks from the base class data and use them to meta-train the few-shot learner to excel in the scenario it will encounter in receiving novel class data, even though the base class data are indeed many-shot. In other words, what the few-shot learner learns from the base class data is the ability to generate a classifier using few-shot data.

To implement meta-learning for FSL, most existing works follow [10] to formulate each simulated few-shot task by a support set and a query set whose data are sampled from the same subset of base classes. The support set is the few-shot training data that is used by the few-shot learner to generate the classifier; the query set is used to evaluate the resulting classifier’s quality. In essence, the classification loss on the query set is the main supervision that is used to meta-train the few-shot learner for how to learn from the support set.

In this paper, we argue that the classification loss on the query set is not an efficient form of supervision, especially when the query set size is limited. Concretely, in meta-training the few-shot learner, one usually constructs a mini-batch by sampling one or several few-shot tasks. Thus, the limited query set size could result from the memory constraint or a large support set that consumes most of the space of a mini-batch². 1 gives an illustration, in which we experimented with different numbers of shots: the larger the shot number is for the support set, the smaller the query set size is. We showed that, even in the standard 5-way 5-shot FSL setting, meta-learning methods (i.e., ProtoNet) [13] can be outperformed by a simple nearest centroid classifier (i.e., PT-EMB) [9], [18] whose features are trained with all the base class data in a conventional, non-meta-learning way. Moreover, the gap between these two methods becomes more pronounced in a larger shot setting where the query set size shrinks³. While one may argue that meta-learning should just be used in few-shot settings, the fact that it is effective only in a narrow shot range greatly limits its applicability.

The above issue motivates us to re-visit another implementation of meta-learning, which is to directly teach the few-shot learner to generate a strong classifier like the one trained using many-shot data [16], [19]–[21]. In this way, the query set in a simulated few-shot task is replaced by a “target” classifier pre-trained using the (entire) base class data⁴. The loss for meta-training the few-shot learner is instantiated by a distance measure between the generated classifier and the target classifier in the parameter space. This implementation bypasses the limit of query set sizes. However, its applicability is limited by another factor: the lack of a suitable distance metric in the high-dimensional parameter space. Therefore, prior works pre-train a feature extractor in a non-meta-learning way, and constrain the few-shot learner to generate only the last fully-connected layer.

In this paper, we propose a novel meta-learning approach for FSL that incorporates the advantages of these two ways of implementations. On the high level, we follow the notion of constructing target classifiers to teach the few-shot learner, which bypasses the limit of query set sizes. However, instead of meta-training the few-shot learner to generate a classifier that is exactly like the target classifier in the parameter space, we meta-train the few-shot learner to generate a classifier that predicts like the target classifier on test (i.e., query) examples. That is, in our approach, a simulated few-shot task contains both the target classifier and the query set besides the support set. Given the support set, our approach encourages the few-shot learner to generate a classifier that would make predictions (e.g., predicted logits or class probabilities) on the query examples similar to the target classifier. By using the difference in predictions rather than in the parameter space to supervise the few-shot learner, our approach enables the few-shot learner to learn to generate the entire classifier (i.e., the feature extractor plus the final fully-connected layer). Moreover, measuring the difference in predictions disentangles the model architectures used by the target classifier, the few-shot learner, and the generated classifier, making our approach a plug-and-play meta-learning objective that is compatible with most existing meta-learning based FSL methods. Even if the query examples are limited, the few-shot learner can receive extra and more informative supervision — how a stronger target classifier makes predictions on them — beyond the one-hot query example labels.

We investigate multiple ways of constructing the target classifiers. One intuitive approach is to train a multi-class classifier using the entire base class data [17]. However, since a simulated few-shot task usually contains only a subset of base classes (e.g., \(5\) classes from the \(64\) base classes [10]), the prediction of this target classifier must be properly filtered and calibrated. We therefore propose to leverage this pre-trained classifier’s feature extractor to (a) construct a target nearest centroid classifier [9] or (b) dynamically train a target logistic regression, both using all data from the subset of base classes. Besides, we further investigate multiple ways of querying the target classifiers’ predictions during meta-training. We found that imposing variations to the query data (e.g., resizing the images to be larger or adding noise) improves the performance, which is reminiscent of the recent studies in [22]. During meta-testing on novel classes, we do not impose any variations to ensure a fair comparison to existing methods. We name our approach LastShot: Learning with A Strong Teacher for few-SHOT learning.

We validate LastShot in combination with representative FSL methods [13], [23]–[25] on multiple benchmarks, including miniImageNet [10], tieredImageNet [26], CIFAR-FS [27], FC-100 [28], and CUB [29]. LastShot consistently improves these methods and achieves the state-of-the-art accuracy. We further study LastShot under varying numbers of shots in the support set. LastShot shows highly robust performance, especially when the shots go larger and existing meta-learning based methods are outperformed by simple nearest centroid approaches [9] (see 1). The code in available at https://github.com/Han-Jia/LastShot.

The contributions of this work are as follow:

• We propose a novel meta-training objective that encourages the few-shot learner to generate classifiers that perform like the corresponding target strong classifiers. Our proposed objective is an easily plug-and-play term to existing meta-learning based FSL methods.

• We present an efficient way to construct the target strong classifier. While we meta-train the few-shot learner with few-shot tasks, we use the strong classifiers trained in a non-meta-learning way with ample examples to supervise the meta-training process.

• We conduct extensive experiments to validate our approach, which includes multiple benchmark datasets and multiple settings. Our LastShot can consistently outperform conventional meta-learning and non-meta-learning based methods across a range of shot numbers.

1.1 Comparisons to related work↩︎

On the surface, our approach is reminiscent of knowledge distillation [30]. However, there are several notable differences from other FSL methods that are based on knowledge distillation [31]–[34]. For instance, Tian et al. [34] studied how distillation with self-ensemble improves the pre-trained features for the downstream FSL task. In contrast, we focus on how to fundamentally improve meta-learning based FSL methods by leveraging distillation in the meta-training process. That is, our approach is complementary to their efforts as we apply distillation at a different training stage (meta-training vs. feature pre-training)⁵. Wang et al. [33] studied how to generate extra support set examples such that the generated classifier by the few-shot learner could perform similarly to the target classifier. In contrast, we do not introduce the extra data generation task, making our method more easily applicable. We also note that, our method is different from self-distillation [36]–[39] — our teacher is a classifier trained with ample data while our student is a few-shot learner. Please see 2 for more details and discussions.

2 Related Work↩︎

2.1 Few-shot learning (FSL)↩︎

FSL generally focuses on the \(C\)-way classification scenario in which the few-shot learner receives few labeled examples for \(C\) new classes and needs to generate a classifier to recognize them (\(C\) is usually set between \(5\) to \(30\)). FSL can be studied from roughly three aspects from data, models, and algorithms [40]. We describe non-meta-learning based FSL methods in this subsection, and describe meta-learning based methods in the next subsection.

From the aspect of data, FSL can be resolved if one has sufficient data for the \(C\) novel classes. Data augmentation approaches hallucinate additional examples of novel classes based on their few examples [41]–[43]. The hallucinated examples are used together with the original ones to train a better classifier for the novel classes. Since there are more training examples, it prevents the model from over-fitting.

From the aspect of models, FSL can be resolved if one can extract features that faithfully capture the semantic and discriminative information of each example to support nearest-neighbor classifiers. Feature or metric learning approaches learn a feature extractor or metric that encourages examples from the same classes to be closer and examples from different classes faraway [18], [44], [45]. The resulting features or metrics are used in a linear or nearest-neighbor classifier that is trained on the examples of the novel classes [9], [46], [47]. Another direction to resolve FSL from the aspect of models is to construct generative models, which are learned to generate visual content conditioned on stochastic binary variables that indicate object identities. Recognizing novel classes involves adding new variables to the model and estimating the parameters associated with it based on few labeled data [48]–[50].

From the aspect of algorithms, one could alter the optimization strategy to search for a better model given limited data. Hypothesis transfer reuses a well-trained model from a related dataset. Through fine-tuning it, we can leverage the knowledge learned from the related dataset to avoid over-fitting to few-shot data [11], [51]. [52]–[54] proposed special regularization strategies used during fine-tuning.

2.2 Meta-learning for FSL↩︎

Here we mainly discuss how meta-learning helps FSL. Meta-learning constructs simulated few-shot tasks, called episodes [10], [15], from the base class data to mimic future few-shot tasks, and trains a few-shot learner to maximize the classification accuracy on the episodes. It is expected that the few-shot learner could be generalized to the few-shot tasks with novel classes. Various strategies have been investigated to implement the few-shot learner. For example, based on the meta-trained feature extractor, the few-shot learner classifies the query instances with (adapted) embeddings [3], [10], [13], [28], [55]–[57]. Based on the meta-trained model initialization [7], [58]–[62], the few-shot learner can adapt it to a good classifier for each few-shot task with a small number of gradient descent steps. The optimization strategy for a few-shot learner to fit the few-shot examples can also be meta-trained and shared across episodes [63]–[65], e.g., the learning rate [15], [66]. An image generator could also be meta-trained to augment the support set [67]. We note that meta-learning can explicitly take characteristics of few-shot tasks into account. For instance, the number of novel classes \(C\) and the number of labeled examples per class (i.e., shots) can be specified in the episodes.

Meta-learning based FSL has been applied in various fields, such as unsupervised learning [68], human motion prediction [20], domain adaptation [69], image generation [70], and object detection [71], [72]. Some novel directions have also been explored recently. For example, [73], [74] leveraged the graph propagation to improve meta-learning; [75], [76] constructed a generalized FSL objective towards adaptive calibration strategies between base and novel classes; [77] investigated specialized data augmentations; [78] studied how to identify invariant features in meta-learning. Auxiliary self-supervised tasks are also incorporated to improve the generalization ability of the resulting few-shot learners or classifiers [79]–[81].

Most of the meta-learning approaches rely on the classification loss on the query sets to drive the meta-training of the few-shot learner. [16], [21], [82] were the first to propose the notion of constructing target classifiers to guide few-shot learners. Specifically, they measured the Euclidean distance between the target classifier and the classifier generated by the meta-learner as the loss. However, due to the large number of parameters, they only managed to meta-train the few-shot learner to generate the linear classifiers (i.e., the final fully-connected layers).

Some recent methods emphasize the importance of pre-trained features [19], [23], [34], [35], [44], [64]. In particular, the parameters of the backbone model are initialized by optimizing a classification objective over the base class data, which results in more discriminative features. Fine-tuning over the backbone leads to strong FSL baselines [11], [51]. The influence of the number of shots (e.g., between meta-training and meta-testing) is discussed in [28], [83], [84].

We empirically find that the effectiveness of meta-learning diminishes sharply with increasing shots. Specifically, compared to the simple classifiers built upon the pre-trained embeddings, the episodic meta-training could not get further improvement. We propose a plug-and-play module LastShot for meta-learning based FSL to encourage the few-shot learner to generate classifiers which perform like strong classifiers. This helps the resulting few-shot learner outperform other methods across different shot numbers.

2.3 Knowledge distillation↩︎

As mentioned in 1.1, our approach is reminiscent of knowledge distillation [30] as we leverage target classifiers trained on ample examples to supervise the meta-training of the few-shot learner. In this subsection, we therefore review knowledge distillation and contrast our usage of it to other existing methods for few-shot learning [31]–[34].

Knowledge distillation (KD) [30] is a training strategy that leverages the dark knowledge from one (teacher) model to facilitate the learning progress of another (student) model. One popular way to extract the dark knowledge is by the instance-wise prediction, e.g., the posterior probability on a data instance [30], [85], [86], which provides the relative comparison among classes, in contrast to the conventional one-hot label. By matching the predictions of a “student” model to those of a “teacher” model, the student model obtains richer learning signals and becomes more generalizable. The dark knowledge for KD can be in the form of soft labels [30], hidden layer activation [87], [88], and comparison relationships [89]. Knowledge distillation has also been exploited for model interpretability [90] and incremental learning [91], [92], etc.

Besides [33], [34] mentioned in 1.1, KD was also employed in two recent works of FSL [31], [32], but again based on very different motivations and manners from our LastShot. [32] applied KD mainly to construct a portable few-shot learner; [31] applied self-supervised learning [93], [94] to train robust features and use them to guide the few-shot learner. Both of them only studied a single FSL method, e.g., [7]. In contrast, our approach is more general, compatible with many meta-learning based FSL methods and requiring no self-supervised learning steps. Wang et al. [33] investigated constructing a data generator to create extra support set examples; the generator is learned via KD, from a many-shot target classifier. In LastShot, we take advantage of a plug-and-play loss term to distill knowledge from the target classifier more easily.

Self-distillation is a special case of KD which distills knowledge from the model itself, i.e., the teacher can be the previous generation of the model along the training progress [36], [37], [95]. Tian et al. [34] applied self-distillation in the model pre-training stage to obtain more discriminative features. Our LastShot works in an orthogonal direction to directly improve the meta-learning based FSL methods. Different from self-distillation, the target classifier in LastShot is not the previous few-shot learner or feature extractor, but a stronger classifier trained with ample base class data.

3 Our Approach: LastShot↩︎

In this section, we first introduce the notation of few-shot learning (FSL) for visual recognition and highlight the overall framework of meta-learning for FSL. We then introduce our approach LastShot.

3.1 Notation↩︎

In FSL, a few-shot learner is first provided with a training set \(\mathcal{D}_\text{base} = \{({\boldsymbol{x}}_1, y_1), \dots, ({\boldsymbol{x}}_N, y_N)\}\) that contains \(N\) labeled images from \(B\) base classes; i.e., \(y_n\in\{1,\cdots, B\}\). Here, each class is supposed to have ample training examples. The few-shot learner is then provided with a support set \(\mathcal{D}_\text{support}'\) of labeled images from \(C\) novel classes \(\{B+1,\cdots, B+C\}\), in which each novel class has \(K\) examples. The goal of the few-shot learner is to generate a classifier that can accurately classify among the \(C\) novel classes. This FSL setting is referred to as the \(C\)-way \(K\)-shot setting.

3.2 Meta-learning for few-shot learning↩︎

The core concept of meta-learning is to use the base class data \(\mathcal{D}_\text{base}\) to simulate the FSL scenario for meta-training the few-shot learner, before it receives the few-shot data from the novel classes. Namely, meta-learning samples multiple FSL tasks \(\mathcal{T}\) from \(\mathcal{D}_\text{base}\), each contains a \(C\)-way \(K\)-shot support set \(\mathcal{D}_\text{support}\) whose class labels are from \(\{1,\cdots, B\}\). Meta-learning then inputs each support set to the few-shot learner \(\mathcal{A}\), whose goal is to output/generate a classifier that can classify the \(C\) input classes well. To evaluate the quality of \(\mathcal{A}\), meta-learning approaches usually sample a corresponding query set \(\mathcal{D}_\text{query}\) for each \(\mathcal{D}_\text{support}\), whose examples are from the same \(C\) classes. The quality of \(\mathcal{A}\) is measured by the outputted classifier’s accuracy on \(\mathcal{D}_\text{query}\).

Putting all these notations together, we can now formulate the meta-learning problem as follows \[\begin{align} \min_\mathcal{A} \sum_{\mathcal{T}\sim\mathcal{D}_\text{base}} \;\sum_{({\boldsymbol{x}},y)\in\mathcal{D}_\text{query}} \ell(\mathcal{A}(\mathcal{D}_\text{support})({\boldsymbol{x}}), y), \label{eq95meta} \end{align}\tag{1}\] where each \(\mathcal{T}\) is a tuple \((\mathcal{D}_\text{support},\mathcal{D}_\text{query})\); \(\mathcal{A}(\mathcal{D}_\text{support})\) is the \(C\)-way classifier outputted by \(\mathcal{A}\), after taking the support set \(\mathcal{D}_\text{support}\) as input. \(\ell\) is a loss function on the prediction; e.g., the cross-entropy loss. As shown in 1 , the query set \(\mathcal{D}_\text{query}\) is the major supervision that drives the meta-training of the few-shot learner \(\mathcal{A}\).

After \(\mathcal{A}\) is properly meta-trained, we then apply it to the true \(\mathcal{D}_\text{support}'\) of the novel classes to construct the classifier.

3.2.1 Exemplar few-shot learners↩︎

We review several representative few-shot learners. In model-agnostic meta-learning (MAML) [7], the few-shot learner \(\mathcal{A}\) is modeled by a neural network classifier \(f_{\boldsymbol{\theta}}\) with weights \(\boldsymbol{\theta}\). Denote by \(\mathcal{L}(\boldsymbol{\theta})\) the differentiable loss of \(f_{\boldsymbol{\theta}}\) computed on the support set \(\mathcal{D}_\text{support}\), \(\mathcal{A}(\mathcal{D}_\text{support})\) is formulated, in its simplest form, as an updated neural network \(f_{\hat{\boldsymbol{\theta}}}\) with weights \(\hat{\boldsymbol{\theta}}\): \[\begin{align} \hat{\boldsymbol{\theta}} = \boldsymbol{\theta}- \alpha \times \nabla_{\boldsymbol{\theta}} \mathcal{L}(\boldsymbol{\theta}),\label{e95MAML95inner} \end{align}\tag{2}\] where \(\alpha\) is a step size. That is, MAML performs a single (or few) gradient descent update(s) according to \(\mathcal{L}(\boldsymbol{\theta})\) to obtain the outputted classifier. The meta-learning problem in 1 thus aims to find a good initialization \(\boldsymbol{\theta}\) such that the neural network \(f_{\hat{\boldsymbol{\theta}}}\) can perform well on \(\mathcal{D}_\text{query}\) after only a single (or few) update(s). Many meta-learning approaches are built on MAML [24], [25], [58], [64], [66], [96]–[98].

Another popular way to model \(\mathcal{A}\) is via a feature extractor \(f_{\boldsymbol{\theta}}\). For example, in Prototypical Network (ProtoNet) [13], \(\mathcal{A}(\mathcal{D}_\text{support})({\boldsymbol{x}})\) is formulated as \[\begin{align} \hat{y} = \mathop{\mathrm{arg\,min}}_c \left\|f_{\boldsymbol{\theta}}({\boldsymbol{x}}) - \sum_{{\boldsymbol{x}}'\in\mathcal{D}_\text{support, c}} \frac{f_{\boldsymbol{\theta}}({\boldsymbol{x}}')}{|\mathcal{D}_\text{support, c}|}\right\|_2^2, \label{e95ProtoNet} \end{align}\tag{3}\] where \(\mathcal{D}_\text{support, c}\) is the subset of \(\mathcal{D}_\text{support}\) that contains examples of class \(c\). That is, ProtoNet performs nearest centroid classification and 3 aims to learn a good feature extractor. Similar approaches are [3], [10], [23], [82], [99].

3.2.2 Meta-learning with the target classifier↩︎

In 1 , the query set \(\mathcal{D}_\text{query}\) is the main supervision for meta-training the few-shot learner \(\mathcal{A}\). That is, \(\mathcal{D}_\text{query}\) is used to measure the loss of the classifier \(\mathcal{A}(\mathcal{D}_\text{support})\) outputted by \(\mathcal{A}\). Such a loss is then used to derive gradients to update \(\mathcal{A}\).

Instead of using the query set \(\mathcal{D}_\text{query}\) to measure the quality of \(\mathcal{A}(\mathcal{D}_\text{support})\) in meta-training the few-shot learner \(\mathcal{A}\), [16], [21] constructed the “target” classifier \(h^\star\), which is the \(C\)-way classifier trained with ample labeled examples of the same classes in \({D}_\text{support}\). The target classifier \(h^\star\) can be seen as the classifier that we hope the few-shot learner \(\mathcal{A}\) to output given \(\mathcal{D}_\text{support}\): after all, what we ask the few-shot learner to do is to generate a classifier as if it were trained with ample examples. To this end, one can measure the loss of \(\mathcal{A}(\mathcal{D}_\text{support})\) by its difference from the target classifier \(h^\star\) in the parameter space, and use the loss to meta-train the few-shot learner \(\mathcal{A}\). We note that computing the difference in the parameter space does not require the query set \(\mathcal{D}_\text{query}\). For instance, in [16], a simulated FSL task \(\mathcal{T}\in\mathcal{D}_\text{base}\) is formulated as a tuple \((\mathcal{D}_\text{support},h^\star)\), in which the target classifier \(h^\star\) is obtained by training with all the data in \(\mathcal{D}_\text{base}\) (that are from the same classes in \(\mathcal{D}_\text{support}\)). Accordingly, 1 for meta-training \(\mathcal{A}\) can be reformulated as follows \[\begin{align} \min_\mathcal{A} \sum_{\mathcal{T}\sim\mathcal{D}_\text{base}} \; \ell_\text{model}(\mathcal{A}(\mathcal{D}_\text{support}), h^\star), \label{eq95meta95target} \end{align}\tag{4}\] where \(\ell_\text{model}\) is a distance measure (e.g., Euclidean distance) between classifiers in the parameter space.

One advantage of 4 over 1 , or a few-shot task \((\mathcal{D}_\text{support},h^\star)\) over a few-shot task \((\mathcal{D}_\text{support},\mathcal{D}_\text{query})\), is the effective amount of supervised signals associated with each support set \(\mathcal{D}_\text{support}\). Unlike \((\mathcal{D}_\text{support},\mathcal{D}_\text{query})\), which is limited by the query set size \(|\mathcal{D}_\text{query}|\), \(h^\star\) can be trained with all the data in \(\mathcal{D}_\text{base}\) (from the same classes in \(\mathcal{D}_\text{support}\)) and essentially provides more information for meta-training.

Denote by \(\boldsymbol{\phi}\) and \(\boldsymbol{\phi}^\star\) the parameters of \(h=\mathcal{A}(\mathcal{D}_\text{support})\) and \(h^\star\), [16], [20], [21] computed the squared Euclidean distance \(\ell_\text{model}(h, h^\star) = \|\boldsymbol{\phi}-\boldsymbol{\phi}^\star\|_2^2\) to drive meta-training. Doing so, however, has two drawbacks. First, the Euclidean distance may not be suitable in the high-dimensional parameter space. Indeed, [16], [21] have limited their approaches to only meta-train the last fully-connected layer of a neural network. Second, the Euclidean distance requires the two classifiers to have exactly the same architecture, meaning that we have to re-construct \(h^\star\) for different meta-learning algorithms and models.

3.3 LastShot: Learning with a strong teacher↩︎

Let us now introduce our approach. We define each simulated FSL task \(\mathcal{T} = (\mathcal{D}_\text{support}, \mathcal{D}_\text{query}, h^\star)\sim\mathcal{D}_\text{base}\) as a tuple that contains both the query set and the target classifier beside the support set. The target classifier \(h^\star\) is constructed similarly as described in 3.2.2, i.e., using all the data in \(\mathcal{D}_\text{base}\) that are from the same classes as in \(\mathcal{D}_\text{support}\). We will provide more details in 3.4. We propose a new meta-learning objective to meta-train the few-shot learner \(\mathcal{A}\) with this new definition of \(\mathcal{T}\),

\[\begin{align} \min_\mathcal{A} \sum_{\mathcal{T}\sim\mathcal{D}_\text{base}} \;\sum_{({\boldsymbol{x}},y)\in\mathcal{D}_\text{query}} & {\ell_\mathrm{\small LastShot}(\mathcal{A}(\mathcal{D}_\text{support})({\boldsymbol{x}}), h^\star({\boldsymbol{x}}))} \nonumber \\ & + \lambda\times\ell(\mathcal{A}(\mathcal{D}_\text{support})({\boldsymbol{x}}), y). \label{eq95Last} \end{align}\tag{5}\] This objective is as general as 1 but is able to benefit from the target classifiers \(h^\star\). Specifically, 5 involves two loss terms on the query data. One is exactly the same as in 1 , comparing \(\mathcal{A}(\mathcal{D}_\text{support})({\boldsymbol{x}})\) to \(y\). The other term compares \(\mathcal{A}(\mathcal{D}_\text{support})({\boldsymbol{x}})\) to \(h^\star({\boldsymbol{x}})\) via a loss function \(\ell_\mathrm{\small LastShot}\), which can be interpreted as a relaxed version of 4 : to match the predictions of the two classifiers, not their parameters. This relaxed version effectively overcomes the two drawbacks of 4 . On the one hand, we are now able to employ a stronger target classifier (e.g., the entire neural network model) to teach the few-shot learner, since the comparison between them is in the lower-dimensional space govern by the number of ways \(C\), not by the number of parameters of \(h^\star\). On the other hand, comparing the predictions allows us to disentangle the model architectures of the few-shot learner, its generated classifier, and \(h^\star\). In other words, the same \(h^\star\) associated with each \(\mathcal{D}_\text{support}\) can now be applied to train different kinds of few-shot learners, making 5 a widely applicable objective for meta-learning. Indeed, wherever 1 can be applied, 5 should be equally applicable.

We realize the loss \(\ell_\mathrm{\small LastShot}\) in 5 by the Kullback–Leibler (KL) divergence, inspired by [38]. That is, instead of comparing the hard one-hot predictions, we compare the soft predictions that encode the target classifier’s confidence as well as the learned semantic relationship among classes. More specifically, we take the logits \(\boldsymbol{\psi}({\boldsymbol{x}})\in\mathbb{R}^C\) and \(\boldsymbol{\psi}^\star({\boldsymbol{x}})\in\mathbb{R}^C\) produced by \(h=\mathcal{A}(\mathcal{D}_\text{support})\) and \(h^\star\) on a query example \({\boldsymbol{x}}\), smooth the latter with a temperature \(\tau > 0\) [100], and normalize them with the \(\mathbf{softmax}\) function before computing the KL divergence, \[\begin{align} & \ell_\mathrm{\small LastShot}(\mathcal{A}(\mathcal{D}_\text{support})({\boldsymbol{x}}), h^\star({\boldsymbol{x}})) = \nonumber \\ & \mathbf{KL}\left( \mathbf{softmax}(\frac{\boldsymbol{\psi}^\star({\boldsymbol{x}})}{\tau}) \;\big{\|}\; \mathbf{softmax}\left(\boldsymbol{\psi}({\boldsymbol{x}})\right)\right)\;. \label{eq95KL} \end{align}\tag{6}\] We note that the KL divergence is not the only way to realize \(\ell_\mathrm{\small LastShot}\). One can apply other measures that can characterize the difference of predictions between two classifiers.

3.4 Construction of the target classifier↩︎

So far, we assume that the target classifier \(h^\star\) is well-specified and can be easily prepared for each sampled FSL task \(\mathcal{T}\). In this subsection, we investigate several ways to construct it. We note that, a target classifier \(h^\star\) for each \(\mathcal{T}\) should be a \(C\)-way classifier; the \(C\) classes are the same as those contained in \(\mathcal{D}_\text{support}\) and \(\mathcal{D}_\text{query}\). That is, \(h^\star\) would be different across the simulated FSL tasks.

3.4.1 Masking the \(B\)-way base classifier↩︎

Given the \(B\)-class many-shot training set \(\mathcal{D}_\text{base}\), one intuitive way to construct \(h^\star\) is to directly train a \(B\)-class classifier. However, since most of the FSL settings have \(C\) (i.e., the number of ways in an FSL task \(\mathcal{T}\)) much smaller than \(B\), we must mask out the logits of \(h^\star\) not from the \(C\) classes in \(\mathcal{T}\), followed by calibration. In our experiments, we found such a mismatch in class numbers does degrade the performance of LastShot(please see 4.6). We thus develop two simple, efficient, yet effective alternatives to construct \(h^\star\).

3.4.2 Nearest centroid classifiers (NC)↩︎

Let us denote by \(f^\ddagger\) the feature extractor of the learned \(B\)-class classifier and by \(\mathcal{C}_\mathcal{T}\) the class label space of \(\mathcal{T}\). Our first idea is to build a \(C\)-way nearest centroid classifier [9], [18]. Denote by \(\mathcal{D}_\text{base, c}\) the subset of \(\mathcal{D}_\text{base}\) of class \(c\), we can pre-compute the class mean \(\boldsymbol{\mu}_c\) for each class \(c\) \[\begin{align} \boldsymbol{\mu}_c = \sum_{{\boldsymbol{x}}'\in\mathcal{D}_\text{base, c}} \frac{f^\ddagger({\boldsymbol{x}}')}{|\mathcal{D}_\text{base, c}|}. \end{align}\] Then, given a \(C\)-way few-shot task \(\mathcal{T}\), we can retrieve \(\boldsymbol{\mu}_c\) for class \(c\in\mathcal{C}_\mathcal{T}\) and compute for \({\boldsymbol{x}}\in\mathcal{D}_\text{query}\) the logit of class \(c\) \[\begin{align} \psi^{\star}_c({\boldsymbol{x}}) = -\left\|f^\ddagger({\boldsymbol{x}}) - \boldsymbol{\mu}_c\right\|_2^2. \end{align}\] That is, the logit vector \(\boldsymbol{\psi}^\star({\boldsymbol{x}})\in\mathbb{R}^C\) of the target classifier for \({\boldsymbol{x}}\) is the negative squared distance between \(f^\ddagger({\boldsymbol{x}})\) and the \(C\) class means, which is then used in 6 . We note that all the \(f^\ddagger({\boldsymbol{x}}), \forall {\boldsymbol{x}}\in\mathcal{D}_\text{base}\) can be pre-extracted.

3.4.3 Logistic regression (LR)↩︎

Our second way is to train a \(C\)-class logistic regression using the features pre-extracted by \(f^\ddagger\). Specifically, given a \(C\)-way few-shot task \(\mathcal{T}\), we gather the pre-extracted features \(f^\ddagger({\boldsymbol{x}})\) for data in the subset \(\cup_{c\in\mathcal{C}_\mathcal{T}} \mathcal{D}_\text{base, c}\) of \(\mathcal{D}_\text{base}\) and train a \(C\)-class logistic regression. The logit vector \(\boldsymbol{\psi}^\star({\boldsymbol{x}})\) of the target classifier for \({\boldsymbol{x}}\) is the output of the LR model. Training the linear LR is efficient given the extracted features.

3.5 Querying the target classifier in meta-training↩︎

Besides different ways of constructing \(h^\star\), we also investigate how to query \(h^\star\); i.e., how to obtain \(\boldsymbol{\psi}^\star({\boldsymbol{x}})\) for 6 . The basic way is to input the query example \({\boldsymbol{x}}\) directly. Motivated by recent studies [22], [101], where strengthening the teacher or weakening the student (e.g., adding noise) could further improve the distillation or self-training performance, we explore (a) varying the size of the input image for \({\boldsymbol{x}}\) or (b) applying autoaugment [102] to \({\boldsymbol{x}}\). We note that both operations are applied only in the meta-training phase.

3.5.1 Varying the input image size↩︎

One way to strengthen the teacher target classifier \(h^\star\), i.e., to increase its prediction accuracy, is to operate on images with a larger size. Specifically, when we apply \(f^\ddagger\) to extract features to construct \(h^\star\), we enlarge the input images (e.g., by \(140\%\)). We do so also when we query \(h^\star\); i.e., we enlarge \({\boldsymbol{x}}\) before applying \(h^\star({\boldsymbol{x}})\). Namely, the target classifier is constructed with and applied to images of a larger size, resulting in better logits \(\boldsymbol{\psi}^\star({\boldsymbol{x}})\) to be matched in 6 .

We note that, we enlarge the image size only for constructing and querying the teacher classifier during meta-training. We do not change the image size for the few-shot learner and its generated classifier. Therefore, our method does not raise any concern of unfair comparisons to existing methods.

3.5.2 Applying autoaugment [102]↩︎

We further consider weakening the student model (i.e., the few-shot learner and its generated classifier) to encourage it to learn more from the teacher. Specifically, we apply autoaugment [102] to each query example \({\boldsymbol{x}}\), so that both the teacher \(h^\star\) and the student \(h=\mathcal{A}(\mathcal{D}_\text{support})\) make their predictions on a noisy query example. Since the teacher is trained on a larger data set \(\mathcal{D}_\text{base}\), it suffers less than the student from the noise, creating a larger training signal in 6 .

We find that both treatments can improve LastShot on some of the datasets, and strengthening the teacher is more stable. Thus, we mainly consider it in the experiments.

3.6 Evaluation of the learned few-shot learner \(\mathcal{A}\)↩︎

So far in 3, we focus on how to meta-train the few-shot learner \(\mathcal{A}\) that can output a classifier \(h=\mathcal{A}(\mathcal{D}_\text{support})\) given the support set \(\mathcal{D}_\text{support}\) of a few-shot task \(\mathcal{T}\). The few-shot task \(\mathcal{T}\) during meta-training is sampled/simulated from \({D}_\text{base}\), and one can flexibly define it as \(\mathcal{T}=(\mathcal{D}_\text{support}, \mathcal{D}_\text{query})\), \(\mathcal{T}=(\mathcal{D}_\text{support}, h^\star)\), or \(\mathcal{T}=(\mathcal{D}_\text{support}, \mathcal{D}_\text{query}, h^\star)\) (cf. 3.2, 3.2.2, 3.3, respectively).

During meta-testing, in which the trained few-shot learner \(\mathcal{A}\) is to be evaluated on novel classes (cf. 3.1), we follow the standard evaluation protocol (cf. 4.2): each meta-testing task \(\mathcal{T}'\) is a tuple \((\mathcal{D}_\text{support}', \mathcal{D}_\text{query}')\). The accuracy on a task is defined as \[\begin{align} \frac{1}{|\mathcal{D}_\text{query}'|}\sum_{({\boldsymbol{x}},y)\in\mathcal{D}_\text{query}'} \boldsymbol{1}[\mathcal{A}(\mathcal{D}_\text{support}')({\boldsymbol{x}}) == y], \end{align}\] where \(\boldsymbol{1}\) is an indicator function whose value is 1 if the input is true; otherwise, 0. In other word, during meta-testing, we do not need a target classifier for each meta-testing task. We construct the target classifiers only to facilitate meta-training, and this is possible because the base class data \({D}_\text{base}\) contains sufficient examples.

4 Experiments on Few-Shot Classification↩︎

4.1 Datasets↩︎

We evaluate our approach LastShot using multiple benchmark datasets, including miniImageNet [10], tieredImageNet [26], CUB [29], CIFAR-FS [27], and FC-100 [28]. There are 100 classes in miniImageNet with 600 images per class. Following the split of [15], the meta-training/validation/testing sets contain 64/16/20 (non-overlapping) classes, respectively. In tieredImageNet [26], the class numbers in the three sets are 351/97/160, respectively, again without overlapping. For CUB, we follow the split of [23] and set 100 classes for meta-training. Two 50-class splits from the remaining classes are used for meta-validation and meta-testing. CIFAR-FS [27] and FC-100 [28] are two variants of the CIFAR-100 dataset [103] for few-shot learning but with different partitions. The classes in CIFAR-FS are randomly split into 64/16/20 for meta-training/validation/testing, while FC-100 takes the split of 60/20/20 classes following the super-classes. There are 600 images in each class in CIFAR-100. All images are resized to 84 by 84 for the first three datasets and are resized to 32 by 32 for the latter two CIFAR variants in advance. The detailed statistics are listed in 1.

4.2 Evaluation protocols↩︎

We follow the evaluation in the literature [19], [23], [35], [64], where 10,000 \(C\)-way \(K\)-shot tasks are sampled from the meta-testing set. In each task, the query set contains 15 instances in each class. Mean accuracy (in %) as well as the 95% confidence interval are reported. For every meta-training task, we also sample 15 instances per class to construct the query set, unless stated otherwise.

4.3 Implementation details↩︎

LastShot is a plug-and-play approach that is compatible with most of the existing meta-learning FSL methods. We apply LastShot to four representative meta-learning methods: ProtoNet [13], FEAT [23], ProtoMAML [24], and MetaOptNet [25]. The first two are embedding-based methods to learn good feature extractors; the latter two are similar to MAML that learn in a two-stage inner/outer-loop manner.

4.3.1 Model architectures for few-shot classification↩︎

We follow [25] to use a ResNet-12 [17] for the feature extractor \(f_{\boldsymbol{\theta}}\) (cf. 3.2.1), which contains a wider width and the Dropblock module [104] to avoid over-fitting.

4.3.2 Initialization of the feature extractor↩︎

Following [9], [23], we initialize the feature extractor \(f_{\boldsymbol{\theta}}\) used in our few-shot learner and our target classifier using the base class data \(\mathcal{D}_\text{base}\). Concretely, we learn \(f_{\boldsymbol{\theta}}\), together with a \(B\)-way linear classifier \(W\) on top⁶, to classify the base classes. For instance, on miniImageNet, \(B\) is \(64\), meaning that we pre-train a \(64\)-class classifier. More specifically, we train the classifier by minimizing the following objective over examples \(({\boldsymbol{x}}_i, y_i)\) in \(\mathcal{D}_\text{base}\): \[\begin{align} \min_{W, f_{\boldsymbol{\theta}}}\; \sum_{({\boldsymbol{x}}_i, y_i)\in{\mathcal{D}_\text{base}}} \; \ell(W^\top f_{\boldsymbol{\theta}}({\boldsymbol{x}}_i), \;y_i).\label{eq:cls} \end{align}\tag{7}\] Here, \(\ell(\cdot, \cdot)\) measures the discrepancy between the predicted label and the ground-truth label, and we use the cross-entropy loss. We follow [23] for optimization. For instance, on miniImageNet, 7 is optimized by stochastic gradient descent (SGD) with momentum 0.9, weight decay 0.0005, batch-size 128, and initial learning rate 0.1. After each epoch, we validate the quality of \(f_{\boldsymbol{\theta}}\) on the meta-validation set by sampling 200 1-shot \(C\)-way tasks where \(C\) equals the number of classes in the meta-validation set (e.g., 16 for miniImageNet). For each task, we use the ProtoNet decision rule in 3 to evaluate the classification performance. Similar procedures are employed for other datasets.

With the pre-trained \(f_{\boldsymbol{\theta}}\), we can initialize the few-shot learner in the meta-training phase. For fair comparisons, we use the notation \(\dagger\) to denote our re-implemented results of the baseline methods using this pre-training procedure, which usually leads to higher accuracy than the reported results. We note that, FEAT [23] already uses the above procedure, so we compare with the published results directly.

4.3.3 Meta-Learning with LastShot↩︎

In our implementation, we do not use the meta-validation set to learn the few-shot learner; i.e., the meta-validation set is only used to select the best hyper-parameters such as the balance coefficient \(\lambda\) in 5 or the temperature \(\tau\). For LastShot with LR, we randomly select at most 50 instances per class to construct a linear logistic regression for each few-shot task. To strengthen the teacher, we enlarge images during training and querying the target classifier \(h^\star\). For miniImageNet, tieredImageNet, and CUB, we enlarge the images to 116 by 116. For CIFAR-FS and FC-100, we enlarge the images to 50 by 50. We note that, we do not enlarge the images for the few-shot learner and its outputted classifier. Thus, our results can be fairly compared to the literature.

4.3.4 Non-meta-learning based FSL methods↩︎

The initialized \(f_{\boldsymbol{\theta}}\) (cf. 7 ) can be used for few-shot classification directly. For instance, by extracting features on the support set data of the novel classes using \(f_{\boldsymbol{\theta}}\), we can apply 3 to classify a query example according to the nearest centroid classification rule. We call this method PT-EMB, which stands for pre-trained-embedding. We also compare to SimpleShot[9], which employs several simple pre-processing steps to further improve the accuracy.

4.4 Benchmark comparisons↩︎

We first evaluate LastShot with 5-Way 1-Shot and 5-Way 5-Shot tasks on miniImageNet (2), tieredImageNet (3), CUB (4), CIFAR-FS (5), and FC-100 (5). We highlight the results when a baseline method is combined with LastShot via a cyan background in the tables, and make the best performance in each setting in bold. We denote the two implementations of the target classifiers, i.e., the nearest centroid classifier and the logistic regression, by NC and LR, respectively.

We find that LastShot can consistently improve each baseline method on all the datasets by 1\(\sim\)​3%. For example, with LastShot, the 1-Shot classification accuracy of FEAT improves from 77.97% to 80.20% in 4. By further investigating the four baseline methods, we find that MetaOptNet and FEAT are stronger than ProtoNet and ProtoMAML. In general, LastShot can lead to a larger boost for the relatively weaker baselines, but it can still improve MetaOptNet and FEAT. The target classifiers based on the nearest centroid and LR have similar performance. Overall, LastShot variants achieve the state-of-the-art performance on miniImageNet and CUB, while obtaining competitive results on tieredImageNet. These verify that a strong teacher can provide better supervision in meta-learning.

Last but not the least, comparing to Model Regression [16] and RFS [34], in which the former only uses the target classifier to guide the few-shot learner in the last fully-connected layer while the latter uses distillation to strengthen the pre-trained features, our LastShot outperforms them on miniImageNet, CUB, and CIFAR-FS, and performs on par with them on other datasets. The promising performance further demonstrates the effectiveness of our approach.

4.5 Larger-shot comparisons↩︎

Besides the benchmark results, we further investigate the scenarios when there are more shots in the support set for the novel classes (i.e., during meta-testing). For instance, \(K=50\) is much larger than the usual few-shot learning scenario (where \(K\)=1 or \(K\)=5), but is still smaller than usual many-shot datasets like ImageNet [110] for training a complex deep neural network. We consider \(K=\{1, 5, 10, 20, 30, 50\}\). For meta-training on the base class data, we also consider \(K=\{1, 5, 10, 20, 30, 50\}\). However, instead of setting the same \(K\) for meta-training and meta-testing, we pick the best \(K\) in meta-training for each \(K\) of meta-testing, according to the accuracy on the meta-validation set. We note that, for a larger \(K\) in meta-training, the query set size may be shrunk due to the memory constraint. For example, for \(K=50\), the number of query examples per class becomes \(2\). For meta-testing, since no gradients need to be computed for the few-shot learner \(\mathcal{A}\), we can still keep the query set size intact, i.e., \(15\) examples per class.

The results are shown in 6 and 7: LastShot can consistently improve the corresponding meta-learning method at all the different shot settings. From the further analysis in 1, we see that the strength of conventional meta-learning decreases when \(K\) increases. For instance, the improvement by the meta-learned method ProtoNet over the plainly-trained method “PT-EMB” diminishes when the number of shots becomes large. PT-EMB even outperforms ProtoNet when \(K>5\). (Note that, both methods use exactly the same prediction rule.) In 3, we see a similar trend when comparing the best-performing meta-learning method FEAT and the best-performing non-meta-learning method SimpleShot. SimpleShot outperforms FEAT when \(K = 50\).

While one may argue that this is not surprising given that meta-learning is designed for small \(K\), the fact that there seems to exist a turning point that one must switch from meta-learning methods to non-meta-learning ones is notoriously annoying in practice, especially that such turning point is quite unstable (it is around 5 for ProtoNet but 50 for FEAT). It is desired to design learning mechanisms that can seamlessly combine the advantages of these two methods.

We claim that LastShot is one of such mechanisms. On the one hand, LastShot is itself a meta-learning method, drawing few-shot tasks to train the few-shot learner. On the other hand, LastShot explicitly uses the non-meta-learning based methods with a larger \(K\) to construct the target classifier to teach the few-shot learner in the meta-training process. The results on miniImageNet and tieredImageNet (in 6 and 7, respectively) further demonstrate this. Our LastShot combined with FEAT consistently improves FEAT and outperforms SimpleShot at every \(K\) values, making LastShot the best-performing methods to be applied across a wide spectrum of \(K\) values.

4.6 Ablation studies↩︎

4.6.1 Advantage of LastShot with a small query set↩︎

We evaluate the influence of the query set size during meta-training. With a small query set, the few-shot learner only receives limited supervision. We investigate ProtoNet with LastShot. As shown in 8, the classification accuracy of ProtoNet improves when more query instances are involved, especially from 1 to 5. With LastShot, the few-shot learner performs much favorably. Even with 1 query instance per class, its performance can already outperform ProtoNet trained with 50 query instances but without LastShot.

4.6.2 Comparison on the target classifiers↩︎

One intuitive way to construct the target classifier \(h^\star\) is to train a \(B\)-class classifier and then select the corresponding \(C\) logits according to the few-shot task at hand. As shown in 9, using part of the \(B\)-class base classifier as the teacher could not introduce further improvement, which may result from the mismatch in class numbers and the poorly calibrated logit values.

4.6.3 Strengthening the teacher or weakening the student?↩︎

There are three ways to query \(h^\star\), i.e., directly querying it, weakening the student (e.g., adding noise on the query set with autoaugment (Autoaug) [102]), or strengthening the teacher with enlarged inputs. We show the influence of the choices in 10. The vanilla case means that we query \(h^\star\) directly. By adding noises to the query set, we enlarge the learning signal between the teacher and the student; by enlarging the input images for the teacher, the teacher can provide higher-quality supervision. Both ways can improve the vanilla method. We empirically find that strengthening the teacher leads to the most stable improvement.

To verify if adding noise itself can improve meta-learning alone, we also apply Autoaug to FEAT. As shown in 10, Autoaug, without LastShot, can degrade the performance of FEAT significantly. Similarly, enlarging the input images without LastShot cannot improve FEAT.

4.6.4 Influence of the coefficient \(\lambda\)↩︎

We investigate the influence of \(\lambda\), the coefficient for the vanilla meta-learning objective in 5 , when we apply LastShot(NC) with FEAT. The larger the value of \(\lambda\), the smaller the influence of the target classifier’s supervision during meta-training. We show the {1,5,10,20,30,50}-Shot 5-Way classification accuracy on miniImageNet in 11. We find that when there are more shots in a task, meta-learning with more supervision from the target classifier (with smaller \(\lambda\)) works better.

4.6.5 Will a direct ensemble help?↩︎

Since we construct a stronger teacher (not based on meta-learning) to supervise the few-shot learner, one may ask whether the stronger teacher itself can outperform the final few-shot learner trained with LastShot. We note that, the strong teacher cannot be applied to the novel meta-testing tasks directly since we do not have ample examples for them to construct the strong teacher. We therefore investigate an alternative question: can we achieve better few-shot accuracy by combining a non-meta-learning based and a meta-learning based FSL methods directly? We investigate this idea by taking the ensemble of their predictions. In detail, for a meta-testing task, we apply both SimpleShot [9] and ProtoNet [13] for classification, and we further average their predictions per example. We also apply the ensemble method to FEAT [23]. The results are reported in 12. The ensemble approach gets marginal improvement compared to the vanilla meta-learning methods, while our LastShot shows consistent improvements over the meta-learning methods with a single final model in various cases.

4.6.6 Will LastShot help graph-based FSL methods?↩︎

We apply LastShot to the GCN-based few-shot learning algorithm. We use [111] as the basic GCN-based algorithm (denoted as “GCN”), following [112]. With ResNet-12 as the feature extractor, the 1/5-Shot 5-Way classification accuracy (plus 95% confidence interval) with 10,000 trials on miniImageNet is reported in 13. Our LastShot can further improve the GCN-based algorithm, arriving at 65.17% w/ LastShot(NC) and 65.25% w/ LastShot(LR), respectively. The results indicate that LastShot is general for meta-learning and can improve GCN-based algorithms.

4.6.7 Will LastShot be compatible with other backbones?↩︎

As shown in several recent works like [9], [23], [113], the backbone architectures of the feature extractor can largely affect the resulting classification or FSL accuracy. In this subsection, we investigate whether the superiority of LastShot extends to more backbones. We consider two widely used backbones, i.e., a four-layer convolution network and Wide ResNet-28-10 [114]. The four-layer ConvNet contains four repeated blocks, and each block includes a convolutional layer, a Batch Normalization layer [115], a ReLU, and a Max pooling [10], [13]. We add a global average pooling layer at last to reduce the dimension of the embedding to 64. We follow [23], [64] and use the WRN-28-10 structure, which sets the depth to 28 and width to 10. We study two meta-learning based algorithms, ProtoNet and FEAT. The 1/5-Shot 5-Way classification accuracy (plus 95% confidence interval) with 10,000 trials on miniImageNet is reported in 14. LastShot improves ProtoNet and FEAT on both backbones.

5 Experiments on Few-Shot Regression↩︎

Besides the experiments on few-shot classification in 4, we further follow [7] to study a one-dimensional regression problem to illustrate the effectiveness and applicability of our approach LastShot.

5.1 Datasets↩︎

We generate synthetic, one-dimensional few-shot regression tasks with inputs \(x\) sampled uniformly from \([-5,5]\). Every few-shot task is characterized by a true sine function parameterized by \((a,v,b)\): \[\begin{align} y = a\times\sin(v x+b) + \epsilon. \label{e95regreg} \end{align}\tag{8}\] In this sine curve, the amplitude \(a\) is sampled uniformly from \([0, 2]\), the frequency \(v\) is sampled uniformly from \([2,4]\), the bias \(b\) is sampled uniformly from \([0, 2\pi]\), and \(\epsilon\) is sampled from \(0.3\times\mathcal{N}(0,1)\), where \(\epsilon\) is an additional noise. Given \((a,v,b)\), we can then sample the support and query sets by first sampling \(x\sim [-5, 5]\) and then obtaining the corresponding \(y\) using 8 . We note that, the sine curves defined in 8 have higher frequencies than those studied in [7]. Thus, it requires more support set examples to learn a regression function.

5.2 Evaluation and training protocols↩︎

The goal is to build a regression function using the support set, such that it can predict the real-valued labels well for the corresponding query set. We randomly sample \(1,000\) meta-testing tasks (i.e., \(1,000\) sin curves) for evaluation. From each, we randomly sample \(K\) pairs of \((x, y)\) according to 8 to form the few-shot support set \(\mathcal{D}'_\text{support}\), and another non-overlapping 100 pairs as the query set \(\mathcal{D}'_\text{query}\). (In our experiments, we consider \(K\in\{5, 50\}\).) The performance is measured by the average Mean Square Error (MSE) and 95% confidence intervals over the \(1,000\) tasks.

We investigate training a meta-learning based few-shot learner \(\mathcal{A}\) to output the regressor. Specifically, we model the regression function by a three-layer fully-connected neural network \(f_{\boldsymbol{\theta}}(x)\) as the feature extractor. The hidden and output dimensions are all set to \(100\). We can learn such a feature extractor using nearly all the existing meta-learning methods. For example, with MAML [7], we will add a linear regressor \(w\) on top. When embedding-based methods like ProtoNet [13] are used, it is equivalent to learning a non-parametric regression \[\begin{align} \hat{y} = \sum_{(x', y')\in\mathcal{D}_\text{support}}\mathbf{Softmax}\left(-\|f_{\boldsymbol{\theta}}(x) - f_{\boldsymbol{\theta}}(x')\|_2^2\right)\times y', \label{e95reg95ProtoNet} \end{align}\tag{9}\] where \(\mathbf{Softmax}(\cdot)\) is taken over \(K\) examples in \(\mathcal{D}_\text{support}\).

We train \(\mathcal{A}\) with at most \(1,280,000\) few-shot tasks, each is associated with a specific \((a,v,b)\), a support set \(\mathcal{D}_\text{support}\), and a query set \(\mathcal{D}_\text{query}\). We pack every \(32\) tasks into a mini-batch, and train \(\mathcal{A}\) for \(40,000\) iterations. We apply stochastic gradient descent (SGD) with a momentum of \(0.9\) as the optimizer. We set the initial learning rate as \(0.001\), and multiply it by \(0.5\) every \(160,000\) tasks. We perform early stopping using another \(1,000\) validation tasks.

5.3 LastShot for regression↩︎

To apply LastShot for few-shot regression, we must (a) construct the target regressor \(h^\star\) for each few-shot regression task, and (b) define the suitable \(\ell_\mathrm{\small LastShot}\) for supervision.

For the former, to efficiently construct \(h^\star\) without sampling extra data from each few-shot task, we discretize the ranges of \(a\in[0, 2]\), \(v\in[2,4]\), and \(b\in[0, 2\pi]\) each by \(0.1\) to construct a set of “anchor” tasks. For each of them, we can sample many-shot examples (e.g., \(1,000\) in our experiments) to construct a many-shot regressor. Here we train a neural network regressor using the regularized least square loss (i.e., the loss used in ridge regression) for each anchor task, whose hyper-parameter is tuned by cross-validation. Then during meta-training, given a few-shot task and its associated \((a, v, b)\), we retrieve the anchor task at the same grid to be the target model \(h^\star\) — the rationale here is that sine curves with similar \((a, v, b)\) are shaped similarly.

With the target regressor \(h^\star\), we realize \(\ell_\mathrm{\small LastShot}\) (cf. 5 ) by a weighted square loss [116], \[\begin{align} \sum_{(x,y) \in\mathcal{D}_\text{query}} & \ell_\mathrm{\small LastShot}(\mathcal{A}(\mathcal{D}_\text{support})(x), h^\star(x)) = \nonumber \\ \sum_{(x,y)\in\mathcal{D}_\text{query}} & \mathbf{Softmax}\left(-(h^\star(x) - y)^2\right) \\ & \times\left(h^\star(x) - \mathcal{A}(\mathcal{D}_\text{support})(x)\right)^2\;, \nonumber \end{align}\] in which \(\mathbf{Softmax}\) is taken over the examples in \(\mathcal{D}_\text{query}\). The \(\mathbf{Softmax}\) measures how well the target regressor \(h^\star\) performs on a query example — we measure this because the retrieved anchor regressor does not exactly match the few-shot task at hand. The better \(h^\star\) performs (i.e., a smaller error between \(h^\star(x)\) and \(y\)), the larger the score is after \(\mathbf{Softmax}\). We then use these \(\mathbf{Softmax}\) scores to perform weight average over the error between the few-shot regressor \(A(\mathcal{D}_\text{support})(x)\) and the target regressor \(h^\star(x)\) for each query example \(x\). Namely, the few-shot learner pays more attention to those query examples that \(h^\star(x)\) truly performs well on.

5.4 Results↩︎

We compare LastShot with ProtoNet [13], MAML [7], MetaOptNet [25], and FEAT [23] in ¿tbl:tab:regression?. Specifically for MetaOptNet, we implement its top-layer classifier as ridge regression and tune the hyper-parameter. We make the following observations. First, predicting a sine curve is difficult given few-shot examples⁷, and all methods perform better when there are more shots. Second, FEAT and MetaOptNet, being more advanced meta-learning methods, outperform MAML and ProtoNet. Third and most importantly, LastShot can notably improve each of the baseline methods.

We further visualize the few-shot learners’ performance on three few-shot tasks whose true, oracle curve are \(y=\sin(2x)\), \(y=\sin(3x+\pi)\), and \(y=2\sin(2x+\pi)\) in 4. Specifically, we consider both \(5\) and \(50\) shots cases, and plot the oracle curve (cyan), predicted curve without LastShot(blue), and predicted curve with LastShot(red). We see that the predicted curves with LastShot can better match the oracle ones. It is worth noting that for each of these three curves, there is more than one period in the range. Therefore, it is difficult to fit the curve when there are merely five examples: there can be multiple feasible solutions. We surmise that this is why ProtoNet creates a nearly horizontal curve. By learning with a stronger teacher via LastShot, ProtoNet can generate more faithful curves in both 5-shot and 50-shot scenarios.

\begin{table}[tbp]

MSE	5-Shot	50-Shot
MAML [7]	0.693\(\pm\)0.034	0.303\(\pm\)0.016
ProtoNet [13]	0.681\(\pm\)0.031	0.299\(\pm\)0.013
+	0.630\(\pm\)0.030	0.294\(\pm\)0.011
MetaOptNet [25]	0.622\(\pm\)0.029	0.162\(\pm\)0.004
+	0.616\(\pm\)0.022	0.159\(\pm\)0.004
FEAT [23]	0.491\(\pm\)0.026	0.189\(\pm\)0.006
+	0.479\(\pm\)0.023	0.167\(\pm\)0.004

\vskip\caption{Mean Square Error (MSE) and 95\% confidence interval over 1000 $\{5, 50\}$-shot regression tasks.  The lower the better.}\label{tab:regression}-5pt

\end{table}

6 Conclusion↩︎

We present a novel meta-learning paradigm to introduce rich supervision for few-shot learning. By coupling a few-shot task in the meta-training phase with a target classifier that is trained with ample examples, the few-shot learner can receive additional training signals from the target classifier besides the query set instances. We propose ways to efficiently construct and leverage the target classifier, leading to a plug-and-play method LastShot, which is compatible with various meta-learning based few-shot learning methods. On multiple datasets, LastShot achieves consistent improvement against baseline meta-learning methods. Importantly, LastShot also maintains robust performance across a wider range of shot numbers — making meta-learning not only work well on the few-shot but also many-shot settings.

Acknowledgments↩︎

Acknowledgment↩︎

This research is supported by National Key R&D Program of China (2020AAA0109401), Nanjing University-Huawei Joint Research Program, NSFC (62006112, 61921006, 62176117), Collaborative Innovation Center of Novel Software Technology and Industrialization, NSF of Jiangsu Province (BK20200313), NSF IIS-2107077, NSF OAC-2118240, NSF OAC-2112606, and the OSU GI Development funds. We are thankful for the generous support of computational resources by Ohio Supercomputer Center and AWS Cloud Credits for Research.

References↩︎

---

1. The term “shot” refers to the number of training examples per class.↩︎

2. Conventionally, we sample one or several few-shot tasks (each includes a support set and a query set) whose size can fit into the mini-batch size (constrained by the memory). Thus, when the support set gets larger, either due to the increasing number of classes or shots, the query set size gets shrunk. In extreme cases, each class may just have one query example.↩︎

3. While one may overcome this by processing a few-shot task using multiple mini-batches, aggregating the gradients, and updating the model at once, this would increase the meta-training time and may involve additional hyper-parameter tuning.↩︎

4. Here, a target classifier means the classifier that we hope a few-shot learner can generate given few-shot examples. One choice according to [16], [19]–[21] is the classifier that is trained on ample labeled data.↩︎

5. Nowadays, nearly all meta-learning based FSL methods are initialized with pre-trained features learned in a conventional way with the base class data [19], [23], [34], [35], and our approach is built upon them.↩︎

6. We omit the bias term for brevity.↩︎

7. One-dimensional sine functions have an infinite VC dimension.↩︎