Abstract–Autonomous systems (AS) often rely on Deep Neural Network (DNN) classifiers to operate in complex and dynamically changing environments. However, during operation, these classifiers may encounter domains that differ from those seen during development, causing performance degradation under distribution shift. Removing systems from operation for labelled data collection and retraining is often impractical, particularly when adaptation must occur quickly and at scale. This paper introduces DIRA-SS, a self-supervised extension of Dynamic Incremental Regularised Adaptation (DIRA) that enables online domain adaptation using only a small number of unlabelled target-domain samples. DIRA-SS augments an existing classifier with an auxiliary retraining branch and adapts the shared feature representation through a rotation-prediction task, while elastic weight consolidation regularises important source-domain parameters to reduce destructive updates. This allows the model to benefit from transfer learning without requiring classification labels during operation. We evaluate DIRA-SS on CIFAR-10C, CIFAR-100C, and ImageNet-C using ResNet architectures under severe common corruptions. The results show that DIRA-SS substantially improves performance over the non-adapted source model, achieves accuracy close to the supervised DIRA method, and outperforms existing unsupervised test-time adaptation baselines on ImageNet-C when using only 100 target-domain samples.

1 Introduction↩︎

Deep neural network (DNN) classifiers are frequently used in the development of autonomous systems (AS) to enable them to interact and adapt to dynamically changing real-world settings to accomplish their intended goals. Using DNNs in autonomous systems has the advantage of producing extremely non-linear decision boundaries to handle the complexity of operating settings since they can learn intricate patterns of complex surroundings. However, verifying the behaviour of DNNs to allow for their deployment is challenging. Self-driving vehicles are a common example of these ASs. According to recent research, an impractical amount of testing is necessary for every software update of a self-driving vehicle to validate the system before it can be put into use [1]. Alternatively, Koopman et.al. [2], proposed that after an AS passes a minimum safety validation case, the system is put into use and enhanced during operation to improve reliability.

In retraining of DNN classifiers, the optimiser modifies the decision boundary based on the samples used in training. During operation, there are usually only a few samples available for retraining. Retraining using few samples can result in a phenomenon known as catastrophic forgetting, where the model fails to generalise to the domain distribution [3]. Several methods have been developed, focusing on during operation adaptation of classifiers using few samples both in a supervised and unsupervised (or self-supervised) manner, e.g. [4]–[7]. State-of-the-art (SOTA) performance of adaptation during operation using few samples has been demonstrated by DIRA (Dynamic Incremental Regularised Adaption) [4]. However, DIRA relies on labelled retaining samples.

In the theme of accelerating machine learning, waiting for human operators to provide labels hinders the adoption of such retraining approaches for systems that need to adapt promptly. Additionally, having a human in the loop to provide labels on demand, creates a cost bottleneck for the utilisation of adaptable machine learning techniques on a large scale. In this paper, we discuss the potential of making DIRA self-supervised, to create DIRA-SS. An experimental evaluation of the success, effectiveness, and limits of DIRA-SS is discussed in this section. The rest of this paper is organised as follows, in Section 2 we discuss related work. Section 3 introduces the method for DIRA-SS. Experimentation setup is explained in section 4. Results and discussion handled by section 5. Further ablation studies are discussed in section 6. Lastly, section 7 concludes and discusses future works.

2 Related work↩︎

2.1 Model generalization↩︎

When training machine learning models, the assumption of independent and identically distributed (i.i.d.) data implies that the training data (source) is drawn from the same distribution as the test data (target) [[8]][9][[10]][11]. The term ‘generalisation’ is used to refer to a model performing well in a distribution where the i.i.d of data assumption holds. To formalise this, given a training dataset \(D_{train}\) drawn from source distribution \(P_{source}\), a model \(M\) trained on \(D_{train}\) generalise to any dataset \(D_{test}\) drawn from distribution \(P_{source}\). For many autonomous systems, their operational environments include interacting with an open world, e.g. self-driving vehicles, where the i.i.d assumption is often violated due to the source and target domains exhibiting variations in their data distributions. In other words, the dataset \(D_{test}\) would be drawn from a distribution \(P_{target}\). Hence, the concept of generalisation is difficult to precisely define when the system is expected to encounter environments out of the source data distribution. Maintaining the i.i.d assumption by rigorously defining the operational environment and having a full understanding of the limitations makes autonomous systems lose their purpose. The adoption of autonomous systems in the current world is mainly driven by their visioned ability to deal with various scenarios in environments where the possibilities are vast and are impractical to encapsulate during the development of the system.

Instead, generalisation can be extended to describe a system’s ability to adapt to its surrounding operational environment distribution. This is a much more scalable approach, goes beyond the assumption of i.i.d data, and opens the venue for systems to operate and generalise in partially defined (or even undefined) environments. The work introduced in this paper contributes to this latter approach of generalisation whereby machine learning models can adapt to their operational distributions, allowing autonomous systems to fulfil their intended purposes.

2.2 Continual learning↩︎

For a formal definition of continual learning and incremental learning please refer to Ghobrial et al.[4]. The concept of continual learning is usually concerned with learning a series of sequential tasks or domains whilst minimising forgetting of previously learnt ones, e.g. [[12]][13][14]. This is done to allow for a model to generalise to a wider set of distributions, through retaining knowledge. In the case of DIRA-SS, we are approaching the concept of generalisation by equipping models with the ability to quickly adapt and generalise to the current operational environment regardless of the model’s performance in other operational environments it operated in previously or will encounter in the future. The model will adapt to its current environment dynamically and continuously every time it changes operational domains. The retaining of knowledge from previous domains is used by DIRA-SS to utilise transfer learning in order to allow for efficient adaptation using very few samples. However, the goal of the model is to maximise its performance on the current domain and drop of performance on previous domains is ignored.

2.3 One/Few-shot learning↩︎

One-shot or few-shot learning addresses the challenge of training models with limited data to learn a new task. Unlike traditional machine learning approaches that require training on large datasets to achieve some level of effective performance, one/few-shot learning enables a model to gain good levels of performance from training on a data containing a single or a few instances. This is achieved provided a general representation has been learned by the model previously using a diverse set of samples, e.g.[[15]][16][[17]][18][19]. The concept of one/few-shot learning is useful in situations where obtaining extensive data is costly or impractical. By utilising methods such as transfer learning one/few-shot learning can equip models with the ability to rapidly adapt to its operational environment to make more accurate predictions. The method introduced in this paper, DIRA-SS contributes to the field of one/few-shot self-supervised learning, as it aims to achieve domain adaptation dynamically using few examples that don’t have classification labels provided with them.

2.4 Domain Adaptation Frameworks↩︎

Several approaches have been presented in the literature that address the problem of domain incremental adaptation, e.g. [5]–[7], [20], [20], [21], [21], [22]. Refer to Mirza et al. [5] for a brief breakdown of categories of different approaches. Here we will cover the two SOTA methods relevant to our approach.

The first method is DIRA [4], which aims at achieving adaptation through the regularisation of old information, whilst retraining on few available samples from the target domain. The model is retrained on labelled samples from the target domain. DIRA uses elastic weight consolidation [14] as a regularisation technique to avoid catastrophic forgetting and achieves adaptation using very few samples. This allows the model to benefit from transfer learning of information from the initial domain to the target domain.

The second method is TTT (Test-time training) [[7]][6]. The method combines different self-supervised auxiliary contexts to achieve domain adaptation. They break down neural network parameters into three parts, such that pictorially the architecture has a Y-structure. The bottom section of the Y-structured architecture represents the input layer and the layers responsible for the shared feature extraction, whilst the other two sections contain layers for learning and outputting labels for the main and auxiliary tasks independently. An example of this auxiliary task is being able to tell the rotation of the input image at 90-degree increments i.e., 90, 180, 270 degrees. During training time the whole neural network is optimised using a combined loss function that aims to maximise performance on both the main and auxiliary tasks. During retraining to adapt to a new domain, only parameters of the shared feature extraction and the auxiliary task sections are allowed to change. By doing so, the shared feature extraction section of the network modifies to learn the new domain, so that the network may output correct predictions on the unchanged branch of the network responsible for the main task. The advantage of TTT is that it is self-supervised because the retraining samples for the auxiliary task can be created by giving a 90-degree incremented rotation to the new domain unlabelled data and the appropriate rotation label is attached. The assumption here is that the new domain input data is always at 0-degree rotation. However, the drawback of TTT is that it requires a large number of samples to achieve this self-supervised adaptation.

This paper assesses the feasibility of converting DIRA to a self-supervised technique by using a similar concept of retraining on auxiliary tasks, as is used in the TTT approach. However, due to the regularisation available in DIRA, only a few samples are expected to be required for adaptation. Hence, the new method, DIRA-SS, would benefit from the strengths of both DIRA and TTT. Furthermore, the optimisation problem used by DIRA-SS in the initial training stage is different from that used by TTT. TTT tries to optimise on both the main and the auxiliary task at initial training, then in retraining/adaptation, it only optimises on the auxiliary task. However, DIRA-SS only optimises on the main task during initial training and only optimises on the auxiliary task during retraining/adaptation.

3 Method↩︎

This section describes the algorithmic details for DIRA-SS. Let \(M_0\) be the model trained on the original domain dataset \(X_0\). The aim of domain adaption is to adapt the trained model parameters, \(\theta\), to out-of-distribution target data \(X_T\). The goal of DIRA-SS is to achieve this using a few samples from the target domain and without ground truth labels being provided. During standard training the optimisation problem tries to solve the loss function \(\mathcal{L}\) given by equation 1 . This usually requires a large number of training samples. DIRA [4] modified this optimisation problem and included a regularisation term based on elastic weight consolidation [14]. This is to allow the model being retrained to benefit from transfer learning and thus reduce the number of samples required and improve the accuracy achieved through adaptation.

\[\displaystyle{\mathop{\mathrm{min\text{ }}}_{\theta}} \mathcal{L}(\theta) \label{eq:DIRA95SS95optim95base}\tag{1}\]

\[\displaystyle{\mathop{\mathrm{min\text{ }}}_{\theta}} \mathcal{L}_T(\theta) + \sum_{j}^{}\lambda F_{0,j}(\theta_j - \theta^{*}_{0,j})^2 \label{eq:DIRA95SS-DIRA}\tag{2}\]

The optimisation problem DIRA tries to solve is given by equation 2 , where the loss function for the target domain is represented by the \(\mathcal{L}_T\) and \(\sum_{j}^{}\lambda F_{0,j}(\theta_j - \theta^{*}_{0,j})^2\) is an approximation of the original domain implemented based on the method of elastic weight consolidation [14]. The approximation term limits how much weights in the model can change, depending on their importance for the original domain. Hence benefiting from transfer learning of information from the original domain to learn the target domain. The importance of each weight to the information in the original domain is captured in the Fisher matrix, \(F_{0,j}\), which is calculated only once when training on the original domain initially. This is a step that would be performed during the development phase of a system employing DIRA, i.e.it is not calculated during operation when the system is required to adapt to new domains. \(\lambda\) is a hyper-parameter to allow for fine-tuning to minimise forgetting, and \(j\) labels each parameter in the neural network.

The previously developed optimisation problem by DIRA requires labels for the retraining samples to be provided. The method in this paper, DIRA-SS, aims at extending the optimisation problem proposed by DIRA further to achieve adaptation using a few samples from the target domain and without labels being provided. The goal is to achieve this by incorporating self-supervision in DIRA. To achieve self-supervision, the neural network is modified to become a multi-headed model. One head will be responsible for outputting a classification of the object and the second head is responsible for classifying some auxiliary task. The auxiliary task head is used in the retraining setting, whilst the classification task is used in the prediction (operational) mode.

In this work we choose the rotation predict task [18], which has been shown to be effective at representation learning [7], in the application of visual recognition, which is the main focus of our work. Other auxiliary tasks may need to be explored for different applications, such natural language processing (NLP), this is beyond the scope of our current work but is for future work. The auxiliary task we use rotates the image by 0, 90, 180, 270 degrees and then the model predicts the angle of rotation as a four-way classification problem.

In the retraining setting, the auxiliary head is attached and the input images are given a random rotation in multiples of 90 degrees. Note, that samples available for retraining are assumed to all be at approximately zero degrees rotation. Therefore, when a random rotation of 90 degrees is assigned to an image sample, the appropriate label of 0, 90, 180, or 270 can be assigned to the image. Retraining on the auxiliary task allows the model’s initial layers to adapt to new domains without providing classification labels of objects in images. This way, samples are allowed to propagate through the neural network and optimise the weights to interpret the new domain better. Figure 2 shows a representation of how we suggest DIRA can be modified to create DIRA self-supervised (DIRA-SS). The letter ‘A’ symbolises relevance to the auxiliary task, ‘0’ to the initial domain, and ‘T’ to the target domain. The point at which the multi-head split of the neural network is done can start at any layer \(k\in\{1,..., K\}\), where \(K\) is the number of layers in the neural network.

To formalise DIRA-SS, the main and auxiliary tasks share some of the model parameters \(\boldsymbol{\theta}_{S} = (\theta_1, ..., \theta_k)\) which include parameters in the input layer up until layer \(k \in \{ 1, ..., K\}\). The main task branch \(\boldsymbol{\theta}_{M} = (\theta_{k+1}, ..., \theta_K)\) contains the rest of the pre-trained layers used in predicting the main task. The auxiliary task gets its own branch with another set of parameters \(\boldsymbol{\theta}_{R} = (\theta^{'}_{k+1}, ..., \theta^{'}_K)\), this is the retraining branch. See figure 3 for a pictorial description of the joint architecture. For the retraining and predictions branches, they have the same architecture except for the output layer, due to the different number of classes between the main and the auxiliary tasks.

Initial training is done for model parameters \(\theta_S\) and \(\theta_M\) on the main task using data samples from the original domain, to create the initial model \(M_0\). After training on the main classification task, the auxiliary branch is then added to the model. The auxiliary branch, comprising of weights \(\theta_R\), is allowed to fine-tune on the auxiliary task whilst all other layers in the model are frozen. Note, this allows our method to be integrated with any existing single-headed pretrained model.

During domain adaptation, the model parameters \(\theta_S\) and \(\theta_R\) are allowed to optimise on the auxiliary task using samples from the target domain. This is done whilst parameters \(\theta_S\) are regularised on the original domain using elastic weight consolidation as was done in DIRA, to achieve adaptation using a few retraining samples. The optimisation problem therefore for DIRA-SS can be formalised as shown by equation 3 .

\[\displaystyle{\mathop{\mathrm{min\text{ }}}_{\theta_S,\theta_R}} \mathcal{L}_A(\theta_S, \theta_R) + \sum_{j}^{}\lambda F_{0,j}(\theta_{S,j} - \theta^{*}_{0,j})^2 \label{eq:DIRA-SS}\tag{3}\]

To utilise elastic weight consolidation during retraining there are certain assumptions made in DIRA. Since DIRA-SS utilises elastic weight consolidation as well, these assumptions need to be maintained. Satisfying these assumptions involves two conditions when retraining. The first condition, computing the Fisher matrix requires the distribution to be well expressed using a large pool of samples from the target domain. In the case of DIRA-SS and DIRA, this large pool of samples from the target domain is not available. Therefore, the Fisher matrix used in the optimisation problem is calculated using the original training dataset during initial training and a copy of this calculated Fisher matrix is stored for retraining, omitting the need to keep initial training data. The assumption that comes with the Fisher matrix, is that retraining starts from a model that is already optimised on the dataset represented by the Fisher matrix. This leads to the second condition. Since the system only has access to the Fisher matrix from the original domain, retraining using any number of samples, will need to start from the original model \(M_0\). Practically this is achievable as a copy of \(M_0\) can always be kept onboard of a system. This is to maintain the validity of the Fisher matrix used in the optimisation problem. For more elaboration on these conditions, refer to the Ghobrial et al. [4].

In each training step \(t\), the shared feature extractor represented by model parameters \(\theta_S\) is updated according to Equation 4 , where \(\eta\) is the learning rate. \[\theta_{S,t+1} = \theta_{S,t} - \eta \bigg( \dfrac{\mathcal{L}_{\text{A}}(\theta_S)}{d\theta} - \dfrac{\sum_{j}^{}\lambda F_0(\theta_{S,t,j} - \theta^{*}_{0,j})^2}{d\theta} \bigg) \label{theta95new95substituted}\tag{4}\]

Similar to DIRA, the two hyperparameters, \(\eta\) and \(\lambda\), are critical for the success of DIRA-SS. From empirical testing, done in DIRA it was found that a combination of \(\eta = 1\)e-5 and \(\lambda=1\) yields near optimum adaptation.These values were maintained for DIRA-SS too.

4 Experimentation Setup↩︎

We used the problem of image classification to showcase DIRA-SS. All of our experimentation was based in PyTorch library [23]. This section discusses the experimentation setup details used in evaluating DIRA-SS. Code is available at this repository: https://github.com/Abanoub-G/DIRA-SS

4.1 Benchmarks↩︎

We utilise CIFAR-10C, CIFAR-100C, and ImageNet-C datasets in our experimentation. These are image classification dataset benchmarks. They are aimed at evaluating the robustness of models against common corruptions [24]. The benchmarking datasets introduce different corruptions to the test sets of CIFAR-10/CIFAR-100[25] and ImageNet [26] There are 20 types of corruptions in total. For each corruption, five different levels of severities are provided. Most state-of-the-art (SOTA) domain-incremental retraining frameworks utilise 15 corruptions out of the 20 in their comparisons, e.g. [5], [7]. These are deemed the more common corruptions. We use the same 15 common corruptions used by other methods in the literature.

4.2 Baselines↩︎

We list below the different baselines we assess against our DIRA-SS approach:

1. Source: Refers to results of the corresponding baseline model trained on the incorrupt data, without adaption to the target domain.

2. TTT [7]: Test-Time Training (TTT) adapts parameters in the initial layers of the network by using auxiliary tasks to achieve self-supervised domain adaption.

3. NORM [20], [21]: Ignores the initial training statistics and recalculates the batch normalization statistics using samples from the target domain only (requiring a large number of samples).

4. DUA [5]: Dynamic Unsupervised Adaptation (DUA), takes into account initial training statistics and updates batch normalization statistics using samples from the target domain (requiring few samples).

5. DIRA [4]: Dynamic Incremental Regularised Adaptation (DIRA), is a method that used regularisation to benefit from transfer learning when adpating this is the method that DIRA-SS extends to make it unsupervised. DIRA is a supervised method, but required few samples).

4.3 Models and Hardware↩︎

We used ResNets [27] in our experiments, utilising two versions of ResNet: ResNet-18 (18-layer) and ResNet-26 (26-layer). For CIFAR-10/CIFAR-100, we used ResNet-26 as this is the version of ResNet architecture optimised for CIFAR-10/CIFAR-100 and is used by others in the literature. Initial training for the models was done locally. For ImageNet, we used a pre-trained off-the-shelf ResNet-18 model from PyTorch. Experiments for CIFAR10 and CIFAR100 were done on an MSI GF65 THIN 3060 Laptop with 64GB RAM and a Linux Ubuntu 20.04.2 LTS (64-bit) operating system, whilst for ImageNet a Dell Alienware Desktop PC with 64GB RAM and a Linux Ubuntu 18.04.4 LTS (64-bit) operating system were used.

To achieve reliable comparisons against baselines the starting model parameters from which retraining is done must be the same. Otherwise, the accuracy improvements cannot be reliably attributed to the effectiveness of the retraining method and can be argued that it is due to varying starting model accuracies or paramters. Therefore, in our results for CIFAR10/100 on ResNet-26 we only compare against Source, as we do not have the initial models used in retraining by other SOTA methods. For ImageNet on ResNet-18 we compare against SOTA methods because the starting trained model used by other SOTA retraining methods is the same off-the-shelf ResNet-18 model from PyTorch. This is a controlled experiment with same exact model initialisation and initial training model parameters, and therefore yields a fair and honest comparison between the methods.

4.4 Optimisation Details↩︎

Stochastic Gradient Decent (SGD) was used for optimisation during training and retraining. For an initially trained model, its architecture gets extended to include the auxiliary task. The model is then allowed to fine tune the auxiliary task layers only, whilst the rest of the neural network is frozen, using two epochs of training on the auxiliary task and the initial training dataset. This creates the initial model \(M_0\) that is then used in retraining, when adaptation is required.

During adaption to the target domain we used a learning rate value of \(\eta =\)​1\(e\)-5 and a lambda value of \(\lambda = 1\). Empirically these were found to be suitable hyperparameters for optimised retraining. The retraining is quick as only a small number of samples are used and the retraining is done over 10 epochs. We use top-1 classification accuracy as our assessment metric in all experiments [28].

5 Results & Discussion↩︎

5.1 Initial analysis on layers significance↩︎

An initial analysis was conducted on the layers that get adapted the most during retraining using DIRA. This is done to generate an initial understanding of whether DIRA-SS can potentially be successful. The general idea here is that if the initial layers are the ones that adapt the most, then adapting to representations learnt from the auxiliary task will be beneficial for successful adaptation. On the other hand, if the last layers (layers closer to the output) need to adapt the most then the benefit from adaptation using the auxiliary task may be limited.

To do this analysis the variance for model parameters between different adaptations of a ResNet-18 model using DIRA to different domains (noise types) in the CIFAR-10C benchmarking dataset, using different numbers of samples from 0 to 100 is calculated. Figure 4 shows a 3D visualisation of the variance for the model parameters. The layer index enumeration starts from the input side of the neural network and increases to reach the output side of the network. Layers closer to the input side seem to vary more compared to layers closer to the output side, in particular there are two layers showing the highest variance. This shows affirming indications that retraining using the auxiliary task will result in beneficial representations being learnt by the model. Investigating which representations are learnt by these layers and the exact reasons why specifically these two layers mount to the most effect on adaption is beyond the scope of this work and can be of interest to future work.

5.2 Overall Adaptation↩︎

Figure 5 shows how top-1 classification accuracy changes as the number of samples available from the target domain increases. The plot shows the effect of having different portions of the model parameters being shared between the auxiliary and the main tasks. ResNets, which are used in this study, consist of four blocks of layers. The adaptation is done using four different settings, progressively adding more layers to the shared set of model parameters in each setting. Visualising the results through two perspectives: 1) in the presence of approximately five data samples, and 2) when the data samples exceed five. For the first perspective, adapting all blocks results in the least effective adaptation out of the four settings, whilst adapting only the first and second blocks results in the best outcome. For the second perspective, adapting only the first block of layers results in the worst outcome, whilst all other three settings approximately result in the same outcome. From this analysis, it can be deduced that sharing the first and the second blocks results in the most optimum adaptation setting for DIRA-SS. This also coincides with observations from Figure 4, which shows that for good adaptation to new domains, the model needs to mainly adapt in its initial layers which fall in blocks 1 and 2 for ResNets.

To analyse the success of DIRA-SS method further, Table 1 and Table 2, compares between DIRA and DIRA-SS performance on the fifteen types of common corruptions from the benchmark datasets used for CIFAR-10 and CIFAR-100, respectively. Results are shown for retraining using 100 samples. In both methods, the adaption improves significantly compared to Source. DIRA achieves overall better adaptation than DIRA-SS. The improvement DIRA provides over DIRA-SS, however, is very little (less than \(1\%\)). This is impressive given that DIRA-SS is self-supervised/unsupervised, whilst DIRA is a supervised method.

5.3 Dynamic adaptation scenario for DIRA-SS↩︎

In real-life scenarios, varying domains may occur during operation. To visualise how DIRA-SS adapts to continuously changing domains, Figure 6 is plotted to show the shift to four different domains consecutively from CIFAR-10C. The results depicted show that the accuracy improves as samples from the target domain increase to a certain point then plateaus. The number of samples at which the plateauing happens varies depending on the noise type, but by 100 samples a plateau is reached for the four different types of domains used. Furthermore, Figure 7 shows the adaptation to the fifteen types of domains available in the benchmarking dataset for CIFAR-10C.

5.4 Comparison with SOTA↩︎

To assess how well our approach performs compared with SOTA domain adaptation frameworks we compare results with three domain adaptation frameworks from the literature: TTT, NORM and DUA, on the ImageNet dataset. Table 3 show top-1 classification accuracy for the highest severity level on dataset ImageNet-C. The DIRA-SS method performs competitively against SOTA domain unsupervised adaptation approaches. As can be seen from the table, it achieves SOTA overall performance averaged between the different corruptions in the benchmarking datasets. This is while using a limited number of samples from the target domain (100 samples). DIRA-SS overall performance drops slightly compared to DIRA, however, this small trade-off is expected as one drops the supervised learning provided by DIRA to adopt the self-supervision offered by DIRA-SS.

5.5 Retraining Time↩︎

Retraining time is a crucial aspect of runtime adaption. Specifically in autonomous systems that operate in time or safety-critical applications. Therefore, studying retraining time is very important. Figure 8 shows the time consumption for retraining ResNet-26 using a different number of samples on CIFAR-10C with DIRA-SS. The time consumption increases as the number of samples increases as one would expect. For this particular case study, the time consumption seems to range between 3 to 6+ seconds. Whilst this is relatively fast compared to initial training, it may still not be fast enough for applications where near instantaneous update of models to their operational domains is required i.e. retraining may need to be completed in the matter of milli- or micro-seconds.

Time consumed in retraining can be reduced by making models more efficient, as autonomous systems are expected to adapt in highly dynamic environments where speed is a critical factor for safety. This can be achieved through neural network compression, e.g. [28], optimised to run on hardware. Extending this work to explore the adaptation of compressed models on limited-resource hardware, is a topic of interest and is left for future work.

Furthermore, energy consumption is another topic of interest falling in the same context of efficient models that operate on limited-resources hardware. Making retraining methods more energy efficient can also be useful for applications with limited energy budgets, e.g. space applications.

6 Ablation Studies↩︎

6.1 Samples Type Effect↩︎

In all of the previous experiments, seeds were fixed to ensure the same samples were drawn and in the same order for all experiments. This was done to maintain a fair comparison between experiments and avoid any observed improvements in results happening due to uncontrolled experiment environments. In this section, however, the effect of using different samples, regardless of their order, is studied to generate a better understanding of the effect of changing samples used in retraining. To do this, the fixing of seeds is removed and retraining is run for 5 and 100 samples. The experiment is repeated two hundred times, and the mean and standard deviation are calculated. The source accuracy of the model is \(59.6\%\) Figure 9 plots the error bar results for the experiment. The larger standard deviation over the 200 runs occurs at adaptation on 5 samples, where it achieves \(68.6\pm 0.7\). At 100 samples the model achieves \(74.1\pm 0.1\) As can be seen, the effect of which samples are used and the order is minimal on the improved performance of the model from retraining on DIRA-SS.

6.2 Effect of Augmentation↩︎

In works, such as DUA [5], it was shown that using data augmentation had benefits to their retraining methods performance. Different types of augmentations exist, for example, rotate, vertical-flip, horizontal-flip, crop, etc. Throughout DIRA-SS experiments shown in this paper data augmentations were not used, as it was thought that it might have a negative impact on the auxiliary task. The auxiliary task can be seen as a form of data augmentation in itself because it incorporates rotations into the retraining. Due to the orientation being a building block in driving the self-supervised representation learning for DIRA-SS, data augmentations were avoided.

7 Conclusions and Future Works↩︎

This paper introduced DIRA-SS (Dynamic Incremental Regularised Adaptation - Self Supervised), a retraining method that allows for online domain adaption using few samples from the target domain. The method in essence advances the previously introduced method DIRA to make it unsupervised instead of supervised, by using the concept of auxiliary tasks to achieve self-supervised retraining. This is achieved by adding an additional head to the existing model. The original model head is used in classification. The additional head is used in retraining to allow the initial layers of the model to adapt without affecting the last layers in the model. From analysis carried out, it was found that initial layers in the model are the ones that adapt the most in domain adaptation. The auxiliary task used in DIRA-SS is rotation classification. Assuming all images input into the model during operation are at 0 degrees orientation, one can create augmented orientations of the image and allocate the rotation label to it, thus achieving self-supervised retraining. The method has been evaluated on benchmarks used widely to assess methods for domain adaption. The findings of this paper support the concept of using auxiliary tasks to achieve unsupervised or self-supervised online adaptation of neural network models. DIRA-SS has proven to be able to still achieve effective domain adaption without the need for providing classification labels during retraining. A trade-off in accuracy is experienced by DIRA-SS due to the absence of labels, in comparison with DIRA, this trade-off is typically less than 1% on average. However, DIRA-SS still achieves higher performance than other unsupervised state-of-the-art methods in the literature.

For future works, the retraining method introduced in this paper achieve state-of-the-art domain adaptation accuracies, further advancements are still required to achieve higher adaptation accuracies to reach the necessary levels required for reliable continuous adaption. The paper thesis did not explore thoroughly the retraining effects on neural network layers. An in-depth understanding of representations learned by different layers and the precise effect behind the adaptation of specific layers can perhaps lead to enhanced adaptability performance. Another director for future work is getting adaptation methods to function on real systems where resources are limited and speed of retraining is key to the adoption of a method requires more research. A future direction involves the exploration of retraining methods on compressed models deployed on limited-resource hardware or edge devices.

Acknowledgments↩︎

This research is part of an iCASE PhD funded by EPSRC and Thales UK.

References↩︎