1 Introduction↩︎

Deep Neural Networks (DNNs) have advanced by modeling the human brain at a very high level of abstraction. A key sociobiological trait is transactional knowledge exchange, based on teacher–student relations, which is a foundational dimension of human education systems. However, “DNN knowledge" differs from human knowledge; it is a function \(y = f(\mathbf{x};\mathbf{\theta})\) that can approximate unknown functions \(y = \phi(\mathbf{x})\) by exploiting large training datasets \(\mathcal{D} = \left\{ \left(\mathbf{x}_i, y_i \right), i=1,\dots, N\right\}\). As researchers can abstractly simulate both human cognition and social organization, a notable trend has emerged in the form of teacher-student machine learning methods. This transformative approach has shown great potential in how machines acquire knowledge across a wide spectrum of artificial intelligence (AI) problems [1]. Most teacher–student communities focus on Reinforcement Learning [2], which is out of the scope of this article. Only a few existing methods address multi-node supervised/semi-supervised scenarios (e.g., [3]). In this latter context, Collaborative Knowledge Distillation (CKD) taps the diverse knowledge of multiple DNNs to augment the community’s collective knowledge [4]–[6].

Although general KD and CL have previously been combined [7], [8], the situation is different for CKD. Existing CKD frameworks cannot acquire new tasks on-line: nodes remain limited to predefined tasks, and the sole CKD method that employs Continual Learning (CL) for dynamic acquisition of novel tasks allows only a one-off GAN-to-VAE distillation [9]. The non-task-agnostic nature of these frameworks also limits their potential. The DNN students in such frameworks acquire knowledge only when being aware of the task boundaries, i.e., when the switch of training data from an old task to a new one is fully defined and known [3]–[6]. In classification problems, a task boundary is defined as the last data point of the previous task learned before the first data point of the new task to be learned. Finally, the performance of existing CKD frameworks drops significantly when only a data subset is available during training, which is closer to a real-world scenario. Consequently, they cannot operate in a dynamic realistic environment, where raw unlabelled data is the only input.

The above limitations of current CKD methods have motivated our research. Aiming to address them, this article introduces the Learning-by-Education Node Community (LENC) framework, where nodes autonomously assume teacher or student roles in task-agnostic, on-line, transactional exchanges. Each node self-assesses its familiarity with current streaming inputs; if they are unknown, this node dynamically becomes a student and broadcasts the stream to its peers in order to detect potential teachers. If a teacher is found, its relevant knowledge is transmitted to the current student via neural distillation [10], while CL is employed for retaining prior knowledge; all fully automatically.

The flexibly decided teacher/student role, as well as the task-agnostic nature of the framework, simulates a community of human nodes where an unknown data stimulus coming from the environment triggers the urge to attain relevant knowledge from other humans, e.g., from a specialized teacher. Among multi-node CKD protocols, LENC uniquely couples task-agnostic CL with dynamic role assignment, which is a key characteristic of human communities.

Without loss of generality, the description of the LENC framework below assumes a classification application. Experimental evaluation on a proof-of-concept implementation demonstrates the LENC functionalities across multiple scenarios. The conducted experiments showcase the LENC framework’s ability to gradually improve the average test accuracy of the community of interacting DNN nodes in image classification problems, as well as its ability to learn on-line from small batches of unlabelled data. LENC performs well in the realistic task-agnostic, on-line CKD setting.

2 Related Work↩︎

The concept of teacher-student learning emerged from the idea of a single, untrained student DNN distilling the knowledge held by one or an ensemble of pretrained teacher DNNs [11], to strike a balance between efficiency and performance [12]. LENC is a novel multi-node framework for CKD where multiple peer DNNs can dynamically act either as teachers or students during deployment. They autonomously select their current role based on: a) whether each one is pretrained or not, and b) whether each of them recognizes each current external test input as known or not. To achieve its goals, LENC integrates Knowledge Distillation (KD), Out-of-Distribution detection (OOD) and CL into a common framework. Below, the state-of-the-art in each of the involved areas is surveyed, while positioning LENC with respect to competing methods.

2.1 Knowledge Distillation↩︎

The method in [13] exploits a complex pretrained DNN as a teacher in classification tasks, by using its output distribution as pseudo-labels for unlabelled data utilized in student DNN training. Notably, [10] demonstrated remarkable results by distilling knowledge from an ensemble of DNN teacher models into a single student model. Building upon [10] and [13], later studies explored teacher-student interactions to enhance KD, resulting in student DNNs exhibiting strong classification performance [9], [12], [14] [15], [16], [17]. For example, Relational Knowledge Distillation (RKD) [18] leverages structural information outputs from different teachers to refine knowledge transfer, while Curriculum Temperature for Knowledge Distillation (CTKD) [19] dynamically controls the difficulty level of tasks during a student model’s learning trajectory, by incorporating a dynamic, learnable parameter.

Recent advances have ventured beyond the classical teacher–student paradigm, introducing contrastive and on-line collaborative approaches that enrich training and improve generalization. Methods such as Complementary Relation Contrastive Distillation reframe KD as a structural feature transfer problem, using an anchor-based contrastive loss to align sample representations and inter-sample relations, thus demonstrating strong effectiveness across benchmarks [20]. In parallel, on-line collaborative approaches like Peer Collaborative Learning (PCL) eliminate reliance on a fixed pretrained teacher by training multiple student peers jointly in a multi-branch network; the ensemble and temporal mean of peers serve as dynamic teachers, enabling one-stage, end-to-end KD with strong generalization [21]. Together, these methods underscore a shift toward more interactive and flexible KD frameworks that generate rich representations across domains.

2.2 Collaborative Knowledge Distillation↩︎

In collaborative learning, teachers and students mutually instruct and learn from each other, with peer DNNs possessing either identical or diverse architectures. Notably, [4] introduced Deep Mutual Learning (DML) for on-line KD, focusing on exchanging response-based knowledge between peer DNNs. Similarly, on-line Knowledge Distillation via Collaborative Learning (KDCL) was introduced in [5], leveraging an ensemble of soft-output activations to transfer knowledge between peer DNNs. DML was expanded in [6] with Dense Cross-layer Mutual distillation (DCM), enabling collaborative training of both teacher and student DNNs from the ground up. Additionally, [22] proposed a feature fusion method, known as DualNet, which combines features from two identically structured peer DNNs using a “SUM" operation. In contrast, a mutual KD method via Feature Fusion Learning (FFL) in [23] aims to collaboratively learn a robust classifier by merging features from various peer DNNs, which may have different architectures.

More recent algorithms include On-the-fly Native Ensemble (ONE) [24], which is a learning strategy for multi-branch students. The DNN branches (peers) exchange knowledge to enhance the model’s generalization abilities. Peer Collaborative Learning (PCL) [21] introduces a CKD approach for multi-branch students, which addresses the problem of missing the discriminative information among feature representations of DNN peers (branches). Weighted Mutual Learning (WML) [25] further improves the performance of the branches’ ensemble by estimating each branch’s relative importance. Switchable On-line Knowledge Distillation (SwitOKD) [26] uses a dynamic threshold to switch the mode of the DNN teacher from frozen weights (“expert") to unfrozen weights (”learning") and vice versa.

2.3 Continual Learning↩︎

To enable a single DNN to continuously acquire new skills without forgetting old ones, CL is essential [[27]][28]. The main hurdle is integrating new tasks [29] without “catastrophic forgetting" of prior knowledge [[30]][31]. Regularization approaches tackle this by penalizing changes to important parameters.

For example, Elastic Weight Consolidation (EWC) [32] assigns each DNN parameter an update penalty, based on its importance for previously learned tasks, using the Fisher Information Matrix \(\mathbf{F}\). Thus, DNN adaptation uses the following regularizer: \[\mathcal{L}_c(\theta) = \sum_i \frac{\lambda}{2}F_{ii}(\theta_i - \theta^*_{o, i})^2, \label{loss95cont}\tag{1}\] where hyperparameter \(\lambda\) sets the importance of previous tasks compared to the new ones and \(i\) indexes each DNN parameter. Any change to the \(i\)-th DNN parameter is penalized by a factor proportionate to the \(i\)-th diagonal entry of \(\mathbf{F}\), evaluated once on the parameter set where training on the old task had converged. Details about EWC are in the Supplementary Material.

Learning without Forgetting (LwF) [33] preserves old-task performance by distilling outputs from the DNN’s own previous version while training on new data, without requiring access to old training data. The method deviates from vanilla KD; the teacher is the older version of the student itself and not a different DNN. Encoder-Based Lifelong Learning similarly uses autoencoders to regularize feature changes alongside a KD loss, with an initial warm-up phase [34].

Replay-based CL methods instead store a small set of exemplars from past tasks and train jointly on new data and these exemplars. Incremental Classifier and Representation Learning (iCaRL) selects exemplars via a herding strategy to match class means in feature space [35], while End-to-End Incremental Learning (EEIL) adds a balanced finetuning stage with equal exemplar counts per class [36]. Since the resulting dataset is skewed toward new data – causing task-recency bias – methods like Learning a Unified Classifier Incrementally via Rebalancing (LUCIR) introduce cosine normalization, a less-forget constraint, and inter-class terms to rebalance learning [37], and Pooled Outputs Distillation Network (POD-Net) builds on LUCIR by applying pooled-output KD with a local similarity classifier [38].

2.4 Task-Agnostic Continual Learning↩︎

Task-agnostic supervised CL does not require explicit information regarding the task boundaries, i.e., at which incoming training data point the previous set of classes (task) switches to a new one. Bayesian Gradient Descent (BGD) [39] employs an on-line version of variational Bayes, updating the mean and variance for each parameter using the posterior distribution from the previous task as a prior distribution for the new task, mitigating catastrophic forgetting. However, it relies on the “labels trick" which does break the task-agnostic assumption, since the task identity is inferred from the class labels during training. An alternative called iTAML [40] uses meta-learning to maintain generalized parameters for all tasks, adapting to a new task with a single update at inference. Despite supporting undefined task boundaries, iTAML does require task boundaries to be known per data point during training. In contrast, Hybrid generative-discriminative Continual Learning (HCL) [41] models task and class distribution with a normalizing flow model, using anomaly detection for automatic task identification and combining generative replay and functional regularization to prevent catastrophic forgetting. Continual Neural Dirichlet Process Mixture (CN-DPM) [42] is an expansion-based method that allocates new resources for learning new data, formulating the task-agnostic problem as on-line variational inference of Dirichlet process mixture models. It employs neural experts, each handling a data subset, equipped with short-term memory for managing new data points and creating new experts when needed. Finally, the Task-Agnostic Continual Learning using Multiple Experts (TAME) [43] algorithm uses multiple task experts, which are completely separate DNNs. TAME automatically detects shifts in data distribution by keeping track of the loss value of the expert being trained, thus initializing a new expert on-line upon a high loss value variation, which potentially marks the onset of a new task.

2.5 Out-of-Distribution Detection↩︎

To enable a DNN node to self-assess whether a test sample \(\mathbf{x}\) originates from its training distribution \(p(\mathbf{x}\mid\mathcal{D})\), OOD detectors are employed during deployment to reject inputs whose predictions may be unreliable. Modern OOD detectors fall into classification-, generative-, distance-, or reconstruction-based approaches. Classification methods reshape the semantic label space into hierarchical taxonomies and apply top-down classification or group-softmax training to detect novel inputs [44]. Alternatively, vision–language techniques learn dense label embeddings and flag inputs as “novel" when the model’s regression output lies beyond a predefined distance threshold to those embeddings [45].

Distance-based approaches assume that OOD samples lie far from class centroids in feature space, using metrics such as cosine similarity [46], Radial Basis Function kernels [47], Euclidean distance [48], or geodesic distance [49]. Other variants measure the feature norm in the orthogonal complement of the principal subspace [50] or learn hyperspherical embeddings to maximize inter-class angular separation and minimize intra-class dispersion (CIDER) for clearer ID/OOD separation [51]. In the above cases, the OOD detector employs the main DNN model features (activations). Reconstruction methods leverage instead the higher reconstruction error of Autoencoders on OOD inputs, including denoising variants like MoodCat [52] and hybrid schemes such as Reconstruction Error Aggregated Detector (READ), which maps pixel-level reconstruction errors into the classifier’s latent space [53]. A notable hybrid is asked Image Modeling for Out-Of-Distribution Detection (MOOD), which treats Masked Image Modeling (a self-supervised denoising pretext task) as a proxy for reconstruction-based distance detection [54].

Deep generative methods, such as flow models [55], train a separate DNN to probabilistically model the ID data distribution and thus allow test-time classification of ID and OOD data points. Instead of raw likelihood scores, likelihood ratios can be used [56] to avoid the occasional assignment of high likelihood to OOD data points [57]–[59]. Given that likelihood demonstrates a strong bias towards input complexity [60], Likelihood Regret (LR) [61] has been proposed as an efficient OOD score when using a Variational AutoEncoder (VAE) to model the ID data distribution. LR is the logarithmic ratio between the likelihood obtained by the posterior distribution optimized separately for a given input and the likelihood approximated by the VAE: \[\label{eq:32lr95calculation} LR_\tau(\mathbf{x})= L_\tau(\mathbf{x}; \mathbf{\theta^{*}},\widehat{p}(\mathbf{x}))-L_\tau(\mathbf{x}; \mathbf{\theta^{*}}, \mathbf{\phi^{*}}),\tag{2}\] where \(\widehat{p}(\mathbf{x})\) is the optimal posterior distribution of the Encoder parameters given the input data \(\mathbf{x}\), \(\theta^{*}\) is the optimal VAE Decoder parameters obtained from the training dataset and \(\phi^{*}\) is the optimal VAE encoder parameters obtained from the training dataset. LR relies on the intuition that a well-trained VAE model, when provided with a single ID test data point will exhibit only marginal improvement in likelihood if its current configuration is replaced with the one after optimization, resulting in small LR values. Instead, when presented with an OOD test data point, the model’s current configuration is expected to deviate significantly from the one after optimization, leading to large LR values, because the model has not been exposed to similar data points during its training phase.

Notably, OOD detection has not been previously integrated with CKD and CL simultaneously, with the only exception being HCL [41] (see Section 2.4).

2.6 KD in Advanced Node Communities↩︎

A few multi-node KD frameworks for supervised or semi-supervised learning have been proposed in recent years and do resemble the LENC framework in certain respects [62]. Particularly, Lifelong Learning Teacher-Student (LTS) [9] uses Generative Adversarial Networks (GANs) as teachers to replay previously learned knowledge to their students, to avoid catastrophic forgetting. Unlike LENC, LTS supports only a fixed teacher/student role for each DNN: once a student is trained, it cannot effectively transfer its acquired knowledge to a different student. Additionally, LTS does not operate in a task-agnostic manner, i.e., the boundaries of each task are required to be known. An alternative framework called DiverseDistill [3] focuses on transferring the knowledge of multiple, potentially diverse Foundation Models to smaller students, but does not incorporate CL. Therefore, only untrained students are supported. Similarly, [63] introduces a multi-teacher ensemble feature KD framework that attempts to find the optimal feature transformations/weights before KD. Applying such methods in recommender systems, Pre-trained Recommendation Model Knowledge Distillation (PRM-KD) [64] maps the output of multimodal teachers using a consistent scoring strategy, which is then used to distill their knowledge to a single student. Additionally, Multiple Teachers Model for detecting and Ranking (MulTMR) [65] uses a multi-teacher weighted feature KD loss to transfer the knowledge of multiple multilingual Large Language Model (LLM) teachers into a single student of identical architecture. Data-Agnostic Consolidation (DAC) [66] introduces a data-agnostic approach to the general CL problem, using augmented OOD samples and KD to learn from new unlabelled data. In contrast to LENC, DAC focuses on CL and is not suitable for environments with multiple potential teachers or time-varying data. This also holds for the KD-based Complementary Calibration (CoCa) CL method [67], although CoCa introduces an interesting approach to exploit latent-space features for CL on labeled data streams.

In short, among multi-node CKD protocols, LENC uniquely combines CKD, CL and autonomous teacher discovery in a task-agnostic manner. LENC participants can be dynamically assigned teacher/student roles and exchange knowledge entirely autonomously and in perpetuity, while being deployed, across multiple independent tasks and with unlabelled, raw test data points as their only input. LENC generalizes and extends the functionalities of existing advanced node communities, minimizing the need for manual human intervention even in the presence of multiple unknown tasks.

3 LENC Framework Architecture↩︎

The proposed novel LENC framework defines the protocol for LENC nodes to learn tasks from other peer LENC nodes that participate in a LENC community. Consequently, every deployed LENC node can assess on-line its knowledge of incoming external test data points. In case of ignorance, it can decide to transfer it to other nodes via teacher-student interaction to: a) identify potential teachers, and b) learn from them. The integration of CL and CKD, combined with the ability of each node to self-assess its knowledge, emulates a human community where all nodes cooperate to broaden their knowledge on multiple tasks.

3.1 LENC Node Architecture↩︎

Fig. 1 represents the structure of a LENC node. Each LENC node contains a Feature Module (FM), namely a DNN model \(f\), parameterized by \(\mathbf{w}_s\), which is typically a DNN. \(f\) is assumed to be shared across \(T\) tasks, \(T \geq 0\), on which it has been trained using the appropriate corresponding training datasets \(\mathcal{D}_\tau, \tau = 1,\dots, T\). The shared FM culminates in \(T\) individual Decision Heads (DHs) \(\tilde{f}_\tau, \tau=1,\dots, T\), parameterized by \(\mathbf{w}_\tau\), so that the decision for each task is taken by the function \(y_\tau = \tilde{f}_\tau(f(\mathbf{x};\mathbf{w}_s);\mathbf{w}_\tau), \tau=1,\dots, T\) for an input test data point \(\mathbf{x}\). This structure allows the deployed node to support multiple tasks, which potentially have a different number of semantic classes, using a single DNN. Optionally, \(a_\tau\) can be stored along with \(\tilde{f}_\tau\): it is this node’s known accuracy in the test set of \(\mathcal{D}_\tau\), as measured before deployment using any task-appropriate evaluation metric.

Each node also contains \(T\) Knowledge Self-Assessment (KSA) modules \(g_1, ..., g_T\), which assess if incoming test data points match the distribution of the corresponding original training datasets \(\mathcal{D}_\tau\). Each LENC node can autonomously decide to act as either a student or a teacher DNN, depending on the output of its KSAs. Furthermore, each node contains a set of Interaction Rules (IRs) that specify its communications with other nodes and, thus, shape teacher-student interactions.

If a LENC node’s KSA modules indicate that it does not have sufficient knowledge of current test inputs, this particular node can temporarily become a student and trigger a search for one or more teacher nodes in the LENC community. Once they are found, the student may learn from them using its IRs. This process, when triggered by a specific incoming test data stream that proves to be unknown or not well-known, is called an education cycle: one iteration of KSA-triggered teacher discovery followed by policy-based knowledge transfer.

The various LENC node modules are subsequently detailed.

3.1.1 Knowledge Self-Assessment (KSA) Module↩︎

The \(\tau\)-th KSA module of a LENC node consists of an OOD detector \(g_\tau(\mathbf{x})\) corresponding to the \(\tau\)-th task, \(\tau=1,\dots, T\), the node has been trained on. It is assumed that each OOD detector has been pretrained/preconfigured per node and per task on that task’s ID training data, prior to deployment. During system deployment, it assigns an OOD score to a stream of incoming test data points \(\mathbf{x} \in \mathcal{X}_\tau\), thus assessing the relevant knowledge of the FM. Thus, the \(T\) node’s KSA modules are tailored to the \(T\) known training datasets \(D_\tau\) that the node has encountered before its deployment. In short, the KSA modules provide each node with task-agnostic, on-line knowledge self-assessment capabilities, allowing it to assess on its own the DH most knowledgeable/relevant to the current test data point \(\mathbf{x}\). As a result, during FM inference on \(\mathbf{x}\), the node automatically detects and activates/utilizes the \(j\)-th DH \(\tilde{f}_j, j = 1, \dots, T\) as the one most relevant to \(\mathbf{x}\), since \(j = \arg \min (g_1, \dots, g_T)\) is the index of the supported task with known training data most similar to the ones found in \(\mathcal{X}\). This implies that, assuming the decision for the \(\tau\)-th task is taken among \(c_\tau\) classes, then the node will finally predict \(y_j = \tilde{f}_j(f(\mathbf{x};\mathbf{w}_s);\mathbf{w}_j)\).

Given an unlabelled data point \(\mathbf{x}\) incoming from the external world, each of the node’s KSA modules may output one of three potential verdicts, based on the respective raw \(g_\tau\) value: i) the task is not known at all by the FM (non-expert), ii) the task is known, but the node’s relevant knowledge is limited (limited knowledge), or iii) the task is well-known by the node’s FM (expert). In the first and the second case, an education cycle will be triggered as a response (see Section 3.2), but in the limited existing knowledge scenario the existing DH will be employed for receiving education, instead of appending a new DH. In the third case, no education is kick-started and the node only infers on the incoming data stream. The verdict depends on the KSA’s internally computed OOD score for the current stream and on two relevant manually prespecified (and, in principle, task-specific) thresholds: \(\delta\) and \(\epsilon\). Thus, the first, second, or third case is activated if \(g_\tau(\mathbf{x}) \geq \delta\), \(\epsilon < g_\tau(\mathbf{x}) < \delta\), or \(g_\tau(\mathbf{x}) \leq \epsilon\), respectively.

3.1.2 Interaction Rules↩︎

The LENC node Interaction Rules (IRs) are defined by the LENC framework and serve three basic LENC node interaction functions. The first one is that they specify the interaction between a deployed LENC node and the external environment, which can constantly fetch unlabelled test input data points for analysis in the form of a data stream \(\mathcal{D}^s = \left\{\mathbf{x}_i\right\}\), where \(i=0, \dots, M^s\) and \(M^s > 0\) is the total number of currently received data points. If the node’s KSA modules respond that the FM does not know the current test data distribution well or at all, and is therefore a non-expert or has limited knowledge, an education cycle is triggered: DNN experts are automatically searched for within the LENC node community, in order to serve as teachers.

The second IR function specifies the transmission of the data stream \(\mathcal{D}^s\) from LENC node \(s\) to the other \(N-1\) LENC community participants and the reception of their responses: \(\left\{q_n, n = 1,\ldots, N, n \neq s\right\}\). The response \(q_n\) from the \(n\)-th LENC node is set to 0 if none of that node’s KSA modules recognizes \(\mathcal{D}^s\) as in-distribution. Otherwise, it is a non-zero numerical score that denotes how well the \(n\)-th node and its most suitable DH know the distribution of data \(\mathcal{D}^s\). Assuming that the KSA modules of the \(n\)-th node indicate that the latter’s \(j\)-th DH/supported task is the most suitable to \(\mathcal{D}^s\), three different policies can be alternatively employed for computing \(q_n\). In turn, \(q_n\) can be used for teacher selection at each knowledge transaction. These IRs are shown in Algorithm 2. The three alternative policies are:

• \(q_n = a^n_j\), where \(a^n_j\) is the optionally prestored average classification accuracy (e.g., Correct Classification Rate).

• \(q_n\) can be a function \(g_j\) of an OOD score internally computed by the \(j\)-th KSA module of the \(n\)-th node, given \(\mathcal{D}^s\): \(q_n = g_j\), where higher \(g_j\) implies worse match⁵.

• \(q_n\) can be a scalar measure of the disagreement between the current student LENC node and the \(n\)-th LENC node. To this end, the churn metric [68] can be used: the misclassification rate of student predictions given as input \(\mathcal{D}^s\), using the corresponding predictions of the \(n\)-th node as pseudo-ground-truth. For a set of \(v\) coupled predictions \(\mathbf{y}^n = \left[y^n_1,...,y^n_v\right]\) and \(\mathbf{y}^s = \left[y^s_1,...,y^s_v\right]\), this is defined as: \[q_n = C\left(\mathbf{y}^n, \mathbf{y}^s\right) = \frac{1}{v}\sum_{i=1}^v r(y^n_i, y^s_i),\] \[r(y^n_i, y^s_i) = \begin{cases} 1, & y^n_i \neq y^s_i \\ 0, & y^n_i = y^s_i. \end{cases}\]

The LENC node with the best non-zero response (\(t = \arg \max(\{q_u, u\neq s\})\) for options a) and c), \(t = \arg \min(\{q_u, u\neq s\})\) for option b)) is then selected for transferring its knowledge of \(\mathcal{D}^s\) to the querying student LENC node. In the case of Disagreement Policy (c), this will lead to learning from the teacher LENC node having the maximal output disagreement with the student LENC node output, among KSA-approved candidate teachers. This policy is intended to prioritize task-relevant peers that carry information not yet matched by the student; however, disagreement is a heuristic and not a correctness guarantee. A confidently wrong teacher could still induce negative transfer, so safety-critical deployments should combine this policy with confidence/OOD gates, teacher-history scores, or multi-teacher agreement.

Third, the IRs are responsible for specifying the actual teacher-student knowledge exchanges. Various policies can be alternatively employed, resulting in messages of different content. The four different knowledge transfer policies implemented in LENC are extensively discussed in Section 3.2.

3.2 Learning a Novel Task↩︎

As in the case of human societies, the external world constantly provides novel data stimuli to be classified by one or more of the deployed individual LENC nodes participating in the LENC community. For each such test input data stream \(\mathcal{D}^s\), the first question that needs to be answered is if the triggered \(s\)-th LENC node is knowledgeable of it. Such a question is answered using this LENC node’s KSA modules. If the task is judged to be known and the LENC node is considered an expert (see Section 3.1.1), no action is taken by LENC and the node proceeds to infer its own predictions for \(\mathcal{D}^s\). In any other case, an education cycle is triggered: the \(s\)-th LENC node temporarily assumes a student role and sends \(\mathcal{D}^s\) to other active LENC nodes within the community. Each of the other \(N-1\) LENC nodes receives \(\mathcal{D}^s\) and forwards it through its own KSA modules. Then, the \(n\)-th node replies to the student with \(q_n\), \(n \in \{1,\dots,N\}\)\\(\{s\}\), which specifies whether it knows the distribution of \(\mathcal{D}^s\) or not. From this point on, in the context of this particular knowledge transaction, any LENC node with non-zero \(q_n\) is considered a potential teacher. In the implemented LENC system the student node automatically selects as actual teacher the LENC node with the highest \(q_n\) score (see Fig. 3). However, in principle, knowledge transfer from multiple teachers to a simple student LENC node can also be supported.

Assuming that the student LENC node already knows \(T^s \geq 0\) tasks, using \(T^s\) existing DHs \(\tilde{f}^s_1, \dots, \tilde{f}^s_{T^s}\), it will now learn the new task driven by the dataset \(\mathcal{D}^s\), using the selected teacher’s knowledge. If the student LENC node has limited prior knowledge of the task (see Section 3.1.1), the existing DH for the current task will be enhanced via knowledge transfer from the teacher. In contrast, if the student has no prior knowledge of the task, a new DH will be generated and appended. As hinted in Section 3.1.2, the LENC framework offers four alternative policies for teacher-student knowledge transfer. Whatever policy is selected, it can be combined with a CL method, so that the student retains its previously acquired knowledge and does not experience catastrophic forgetting, just as in human society. The four knowledge transfer policies are illustrated in Fig. 3, demonstrated in Algorithm 4 and explained below.

Policy 1: Training Data Transfer Policy The student LENC node receives from the selected teacher LENC node the latter’s stored labeled training dataset \(\mathcal{D}^t_j, j\in{1,\dots,T}\), which was utilized before deployment to train the teacher for its \(j\)-th task. This known dataset is the most relevant to the current test data stream \(\mathcal{D}^s\). The student LENC node trains on it in a classical manner, using the selected CL method. Of course, if the student LENC node was previously untrained (\(T^s = 0\)) then the integrated CL algorithm does not need to be activated. Any common problem-specific loss function can be employed. A simple example for classification follows below: \[\mathcal{L}_t = \begin{cases} \mathcal{L}_c + \alpha \mathcal{L}_h(\mathbf{y}, \tilde{\mathbf{y}}^s), & T^s \geq 1 \\ \alpha \mathcal{L}_h(\mathbf{y}, \tilde{\mathbf{y}}^s) , & T^s = 0, \\ \end{cases} \label{eq::KnowledgeTransfer1}\tag{3}\] where \(\tilde{\mathbf{y}}^s\) is the student node’s prediction for input in \(\mathcal{D}^t_j\), \(\mathbf{y}\) is the respective one-hot-encoded ground-truth label from \(\mathcal{D}^t_j\), \(\mathcal{L}_h(\cdot,\cdot)\) is the cross-entropy loss, \(\mathcal{L}_c\) is the EWC CL regularizer and \(\alpha\) is a hyperparameter controlling the relative influence of the main cross-entropy loss term.

Policies 2-3: Knowledge Distillation Policies When the second or third policy is selected, the student receives from the selected teacher node the latter’s soft-output activations \(\mathbf{\tilde{a}}^t_j = \tilde{f}^t_j(f^t(\mathbf{x}^t;\mathbf{w}_s);\mathbf{w}_j)\). Note that, obviously, for classification problems the teacher’s respective prediction is \(\tilde{y}_j^t = \arg \max (\mathbf{\tilde{a}}^t_j)\). There are two alternative options for selecting which inputs these teacher activations are generated from: \(\mathbf{x}^t\) can come either from \(\mathcal{D}^s\) (unknown ground-truth labels) or from \(\mathcal{D}^t_j\) (known ground-truth labels). With the first option, the student is subsequently trained using \(\mathcal{D}^s\) and a KD loss (e.g., the one from [10], for simple classification problems). With the second input option, the teacher sends to the student \(\mathcal{D}^t_j\) as well and the latter is trained by a combination of a KD loss and a common, problem-specific loss (as in Policy 1). CL is utilized with both options, if the student is not previously untrained, as in Policy 1. A simple classification loss function example of the second input option without CL is the following one: \[\mathcal{L}_d = \mathcal{L}_h + \beta KL(\mathbf{\tilde{a}}^t_j, \mathbf{\tilde{a}}^s),\] where KL denotes the Kullback-Leibler (KL) divergence and \(\beta\) is a hyperparameter to control the relative influence of the KD loss.

Policy 3 is similar to Policy 2, with the exception that the teacher also sends its intermediate layer activations \(\mathbf{\tilde{u}}^t = \left\{\mathbf{\tilde{u}}^t_1, \dots, \mathbf{\tilde{u}}^t_k \right\}\), where \(k\) denotes a neural layer index within the teacher’s FM. The intermediate layers of interest must be predefined for each DNN architecture, while this policy is currently only available when the FMs of the teacher and the student share a common FM architecture. In this case, knowledge transfer to the student proceeds by combining a KD loss term with the Fit-Net [14] loss, designed to integrate the knowledge of intermediate teacher node layers. Thus, a simple example loss function for Policy 3 in the case of classification problems, using the first input option (without CL) is the following one: \[\mathcal{L}_f = KL(\mathbf{\tilde{a}}^t, \mathbf{\tilde{a}}^s) + \gamma \sum \| \tilde{\mathbf{u}}^t - \tilde{\mathbf{u}}^s \|,\] where \(\gamma\) is a KD hyperparameter to control the relative influence of the Fit-Net loss.
Policy 4: DNN Model Transfer
In Policy 4 the teacher node sends to the student LENC node its FM and DH parameters (\(\mathbf{w}^t_s\), \(\mathbf{w}^t_j\)) and structure (\(f^t\) and \(\tilde{f}^t_j\)), for its \(j\)-th task, \(j\in \{1, \dots, T^t\}\). This option is only applicable when the student LENC node is fully untrained at the beginning of the current knowledge transaction (\(T^s = 0\)). Essentially, the student LENC node is transformed into a copy of the teacher LENC node.
Policy Selection Rules. The four different knowledge transfer policies support the LENC functionalities under different conditions. Thus, a set of rules has been devised for automatic selection of the current policy based on the external environment within which the LENC community operates, as defined by the user. These user-set conditions are essentially answers to the following three questions:

• Are there model architecture/dataset/parameter privacy limitations during knowledge exchanges?

• Are there network traffic limitations within the LENC community?

• Is there a need to minimize the latency of each knowledge transaction?

For example, privacy and network traffic considerations may lead to limitations on whether each node’s known training datasets, internal parameters, or architectural details can be shared. Under the strongest sharing limitations, where restrictions are placed simultaneously on architectures, datasets and model parameters, only Policy 2 with the first input option (distill the teacher’s soft-output activations, given \(\mathcal{D}^s\) as input) can be actually employed. This is the most privacy-preserving and architecture/dataset-agnostic option, thus it is the default choice.

If it is known that the student and the teacher share a common neural architecture, and there are no bandwidth restrictions, then Policy 3 is selected as a better option due to a higher degree of student guidance by the teacher. For both Policy 2 and Policy 3, the second input option (distill the teacher’s soft-output and/or intermediate activations, given \(\mathcal{D}^t_j\) as input) is selected only if there are no dataset privacy and/or network traffic restrictions since \(\mathcal{D}^t_j\) is likely significantly larger than the current test batch \(\mathcal{D}^s\). Policy 4 is selected as a training-free option only if the following conditions are concurrently true:

• Each knowledge transfer needs to happen instantly due to latency restrictions,

• There are no model architecture/parameter privacy limitations, and

• The student is previously untrained.

Finally, Policy 1 is activated only when the following conditions hold simultaneously:

• There are neither dataset privacy nor network traffic limitations,

• There are no model architecture privacy restrictions (thus the student can know/learn the teacher’s architecture),

• The student’s architecture is significantly more complex than the teacher one.

Obviously, Policy 1 and the second input option for Policies 2-3 assume that each LENC node’s original training dataset for each of its supported tasks is stored along with its FM and DHs. If this is not the case, only Policy 4 and the first input option for Policies 2-3 are applicable.

4 Experimental Evaluation↩︎

Existing CKD literature has several limitations, as it leaves important use-cases unaddressed. Most CKD methods do not consider the potential help of experts who have acquired their knowledge asynchronously with their peers, or cases where the external environment provides only a few unlabelled data points to the node community. Such scenarios resemble more closely human learning in real communities. To evaluate the effectiveness of the LENC framework in similar setups, it was experimentally compared against existing CKD methods in scenarios involving on-line unlabelled CKD from a pretrained teacher.

In order to facilitate direct and fair comparisons with competing CKD algorithms, the main experiments were conducted using Policy 2 with the first input option for knowledge transfer (see Section 3.2) and the Disagreement Policy for teacher selection (see Section 3.1.2 for details on \(q_n\)). The Disagreement Policy does not require cross-node comparability of raw OOD magnitudes.

Two different sets of main experiments were performed: a) whole-image classification with untrained nodes and one expert, where no CL capabilities need to be activated, and b) whole-image classification where most nodes contain prior knowledge. In the first case, a traditional CKD setup is simulated (up to a degree), in order to facilitate comparisons against competing CKD methods. Thus there is only a single task (\(T^s=0\)), with only a single node having been pretrained and able to serve as an expert teacher. However, in contrast to existing CKD methods, this fact is not a priori known to the students, but automatically discovered by the LENC framework. In the second case, LENC is configured to run in a setup that demonstrates a fuller extent of its true capabilities. Most of the participating nodes are pretrained in different ways and LENC showcases its ability to handle on-line learning of multiple tasks on-the-fly, based on incoming unlabelled data points.

CL is employed only in the second experimental setup, while the KSA modules are utilized in both. However, in the CKD experiments the KSA modules are only used for identifying which node is a potential teacher, while in the CL experiments they additionally address task-agnostic CL by automatically identifying the task index (as described in Section 3). The baseline CL method of EWC [32] was selected for integration into the implemented LENC system, due to its combination of simplicity and good performance. These qualities of EWC underlie its continuing use as a building block by newer CL approaches [69], [70]. Similarly, LR [61] was selected for OOD detection within the KSA modules.

Additionally, comprehensive ablation studies were performed. These experiments were designed to showcase the LENC framework’s good performance in on-line unlabelled CKD, the ability of LENC nodes to avoid catastrophic forgetting without access to the task boundaries, the effectiveness of the different knowledge transfer policies and the different teacher selection policies (for computing \(q_n\)), LENC’s CKD performance across varying architectures and data stream sizes, as well as LENC’s performance when key components experience performance degradation.

4.1 CKD Experimental Setup↩︎

Following common CKD evaluation protocols [4], [5], [21], [22], [24]–[26], the LENC framework is evaluated on datasets CIFAR-10 and CIFAR-100, using nodes with the neural architectures ResNet [71], Wide-ResNet (WRN) [72] and VGG [73]. A pretrained ResNet-18 is employed as the only teacher, while two alternative sizes are utilized for the incoming data stream \(\mathcal{D}^s\) that originates in the external environment: 1000 and 5000 data points. The use of a pretrained expert excludes from the comparisons CKD methods for collaborative learning from scratch with neural branches of identical architecture [21], [24], [25]. The data points of a stream \(\mathcal{D}^s\) are randomly sampled from the teacher’s actual training dataset, with 10 different \(\mathcal{D}^s\) sets constructed in this manner. Each student receives sequentially the 10 streams, with each one triggering an education cycle; although the node is no longer entirely untrained after the first cycle, it is not an expert either. The competing CKD methods were adapted to distill the teacher’s response, instead of training with ground-truth, in order to enable fair comparisons with LENC. Although the competing SwitOKD method [26] uses ground-truth labels to calculate the teacher’s influence, it is also included in the evaluation.

The LENC community included two homogeneous students (2 ResNet-18 models) and two heterogeneous students (WRN-16-4 and VGG11). After hyperparameter search, the batch size was set to 128 and Stochastic Gradient Descent (SGD) was adopted as an optimizer, with an initial learning rate of 1e-3 and momentum of 0.9. The number of epochs for knowledge transfer was set to 100. Table 1 reports average community accuracy in the respective test set for the last education cycle, over all students and across 10 independent runs with different random seeds, along with the standard deviation over the different runs. A likely reason for LENC’s higher performance in this setup is the employed teacher selection policy: given the lack of ground-truth annotation, each student node picks a KSA-approved candidate teacher with which it disagrees the most, to leverage diverse knowledge within the community. Instead, the adapted competing CKD methods also consider the non-expert responses of other nodes.

A variant of this unlabelled experiment was also conducted using the more challenging Tiny-ImageNet dataset [74] and Transformer DNNs; this resembles a hard real-world scenario. The LENC community was formed by a ViT-base model as the teacher, achieving on its own an accuracy of 86.35% on the task, and two ViT-small students. In this setup, LENC was compared only against the best competitor [5], according to the outcomes of previous experiments. The data points of \(\mathcal{D}^s\) were randomly sampled from the teacher’s actual training dataset, with 15 different \(\mathcal{D}^s\) sets constructed in this manner. Results indicate that LENC does not clearly outperform the competing method for small \(\mathcal{D}^s\) sizes. However, as shown in Table 2, by gradually increasing the size of \(\mathcal{D}^s\) and repeating the experiment, at a size of 20000 data stream points LENC surpasses the best competitor by a large margin.

4.2 Continual Learning Experiments↩︎

The LENC framework can operate in a completely task-agnostic nature: the current \(\mathcal{D}^s\) data stream is sent to potential teachers to self-assess their knowledge and reply accordingly if they are aware of the current task. This enables the community to include multiple experts from multiple tasks, by providing an autonomous mechanism for finding suitable teachers, while concurrently supporting pretrained students via CL. The ability of the nodes to self-assess their knowledge, via their KSA modules, enables them to identify the task at hand and augment their knowledge of it. This is not feasible at all in existing CKD methods and, therefore, could not be evaluated comparatively to competing CKD algorithms. Instead, a dedicated CL experimental evaluation was conducted.

The experimental evaluation protocol for task-agnostic CL follows the relevant literature [41], [43] and evaluates the LENC framework on the SPLIT-MNIST, SPLIT-CIFAR-10, and SPLIT-CIFAR-100 datasets. Each dataset is split into 5 subsets/tasks containing 2, 2, and 20 classes respectively, resulting in 15 independent tasks overall. This experiment is a variant of the previous one. The community contains 15 pretrained nodes, each one knowing one of the 15 tasks, and 3 initially untrained nodes that will necessarily act as students. Each incoming data stream \(\mathcal{D}^s\), given to a student by the external environment, is of size 1000 and concerns 1 of the 15 tasks. A different \(\mathcal{D}^s\) is randomly sampled and transmitted to a student for each of the 15 tasks, sequentially. Each specific student receives 5 streams \(\mathcal{D}^s\) corresponding to the 5 different tasks of a single dataset (i.e., one dataset “assigned" to each of the 3 students). The different \(\mathcal{D}^s\) streams arrive in a random order and trigger consecutive education cycles: the students autonomously identify their need for education and find a suitable teacher within the LENC community, regardless of the total number of nodes or the tasks they are aware of. As a result, each of the 3 different student nodes is consecutively educated on the 5 CL tasks corresponding to one of the 3 datasets, without any information on the task indices. For these experiments, the batch size was set to 128 and SGD was adopted as an optimizer with an initial learning rate of 1e-3 and momentum of 0.9. ResNet-18 was used as a FM and the EWC hyperparameter \(\lambda\) was set to 500.

The results are visible in Fig. 5. The final average accuracy (mean of the five per-task test accuracies evaluated after learning all five tasks) is \(99.18\%, 68.62\%\) and \(28.35\%\) for SPLIT-MNIST, SPLIT-CIFAR-10, and SPLIT-CIFAR-100 respectively. Although these results decline compared to the results in Table 1 for the LENC framework (ResNet-18 with 1000 data points), it is crucial to notice that the method can achieve CL and adaptation with only a few randomly sampled batches.

4.3 Ablation Studies↩︎

Ablation studies were performed using the main experimental setup, to show whether picking the correct teacher improves performance in an on-line unlabelled CKD setting. It is demonstrated how the size of \(\mathcal{D}^s\) affects all compared methods. Additionally, the performance of the LENC framework for varying architectures is examined. Subsequently, the relative influence of the number of education cycles and of participating nodes is studied. Finally, alternative LENC knowledge transfer and teacher selection policies are assessed, given that the main experiments only rely on Policy 2 with the first input option (for knowledge transfer) and the Disagreement Policy (for teacher selection). Details follow below.

4.3.1 Varying data stream sizes↩︎

Fig. 6 shows the performance of the compared methods for varying \(\mathcal{D}^s\) sizes. All the CKD methods were evaluated for incoming streams of 50, 100, 200, 500, 1000, 2000, and 5000 data points, to comparatively validate the on-line learning capabilities of the LENC framework. LENC proves to be the most tolerant to small batch sizes, thus showcasing its usability in an important real-world use-case: when a node faces unknown current input data and needs to acquire relevant knowledge as soon as possible, in order to respond immediately. DML presents a minor advantage of 1% for data stream size of 50 samples, however it does not scale this well for the rest of the experiments, showcasing an overall lower performance compared to LENC. The latter’s superiority most likely stems from its avoidance of fusing the participating nodes’ responses. Instead, it uses a simple, yet effective way of evaluating all available nodes for tutoring and picking the correct teacher at each education cycle.

4.3.2 Diverse neural architectures↩︎

The use of the Disagreement Policy for teacher selection, using the churn metric, motivated an exploration of the role of diversity in CKD environments. Thus, multiple architectures are independently evaluated for \(\mathcal{D}^s\) streams of size 500, 1000, and 5000, per education cycle. Following relevant literature [21], [24], [25], the neural architectures ResNet-18, WRN-16-4, and VGG11 were employed, along with a more recent Vision Transformer (ViT) [75]. In this experimental setup, there are 3 expert nodes and 4 different groups of nodes. The experts are a ResNet-18, a ViT, and a WRN-16-4, pretrained on the CIFAR-10 dataset with a test accuracy of 92.31%, 80.91%, and 77.94 %, respectively. Varying node groups are used, composed of 2-3 students per expert, to investigate how the architectures affect LENC-powered CKD. The results are demonstrated in Table 3. As it can be seen, the simple presence of ViT within the community, without changing at all the teacher and the other peer nodes, boosts the performance of VGG11 and WRN-16-4. Given that ViT does not perform well on small datasets [76], this result further validates the argument that diversity in training enhances the overall knowledge of the community: diversity allows the other peers to achieve higher test accuracy. Another remark on Table 3 is that ViT proves to be a more effective teacher for WRN-16-4 than ResNet-18.

4.3.3 Scalability studies↩︎

Fig. 7 demonstrates the relative impact of the total number of education cycles and of participating nodes on the LENC framework. As expected, more education cycles lead to higher community performance (see Fig. [fig:ablation95teachingcircles]), assuming a fixed number of nodes (1 teacher and 3 students were used here). Similarly, Fig. [fig:abl95nodes] depicts the average test accuracy as the total number of nodes in the community rises, while keeping the number of education cycles fixed to 5. The performance reaches a peak for 3 nodes (2 students and 1 teacher). Empirical investigation of the results indicates that as the number of nodes increases, the number of education cycles should also increase accordingly in order to retain stable performance. In these experiments, \(\mathcal{D}^s\) streams of size 1000 were randomly sampled and transmitted from the external environment at the start of each education cycle.

4.3.4 Alternative knowledge transfer policies↩︎

Besides Policy 2 with the first input option for knowledge transfer, other LENC knowledge transfer policies were also evaluated and compared in separate, complementary experiments, for on-line unlabelled CKD using a pretrained teacher. The CIFAR-10 dataset was used, while a pretrained ResNet-18 with a test classification accuracy of 91.97% was deployed as a teacher model. Two initially untrained ResNet-18 models were deployed as student LENC nodes. Policy 1 is equivalent to classic training, if the LENC node is initially untrained and teacher selection has been completed. For Policies 2-3, both input options were independently evaluated: a) the first one using the unlabelled dataset \(\mathcal{D}^s\), and b) the second one using the labeled dataset \(\mathcal{D}^t_j\).

All experiments were conducted for a total of 10 education cycles. Table 4 reports the average student LENC node accuracy after the final education cycle, for both input options and for data streams of varying size, randomly sampled at each education cycle, in the unlabelled case. As it can be seen, the first input option for Policies 2-3 (using \(\mathcal{D}^s\)) can lead to very good performance with only minimal network traffic overhead, compared to the second one. Moreover, Policy 2 outperforms Policy 1 when using the second input option and even approaches it in classification accuracy when using the (lightweight) first input option.

4.3.5 Alternative teacher selection policies↩︎

Regarding the teacher selection policies (see Section 3.1.2 for details on computing \(q_n\)), the LENC framework was evaluated on CIFAR-10 using LENC nodes with ResNet [71], Wide-ResNet (WRN) [72] and VGG [73] neural architectures. A pretrained ResNet-18 model is deployed as a potential teacher, while the untrained students continuously receive data streams \(\mathcal{D}^s\) with varying cardinality, ranging from 50 to 5000 data points. Regarding knowledge transfer, the default Policy 2 with the first input option was used here. A student receives sequentially the 10 streams, with each one triggering an education cycle; although this LENC node is no longer entirely untrained after the first cycle, it is not an expert either. The teacher node is available every two education cycles for tutoring, so that the students can capture the diverse knowledge of their peers. Fig. 8 illustrates the average student LENC node classification accuracy after 10 education cycles. As it can be seen, the LENC framework outperforms the state-of-the-art methods for any choice of teacher selection policy among the three possible ones. Between them, the Accuracy Policy leads to the highest performance. However, this policy relies on static, pre-stored accuracy scores, which are not available for the initially untrained LENC nodes; this prevents them from ever acting as teachers, after enough education cycles.

4.3.6 LENC robustness↩︎

A set of ablation studies were dedicated to assess the robustness of the LENC framework against degradation of its critical internal components. First, robustness to suboptimal performance of the integrated CL method was evaluated by artificially tuning the regularizer hyperparameter \(\lambda\) for EWC (see Section 2.3) to various non-optimal values. Fig. 9 displays the results on the CIFAR100-SPLIT dataset, by training using only raw data for different values of \(\lambda \in \left\{100, 200, 500, 1000, 2000\right\}\). All experiments used streams of 5000 data points per task for training. As expected, higher values of \(\lambda\) generally lead to increased final accuracy for the two tasks that are learned first (Task 0 and Task 1). However, the highest average classification accuracy over all tasks was 55.72% for \(\lambda=200\).

Similarly, LENC’s robustness against suboptimal performance of its internal OOD detector was also measured in the on-line setting, where few incoming data points are available, using the integrated LR method (see Section 2.5). The conducted experiment involved independently training the OOD detector with varying number of data points from CIFAR-10, ranging from 50 to 5000, and then querying it on unknown images of CIFAR-10 and SVHN. The experiment indicated that the implemented version of LENC does not work correctly when its OOD detectors are trained with less than 1000 data points, due to failure of its KSA modules to accurately distinguish between known and unknown data.

LENC’s robustness to OOD failures was also evaluated by repeating the CKD experiment of 1 in the scenario of two untrained ResNet-18 students. The data streams contained 1000 CIFAR-10 data points. In this case, the KSA outputs of all nodes were corrupted by various levels of binary noise (to transform “non-expert" verdicts to”expert" verdicts, and vice versa), at a probability range of 0.1 to 0.5. This emulates the behavior of suboptimal OOD detectors, implying partial KSA failure. LENC demonstrated high tolerance to this, as shown in Figure 10, with only a slight decrease in the average accuracy of LENC nodes upon increasing binary noise.

LENC’s robustness on noisy data was also assessed, by repeating the CKD experiment of 1 in the scenario of two untrained ResNet-18 students. The data streams contained 1000 and 5000 CIFAR-10 and CIFAR-100 data points, now corrupted by 2D Gaussian noise of standard deviation equal to 0.01/0.05/0.2, for low/medium/hard noise level, respectively. In the CIFAR-10/5000 case, the students learned the task at hand via LENC with an average test accuracy score of 86.17%/83.93%/16.04%, respectively. As seen in 1, the corresponding accuracy for clean data (zero noise) is 86.31%. Thus, the students failed the challenge of hard corruption, but proved robust at low and medium noise levels. Additional results are provided in the Supplementary Material.

4.4 LENC failure modes↩︎

The ablation study revealed certain limitations of LENC. Given that KSA modules rely on OOD detectors and are responsible for a variety of LENC tasks, it is unsurprising that many of said limitations stem from unexpected KSA module behavior, likely due to OOD failure. In such a scenario, the most important issue arising is wrong teacher selection, i.e., a teacher is misidentified as an expert for the current task. In this case, the current education cycle is not positively contributing to the student’s knowledge. However, due to multiple education cycles, this situation is not irreversible and the student can end up learning the correct task by identifying the appropriate teacher in the next education cycles. Second, if a node already knows a task and the relevant KSA module marks the current data stream as OOD, then the node will append a redundant/duplicate DH, with a new KSA attached to it, for the same task. However, this is not fatal for the node’s operation: during the future education cycles, the duplicate DH/KSA might evolve to recognize such data streams as ID and overcome this issue automatically. The experimental results indicate that the KSA OOD detectors trained with too few data points are not accurate enough. However, KSA training with more than 1000 data points leads to sufficient accuracy. Finally, the LENC framework might be susceptible to highly corrupted data, as shown by the relevant robustness experiments.

Of course, LENC assumes complementary expertise in the community, where at least one peer covers the incoming stream. If all peers’ KSAs indicate non-expertise, LENC can detect this case but cannot distill missing knowledge without labels or an expert. In practice, such no-teacher events should trigger abstention, buffering, active annotation, or later onboarding of a new expert.

4.5 Discussion↩︎

The current LENC framework has limitations. First, scalability to larger communities is primarily constrained by teacher discovery: each education cycle broadcasts streams to all peers and triggers per-node KSA plus inference to compute \(q_n\). Thus, compute and network size grow roughly linearly and the few best teachers attract a disproportionately large share of requests, creating an imbalance that overloads those nodes, raising latency and contention. Communication is the next limiter, with knowledge transfer Policy 3 being orders-of-magnitude heavier than Policy 2. Each student consults only one teacher per cycle, so adding more nodes does not increase how much knowledge is fused/disseminated at each iteration. But since each cycle still broadcasts/collects across multiple heterogeneous peers, frequent cycles inflate per-round memory, bandwidth, and latency. Potential remedies include hierarchical clustering of nodes, selective knowledge routing, or communication compression strategies.

Another open challenge concerns robustness to adversarial or noisy teacher responses: Byzantine nodes or corrupted feedback could mislead students and destabilize the community. Future extensions could incorporate Byzantine-resilient aggregation, trust-weighted teacher selection, or consensus-based filtering to ensure resilience in decentralized and adversarial environments.

A related limitation is the fully open-world case where no peer has relevant expertise. In such a case, LENC should abstain from distillation, flag or buffer the unsupported stream, and optionally cluster repeated unsupported streams for later active annotation, self-supervised representation adaptation, or onboarding of a newly trained expert node. Once such knowledge is created, it can be registered as a new DH/KSA pair and disseminated through ordinary LENC education cycles. Thus, handling no-expert streams requires an outer open-world discovery/annotation layer, rather than a different teacher-selection rule alone.

Details relevant to a potential real-world implementation are in the Supplementary Material. In addition, the manual thresholds \(\delta\) and \(\epsilon\) employed by the KSA modules may be insufficient under concept drift or evolving data distributions. Dynamic strategies for threshold adjustment, e.g., on-line calibration, reinforcement learning, or meta-learning approaches, could provide more adaptive task-confidence decisions for distinguishing between non-expert, limited-knowledge, and expert roles. Addressing these issues will enhance the robustness and scalability of LENC, while moving it closer to viable, real-world deployment.

5 Conclusions↩︎

The Learning-by-Education Node Community (LENC) framework emulates human‐community learning for multi‐node, on-line Collaborative Knowledge Distillation (CKD) with DNNs and unlabelled data. Nodes autonomously adopt teacher or student roles, using no information on the task boundaries or index, to handle diverse data distributions, using Continual Learning (CL) and Out-of-Distribution (OOD) detection to learn on-the-fly from a peer deemed as suitable. Proof-of-concept image classification experiments show that LENC leverages peer diversity to achieve state-of-the-art on-line unlabelled CKD performance while supporting task-agnostic CL. Among multi-node CKD protocols, LENC uniquely combines CKD with task-agnostic, on-line CL from unlabelled data and dynamic role assignment, bringing CKD closer to human communities. Practical deployments may include edge, robotic, automotive, or IoT communities where deployed nodes have complementary expertise and need to exchange distilled knowledge under bandwidth, privacy, or latency constraints. Fully open-world operation, where no peer has relevant expertise, remains outside the present CKD formulation and requires external novelty-discovery, annotation, or expert-onboarding mechanisms.

References↩︎

---

1. Submitted for review on 30 September 2024. This work was supported by European Union’s Horizon 2020 Research and Innovation programme under Grant Agreement No. 951911 (AI4Media). Corresponding author: Ioannis Mademlis.↩︎

2. Anestis Kaimakamidis is with the Aristotle University of Thessaloniki, Thessaloniki, 54124, Greece (e-mail: akaimak@csd.auth.gr).↩︎

3. Ioannis Mademlis is with the Aristotle University of Thessaloniki, Thessaloniki, 54124, Greece (e-mail: imademlis@csd.auth.gr).↩︎

4. Ioannis Pitas is with the Aristotle University of Thessaloniki, Thessaloniki, 54124, Greece (e-mail: pitas@csd.auth.gr).↩︎

5. In the particular LENC implementation evaluated in this article, the OOD score is computed according to Eq. (2 ), i.e., the LR method.↩︎