1 Introduction↩︎

With the popularity of mobile computing and the Internet, more and more users are using mobile terminals for data communication and computing. With the increasing demand for user data privacy and local processing, federated learning technology has emerged in this context.

Federated learning technology [1]–[4] is a new distributed machine learning technology proposed in 2017, with Google being one of the first companies to propose this technology. Unlike traditional centralized machine learning, federated learning technology can balance the contradiction between data value and data privacy well. In this distributed computing paradigm, all participating nodes can collaboratively train a global model while protecting user privacy and data security. The data of each participating node is stored locally and can be updated on the server’s coordination. The trained model can be allocated to various participants or shared among multiple parties [5]–[8]. The advantage of this distributed machine learning technology is that it can effectively avoid data leakage [9]–[11] and data abuse caused by data privacy issues [12]–[15]. At the same time, federated learning technology is widely used in many scenarios, such as medical health, financial technology, intelligent manufacturing, graph retrieval, and recommendation [16], [17].

Recent studies also show that federated and graph-based representation learning are useful for heterogeneous information networks and cross-modal scenarios. FedSIN introduces federated self-adaptive learning for information network representation, while federated supervised cross-modal retrieval extends privacy-preserving learning to distributed multimodal retrieval tasks [4], [7]. Scientific and technological information retrieval and publication representation learning further indicate that semantic, media, and high-order hypergraph relations can be jointly exploited for robust scientific data mining [8], [18]. These works are related to the motivation of this paper because client representations in federated learning are often heterogeneous, multi-view, and dynamically evolving.

Therefore, solving the problem of non-independent and identically distributed data is one of the important directions for federated learning technology to address challenges [19]–[23]. This paper proposes a reinforcement federated aggregation method based on adaptive OPTICS clustering, which can solve the heterogeneity problem [24]–[27] of data distribution in federated learning tasks, named FedRO. Compared with traditional federated learning algorithms, the algorithm proposed in this paper can not only achieve efficient data recording, learning, and updating, but also share and aggregate data while protecting user privacy. Multi-view scholar clustering provides evidence that clustering dynamic academic entities is useful for modeling evolving interests, and modularity-based community detection shows that deep learning can assist graph community discovery [24], [25]. Heterogeneous graph attention networks and T2-GNN further suggest that incomplete features and heterogeneous graph structures can be handled by attention and teacher-student distillation mechanisms [26], [27]. In summary, solving the problem of data heterogeneity in federated learning is of great practical significance.

Although recommendation-oriented sequence modeling is not the main focus of this paper, filter-enhanced MLP and self-supervised graph co-training show that lightweight sequence models and graph self-supervision can capture dynamic user or session patterns [28], [29]. Retrieval-oriented pre-training such as RetroMAE is also related to the construction of discriminative feature representations for downstream retrieval and clustering [30]. These studies are cited as auxiliary evidence for feature learning and dynamic relation modeling.

The main contributions of this paper are as follows:

1. A reinforcement learning-based adaptive OPTICS clustering algorithm is proposed. This algorithm considers the problem of non-independent and identically distributed data across devices in federated learning. By clustering data from different devices, it can effectively solve the problem of uneven data distribution. Specifically, we use the advantages of the OPTICS clustering algorithm and use reinforcement learning to adaptively determine the core distance and minimum sample size of clustering, which can more accurately and reasonably handle data distribution.

2. A reinforcement federated learning method based on the adaptive OPTICS clustering algorithm is proposed. This method clusters clients into different clusters based on features using the adaptive OPTICS clustering algorithm and performs random selection within the clusters, which can make the federated learning more stable and accurate.

3. Experimental results show that the proposed adaptive OPTICS clustering algorithm and reinforcement federated aggregation method can effectively address the problem of data heterogeneity in federated learning tasks with non-independent and identically distributed data across different devices, improving the performance and accuracy of federated learning. The algorithm has good performance on the MNIST, CIFAR-10, and Fashion-MNIST datasets.

2 FedRO↩︎

The core idea of this article is to extract features from local data using the Deep Sets model, upload feature vectors to the server node, and use reinforcement learning to define the state space, actions, and rewards. The clustering environment is modeled as a Markov decision process, and the parameter search direction adjustment process is modeled to find the optimal clustering parameters eps and minPts for the OPTICS clustering algorithm to achieve the best federated aggregation method. Similar clients are assigned to the same cluster, and random selection is performed within the cluster. Each cluster is used to determine a model. The overall architecture of the solution is shown in Figure 1, which depicts the process of implementing reinforcement federated aggregation using the adaptive OPTICS clustering algorithm with three clients as an example.

2.1 Reinforced Federated Learning Method↩︎

Specifically, the search process in round \(k\) takes the following form. As the state needs to represent the search environment of each step as accurately and completely as possible, we consider constructing the representation of the state from two aspects (\(i=1,2,\ldots\)). First, the definition of the state for the global clustering status is: \[s_{\mathrm{global}}^{(k)(i)} = P^{(k)(i)} \cup D_b^{(k)(i)} \cup \left\{ R_{c_n}^{(k)(i)} \right\} .\]

Secondly, for the description of each class, for the local state of class \(c_n \in C\) at step \(i\), this paper defines a local clustering status definition: \[s_{\mathrm{local},n}^{(k)(i)} = \chi_{\mathrm{cent},n}^{(k)(i)} \cup \left\{ D_{\mathrm{cent},n}^{(k)(i)} \mid c_n^{(k)(i)} \right\} .\]

Based on the global state and local state, the current state is defined as: \[s^{(k)(i)} = \sigma\left(F_G\left(s_{\mathrm{global}}^{(k)(i)}\right) \parallel \sum_{c_n\in C} F_L\left(s_{\mathrm{local},n}^{(k)(i)}\right)\right),\] where \(F_G(\cdot)\) and \(F_L(\cdot)\) represent global and local feature encoders, respectively, and \(\parallel\) denotes concatenation.

2.1.0.1 Action.

The action represents the parameter search direction for step \(i\). This paper defines the action space as \(D=\{\textit{left},\textit{right},\textit{down},\textit{up},\textit{stop}\}\), where left and right represent decreasing and increasing the eps parameter, respectively. Down and up represent decreasing and increasing the minPts parameter, respectively, while stop represents stopping the search. Specifically, this paper establishes an Actor as a policy network based on the current state: \[a^{(k)(i)} = Actor\left(s^{(k)(i)}\right) .\]

2.1.0.2 Reward.

This paper uses a small portion of externally measured samples as the basis for rewards. The reward for step \(i\) is: \[R\left(s^{(k)(i)}, a^{(k)(i)}\right) = NMI\left(OPTICS(P',y,c)\right) .\]

2.1.0.3 Termination.

For the entire search process, we use the following termination conditions: stop when exceeding the boundary, or stop when exceeding the maximum number of steps.

The specific algorithm is shown in Algorithm 2.

The feature matrix is clustered using adaptive parameters with OPTICS, and the nodes are grouped. The weights of the entire cluster model are then sent to the worker nodes. The FedAvg algorithm is used to locally update the cluster, and the weights are sent back to the server. Finally, the server receives all the weights and aggregates them with weights.

3 Experiments↩︎

To verify the effectiveness of the FedRO algorithm, this paper evaluates the algorithm by training popular CNN models on three datasets, namely MNIST, CIFAR-10 [31], and Fashion-MNIST [32].

3.1 Datasets and Models↩︎

The statistical information of the experimental datasets and models is shown in Table 1. To demonstrate the reliability of the adaptive clustering algorithm in this paper, four datasets, glass, wine, yeast, and iris, are selected to test the clustering algorithm in this paper, using two evaluation indicators, Adjusted Rand Index (ARI) and Normalized Mutual Information (NMI). The specific information of the datasets is shown in Table 2.

3.2 Experiment Settings↩︎

In this paper, experiments were conducted on different datasets using 100 available devices. The datasets were assigned to different training nodes according to different sampling methods, and each node trained its own data locally. After each round of training, these training nodes sent their training results, i.e., their own model parameters, to the parameter server. The parameter server used a certain algorithm to aggregate these model parameters to obtain a new global model parameter. Then, this new global model parameter was distributed to the next round of training nodes, allowing them to continue training their local data using the latest global model parameter.

The following methods were selected as benchmarks for comparative experiments in this study: FedAvg, FedProx [33], and SHARE [34]. The FedProx algorithm is an optimization aggregation algorithm proposed in 2020 to solve system and statistical heterogeneity in federated networks, and SHARE is a hierarchical federated learning method proposed in 2021.

In federated learning, reducing the number of communication rounds is important due to the limited computing power and network bandwidth of mobile devices. Therefore, we use the number of communication rounds and the highest accuracy as performance indicators. In the clustering experiment, on the datasets shown in Table 1, the clustering indicators of the four clustering algorithms, K-means, DBSCAN, OPTICS, and the reinforcement learning-based adaptive OPTICS algorithm proposed in this paper, were compared. The ARI and NMI of the four algorithms were compared.

3.3 Federated Learning Experiment Results and Analysis↩︎

In reference [35], the parameter \(\sigma\) is used to measure the degree of non-independent and identically distributed data. We also use this method in this article, where a larger \(\sigma\) indicates a stronger degree of non-independent and identically distributed data. We use two parameters, 0.8 and 1.0, respectively. When \(\sigma=1.0\), it means that the data is the most non-independent and identically distributed, and each label belongs to only one device. When \(\sigma=0.8\), it means that 80% of the data belongs to one label and the remaining 20% belongs to other labels.

Figure 3 shows the experimental results of different algorithms using three datasets when \(\sigma=0.8\). Table 3 shows the maximum accuracy that three algorithms can achieve on each dataset. In Table 4, each entry shows the number of communication rounds required to achieve a test set accuracy of 99% on MNIST, 55% on CIFAR-10, and 85% on Fashion-MNIST for CNN. In this article, we conducted experiments at \(\sigma=0.8\) and 1.0, and counted the number of communication rounds required for different algorithms to achieve the standard accuracy. The shorter the number of rounds, the more efficient the algorithm is, the lower the workload, and the higher the performance.

From the experimental results, it can be seen that FedRO has good performance on all three datasets. The experimental results of this article show that FedRO can reduce the number of communications by up to 27% on the MNIST dataset, up to 38% on the CIFAR-10 dataset, and up to 17% on the Fashion-MNIST dataset.

3.4 Experimental Results and Analysis of Clustering Algorithms↩︎

Table 5 shows the clustering performance of four clustering algorithms on four datasets, including the values of ARI and NMI. The closer the value is to 1, the better the clustering effect of the algorithm.

From the experimental results, it can be seen that the reinforcement learning-based adaptive OPTICS clustering algorithm proposed in this article has good clustering performance on all four datasets. The algorithm dynamically adjusts the clustering coefficient by using the NMI value as the output of the reward function, which leads to good NMI values in the clustering task. In contrast, the non-adaptive OPTICS clustering algorithm has poor clustering performance because it cannot effectively handle datasets with no clear boundaries between clusters, and it cannot effectively cluster high-dimensional datasets. However, the proposed clustering algorithm in this article can achieve good results even on high-dimensional datasets by adaptively adjusting parameters.

4 Conclusion↩︎

This article proposes a reinforcement learning-based federated learning method based on adaptive OPTICS clustering, which can solve the heterogeneity problem of data distribution in federated learning tasks. Compared with traditional federated learning algorithms, the proposed algorithm can not only achieve efficient data recording, learning, and updating but also share and aggregate data while protecting user privacy. In the case of data heterogeneity, it reduces the impact caused by different data distributions, improves accuracy, and significantly improves performance.

When clustering data, there needs to be a balance between the number of clusters. The more clusters there are, the higher the accuracy of the models within each cluster, but the worse the global model’s generalization will be. However, if there are too few clusters, the accuracy of the models within each cluster will decrease. By using adaptive parameters, we can determine a range that both clusters the data in a better range and reduces the number of iterations required for adaptive parameter adjustment. In future research, we will further optimize performance through further study and experimentation.

References↩︎