1 Introduction↩︎

With the rapid development of science and technology and knowledge map research [1], a large number of science and technology patents have emerged. Broader innovation studies also indicate that scientific and technological resources interact with education, identity and creative capacity in innovation processes [2]. Scientific resources are often multi-view and dynamic: scholar clustering work has shown that research interests can evolve across views [3], and scientific information retrieval must handle both semantic and media heterogeneity [4]. However, at present, many knowledge graphs [5] related to science and technology patents are constructed by different institutions and individuals. Large graph processing also faces memory and scalability constraints, as shown by second-order random walk research on billion-edge natural graphs [6]. The requirements of these knowledge graphs are specific, and the design and construction are not uniform [7], so there are problems of heterogeneity and redundancy among them. The entity alignment of science and technology patents is the key technology in the process of knowledge fusion of science and technology patents. The main purpose is to find the equivalent entities between different science and technology patent data. Because the knowledge content of different science and technology patent data has different sources and human understanding, the text expression of the same thing in different data sources will be different, so there are significant problems in the integration of science and technology patent knowledge graph.

With the development of entity alignment technology [8], many scholars have proposed different kinds of entity alignment methods for science and technology patents, and a large number of entity alignment research documents have emerged [9]. Heterogeneous graph attention networks further show that typed nodes and relations can improve representation learning when supervision is limited [10]. In recent years, with the rapid development of knowledge representation learning technology, researchers have proposed a number of knowledge representation technology patent entity alignment methods based on deep learning [11]. This method uses knowledge representation learning technology such as graph neural network or graph convolution network to embed the knowledge graph, and maps different embedding spaces to the same vector space according to the aligned entities [12]. Some of these methods integrate other elements of scientific and technological patent entities into vector space through multi-information fusion [13], and get the result of entity alignment according to the distance or similarity of entities in vector space.

Because graph convolution network can capture the attribute characteristics and adjacency information between scientific and technological patent entities, and has strong robustness and generalization ability, the entity alignment method based on graph convolution network has a good effect. However, the existing method of entity alignment of scientific and technological patent documents [14] does not fully use the various information provided by scientific and technological patents, and the method of entity alignment of scientific and technological patents has a lot of room for improvement. Federated self-adaptive learning for information network representation also suggests that decentralized scientific resources can be modeled without forcing all data into a single platform [15], while federated learning with stochastic quantization provides a related direction for reducing communication and computation costs in distributed learning settings [16]. Therefore, the integration of entities from different sources of scientific and technological patents is an urgent problem to be solved.

The existing entity alignment methods for science and technology patents [17] still have shortcomings in multi-information fusion. Most of them [18] are only based on deep learning for structure embedding or based on word bag model for attribute embedding, and few of them fuse the embedded information of the two. Graph neural networks for graphs with incomplete features and structures indicate that missing or noisy information should be handled jointly rather than ignored [19]. Few multi-information fusion methods [20] also mostly use the attribute description representation method with poor effect.

In this paper, we propose an entity alignment method for science and technology patents based on graph convolution network and information fusion. The graph convolution network is used to deeply mine the structure information of knowledge graph, including topological connection, relationship and other information. In addition to the structural information, the entity attribute information in the scientific and technological patent is represented in depth semantics based on BERT to improve the effect of entity alignment.

The main contributions of this paper include three aspects:

1. A new entity alignment method for science and technology patent information fusion based on graph convolution network is proposed.

2. Based on the multi-level alignment method, the topological structure, relationship and attribute information are integrated into the graph convolution network.

3. A multi-information fusion method of science and technology patents based on graph structure embedding and attribute semantic representation is presented.

2 Related Work↩︎

Most of the traditional entity alignment methods of science and technology patents focus on syntax and structure [21], especially the early entity alignment and mapping technology of science and technology patents mainly focus on calculating the distance between labels and characters [22]. The traditional entity alignment method [23] of science and technology patents mainly solves the entity alignment problem from two aspects: one is based on similarity calculation to compare the symbolic features of entities, and the other is based on relational reasoning. Related interpretable machine learning studies for intelligent decision-making also show that feature semantics should remain understandable when models are used in practical decision scenarios [24]. Recent research also uses statistical machine learning to improve accuracy. However, due to the different attributes of entities and the different fields involved [25] in the process of entity alignment of science and technology patents, it is difficult to give a unified similarity calculation function. At the same time, this discrete attribute information ignores the semantic information [26] implied in many aspects, making the alignment effect limited. Therefore, in recent years, more and more scholars have begun to propose a new technology of entity alignment of science and technology patent based on deep learning.

The entity alignment method of science and technology patent based on graph neural network [27] has become the main technology to solve the entity alignment problem of science and technology patent, and has achieved good results. Most of these methods [28] use translation model or graph convolution network for knowledge representation learning, because they have strong robustness and generalization ability. The literature uses graph convolution network to embed the entities and relationships of science and technology patents. This model [29] combines the advantages of graph convolution network and the translation property of the knowledge graph of science and technology patents. It uses convolution function [30] to distinguish the entities of science and technology patents with different roles and effectively capture the structural information, but this method ignores the auxiliary role of the attribute description information of science and technology patents. Semantic-similarity attention with hypergraph convolution has also been explored for scientific publication representation learning, which is closely related to capturing higher-order scientific relations [31].

The NMN (Neighborhood Matching Network) model [32] is proposed in the literature. When the graph convolution network is embedded, the cross-graph attention mechanism is used to obtain the differences of adjacent technological patent entities. Federated learning for supervised cross-modal retrieval provides a related fusion setting in which supervision aligns heterogeneous representations under distributed data constraints [33]. Heterogeneous graph neural networks based on self-supervised reciprocally contrastive learning also suggest that contrastive signals can improve representation consistency across graph views [34]. The literature [35] proposed that the short distance difference and long distance dependence of adjacent scientific and technological patent entities can be captured using graph convolution network and self-attention mechanism, but this method needs a long time to capture the dependence information, and the operation efficiency of the model is not high.

Obtaining structure information based on graph convolution network will ignore the attribute information of science and technology patent entities [36], and realizing multi-information fusion of structure information and attribute information can improve the effect of science and technology patent entity alignment. Community detection with deep learning demonstrates that modular graph structure can provide useful high-level signals for representation learning [37]. The JAPE algorithm is proposed in the literature. The algorithm [38] includes two modules, namely, the structure embedding based on the relation triplet and the attribute embedding based on the attribute triplet. Self-supervised graph co-training in session-based recommendation further supports using complementary graph views when labels are limited [39]. The entity alignment of scientific and technological patents [40] is achieved through the multi-information fusion of the two modules. However, the method [41] is based on the graph structure embedding of scientific and technological patents based on the TransE model, and the accuracy of entity alignment is not high.

The literature [42] puts forward the CTEA framework, which is based on the TransE model to generate the structure embedding and context representation of the structure information of the scientific and technological patent entity. According to the attribute information of the scientific and technological patent entity, the candidate set is generated for each source entity and the subject information of the attribute information is captured. Efficient sequence modeling such as filter-enhanced MLPs is less directly related to entity alignment, but it offers useful evidence that lightweight feature filtering can improve representation tasks [43]; dataset distillation for sequential recommendation is another related data-efficiency problem, where compact training data are used to preserve model performance [44]. However, this method [45] has not been able to obtain the deep semantic representation of the attribute information. Retrieval-oriented masked autoencoder pre-training also suggests that language models should preserve ranking-sensitive semantics for downstream retrieval and matching tasks [46]. The document proposes [47] the MultiKE (Multi-view KG Embedding) framework, which integrates the structure information, attribute information [48] and entity name information at the same time. Incomplete multi-view learning work based on view-category interactive sharing transformers also shows that interaction among views and labels is useful when information sources are not complete [49]. The framework [50] is combined based on the shared space, and each embedding is orthogonally mapped to the same space to achieve multi-information fusion. However, the framework [51] is limited to general data sets and is not applicable to the field of scientific and technological patent documents [52].

3 Entity Alignment Method Based on Graph Convolution Network and Information Fusion↩︎

3.1 MIFEA Method Proposed↩︎

This paper proposes an entity alignment method for science and technology patents based on graph convolution network and information fusion (MIFEA). By integrating the graph structure embedding of science and technology patents and the deep semantic information of the entity attributes of science and technology patents, mapping to the same vector space for entity alignment improves the accuracy of entity alignment for science and technology patents. The method is mainly composed of four parts, including the embedded representation of the structure of the technology patent diagram, the semantic representation of the technology patent attribute, the multi-information fusion and the alignment of the technology patent entity.

The graph structure embedding representation module uses the graph convolution network to deeply mine the structure information of the knowledge graph, including topology connection, relationship and other information. The attribute semantic representation module performs deep semantic representation of the name and description information of entity attributes. The multi-information fusion module combines the graph structure embedding and attribute semantic representation to obtain the vector representation of the scientific and technological patent entity. The entity alignment module obtains the set of similar entities of each technology patent entity through similarity calculation and distance ranking.

3.2 Embedded Representation of Structure of Science and Technology Patent Map↩︎

MIFEA captures information from the graph structure of the graph of scientific and technological patent knowledge based on graph convolution network, and multi-layer graph convolution stack collects features from multi-hop neighbors. GCN can iteratively update the representation of each entity node through the propagation mechanism in the graph. The equivalent entities of science and technology patents tend to be adjacent to the equivalent entities through the same type of relationship, and the equivalent entities of science and technology patents tend to share similar or even the same attributes. GCN allows end-to-end learning of predictive pipes with arbitrary size and shape input. The input of GCN is the eigenvector of nodes and the structure of graph; the goal of GCN is to learn the characteristic function on the input graph and generate node-level output.

The graph convolution network part of this paper is composed of several stacked GCN layers. The input of layer \(l\) of the GCN model is the vertex characteristic matrix \(H^{(l)} \in R^{n \times d(l)}\), where \(n\) is the number of vertices and \(d(l)\) is the number of features in layer \(l\). Through the following convolution calculation, the output of layer \(l\) is the new characteristic matrix \(H^{(l)}\), as shown in equation (1): \[H^{(l+1)} = \sigma(\hat{D}^{-\frac{1}{2}}\hat{A}\hat{D}^{-\frac{1}{2}}H^{(l)}W^{(l)}).\] Where \(\sigma\) is the activation function, \(A\) is the connectivity matrix representing the structural information of the science and technology patent graph, \(D\) is the diagonal node degree matrix of \(A\), \(W\) is the weight matrix of the \(l\) layer of GCN, and \(d(l+1)\) is the dimension of the new fixed point feature.

MIFEA uses two 2-tier GCNs, each of which processes a graph of scientific and technological patent knowledge to generate the structural embedding of its scientific and technological patent entities. For example, two KGs are represented as \(G_1=(E_1,R_1,T_1)\) and \(G_2=(E_2,R_2,T_2)\); the corresponding GCN models are expressed as \(GCN_1\) and \(GCN_2\). For the structural feature vector of scientific and technological patent entities, set the dimension of the feature vector to \(d\) in all layers of \(GCN_1\) and \(GCN_2\). The two GCN models share the weight matrices \(W_s^{(1)}\) and \(W_s^{(2)}\) of the two-layer structure features. The final output of the two GCNs is the \(d\)-dimension embedding of the science and technology patent entities, which is further used to find the alignment of the science and technology patent entities.

In order to align a pair of science and technology patent entities, if the relationship and attributes of adjacent science and technology patent entities are used, noise may be introduced. It is a better choice to focus only on the relationship and attribute characteristics of current scientific and technological patent entities. Therefore, this paper proposes a hybrid multidimensional alignment network to better simulate these different features. The topological embedding \(H^{(l)}\) is obtained by using the \(l\) layer, and then the embedding of relations and attributes is obtained by using the feedforward neural network. The feedforward neural network consists of a full connection layer and a high-speed gate layer. The reason why the algorithm uses the high-speed gate layer is that the high-speed gate layer is generally better than the full-connection layer in terms of convergence speed and effectiveness. The feedforward neural network is defined as formula (2), (3) and (4): \[\begin{align} S_f &= \phi(W_f^{(1)}X_f + b_f^{(1)}),\\ T_f &= \sigma(W_f^{t}S_f + b_f^{t}),\\ G_f &= \phi(W_f^{(2)}S_f + b_f^{(2)})\cdot T_f + S_f\cdot(1-T_f). \end{align}\] Where \(f \in \{r,a\}\) and \(X_f\) represent a certain aspect (i.e. relationship or attribute) and its original characteristics, \(W_f^{(1,2,t)}\) and \(b_f^{(1,2,t)}\) are model parameters, \(\phi(\cdot)\) is ReLU function, and \(\sigma(\cdot)\) is sigmoid function. Based on this, the hybrid multi-aspect entity embedding \(H_y=[H_t^{(l)}\oplus G_r\oplus G_a]\) is obtained and further normalized.

Given two technical patent knowledge graphs \(G_1=(E_1,R_1,T_1)\) and \(G_2=(E_2,R_2,T_2)\) and a group of pre-aligned entity \(I(G_1,G_2)\) pairs as training data, the algorithm is trained in a supervised manner. In the training phase, the goal is to embed the cross-language technology patent entities into the same low-dimensional vector space, where the equivalent technology patent entities are close to each other. The loss function is defined as equation (5): \[Loss=\sum_{(e_1,e_2)\in I}\sum_{(e'_1,e'_2)\in I'}[\rho(h_{e_1},h_{e_2})+\beta-\rho(h_{e'_1},h_{e'_2})].\]

3.3 Attribute Semantic Representation of Science and Technology Patent Entities↩︎

Because different science and technology patent knowledge maps provide text descriptions of entities expressed in different languages, and contain detailed semantic information about entities. The key of attribute semantic representation module is to judge whether the literal description of the equivalent entity of scientific and technological patent is semantically close. However, it is very difficult to directly measure the semantic relevance of the descriptions of two science and technology patent entities, because they are expressed in different languages. Based on the BERT model, MIFEA maps words or sentences in different languages into the same semantic space to bridge the gap between different language descriptions.

MIFEA follows the basic design of BERT, and converts the task of science and technology patent entity to its text matching task. For the two entities \(e_1\) and \(e_2\) of two science and technology patent knowledge graphs from different data sources, the descriptions in this paper are \(t_1\) and \(t_2\), which are respectively composed of word sequences in different languages. The attribute semantic representation module consists of two components, each of which takes the description of an entity (from the source language or target language) as input. The input is designed as the format of BERT input, and then fed to the attribute semantic representation for context encoding.

By constructing training data \(D=\{(e,e'^{+},e'^{-})\}\), each triplet \((e,e'^{+},e'^{-})\in D\) contains a query technology patent entity \(e\in E\), a correctly aligned entity \(e'^{+}\in E'\) and a negative aligned entity \(e'^{-}\in E'\) randomly selected from \(E'\). For each science and technology patent entity in the data set, the pre-trained multi-language BERT accepts its name and description as input, and the CLS of BERT that has been filtered by MLP is embedded into the formula (6): \[C(e)=MLP(CLS(e)).\] The marginal loss of two pairs is used as the loss function of training, as shown in equation (7): \[L=\sum_{(e,e'^{+},e'^{-})\in D}\max\{0,g(e,e^{-})-g(e,e^{+})+m\}.\]

3.4 Technology Patent Information Fusion↩︎

This paper uses multiple information fusion to align scientific and technological patent entities, such as entity labels, topology information, relationship types, attributes and text descriptions. The graph convolution network is used to embed the structure information of scientific and technological patents, while adding various information. The topological information, relationship and attribute information are used as the input of the graph convolution network at the same time, not only as the external auxiliary information. Finally, the text description information of the science and technology patent entity is used for BERT embedding learning, and the structure embedding and text description information are fused to prepare for the subsequent alignment of the science and technology patent entity.

The first two modules of MIFEA extract the embedding representation from the graph structure of the science and technology patent knowledge graph and the attribute text description of the science and technology patent entity, that is, graph and text embedding. The multi-information fusion module will integrate the two kinds of embedding, and then align the scientific and technological patent entities. Use multi-layer GCN to obtain the topology information of science and technology patents, and use the full connection layer and high-speed gate mechanism to embed the entity relationship and attributes of science and technology patents. The technology patent text embedded by BERT is represented as \(T^B\), and the technology patent knowledge graph embedded by graph convolution network is represented as \(T^G\), which is combined based on the weight mechanism, and the two embedded weights are added as the entity embedding, as shown in formula (8): \[T^C=\tau\cdot T^G\oplus(1-\tau)\cdot T^B.\]

3.5 Alignment of Science and Technology Patent Entities↩︎

After obtaining the embedding of scientific and technological patent entities, cosine similarity is used to measure the distance between candidate entity pairs. The small distance reflects the high probability of entity alignment as an equivalent entity. All candidates are ranked for evaluation.

MIFEA takes the connection of the two descriptions of each candidate entity pair as the input to process each possible pair in the training set. At the same time, in order to reduce the calculation cost, it ignores the candidate pairs with low alignment probability. Sampling based on uncertainty can provide additional ranking improvement. MIFEA uses the model based on graph convolution network to generate a candidate pool whose size is far smaller than the entire entity pool. The model based on graph convolution network provides each source entity with the first \(Q\) candidate objects of the target entity (where \(Q\) is a super parameter). MIFEA generates a ranking score for each candidate entity pair in the pool to further rank the candidates.

4 Experiment of MIFEA↩︎

4.1 Data Sets and Evaluation Indicators↩︎

This paper uses the cross-language data sets DBP15K and DBP100K as the data sets. This paper also uses the technology patent data sets, including patent data in both Chinese and English, with a total of 160000 entries. DBP15K and DBP100K are large knowledge graphs that extract structured knowledge from Wikipedia in multiple languages. DBP15K and DBP100K contain three subsets: ZH-EN, JA-EN and FR-EN. Each subset contains information such as entity, relationship, attribute, relationship triplet and attribute triplet.

The evaluation index is Hits@\(K\). It refers to the proportion of correct entities in the top \(k\) of the results. The greater the Hits@\(K\), the better the effect. This paper compares MIFEA with MTransE, JAPE, BootEA, TransEdge and AttrGNN.

4.2 Results and Analysis of Main Experiment↩︎

Based on the general data sets DBP15K, DBP100K and scientific and technological patent data sets, the experimental results and algorithm efficiency of different entity alignment algorithms are compared and analyzed. In this paper, the same segmentation settings are used in the experiment, in which 30% of the data is used for training and the remaining 70% is used for evaluation. Hits@\(K\) is used as an evaluation index; this method measures the proportion of correctly aligned entities in the top \(k\) candidate objects.

From Table 1 and Table 2, it can be seen that the MTransE model has the worst effect. From Table 2, we can see that because MTransE only uses the structural information of the knowledge graph and uses TransE to construct a relational triplet, it lacks consideration of the overall structural information of the knowledge graph, thus losing some structural information, so the alignment effect of MTransE experiment is poor. Based on MTransE, JAPE integrates attribute information for entity alignment. Hits@1 and Hits@10 are 47% and 11% higher than MTransE, respectively. BootEA adds bootstrap training strategy on the basis of TransE, and its effect is also slightly higher than MTransE model. TransEdge is a good model based on the TransE embedding method. The main reason is that TransEdge not only embeds the vector of the head and tail entities of the relationship triple, but also adds the relationship embedding between entities to better represent the knowledge graph. Hits@1 and Hits@10 are 57% and 11% higher than BootEA, respectively.

It can be seen from Table 2 that AttGNN performs well in the experiment and reaches the best in the FR-EN dataset Hits@10. This is 1.3% higher than the MIFEA proposed in this paper. The main reason is that AttGNN is a model based on the GNN embedding method. This embedding method recursively aggregates the eigenvectors of neighbor nodes, which can take account of attribute information and structure information, and make good use of the data information in the knowledge graph.

The algorithm MIFEA proposed in this paper is optimal on ZH-EN and FR-EN data sets Hits@1 and Hits@10. Compared with AttGNN, ZH-EN increased by 2.8% and 2.2% respectively. The main reason is that MIFEA uses the description information of entities, and uses BERT to train the description information. Through the description information, the semantic information of entities can be well obtained, so as to better distinguish. For example, the two language syntax of FR-EN dataset is similar, so MIFEA performs best in this dataset.

4.3 Results and Analysis of MIFEA Ablation Experiment↩︎

This paper carries out ablation experiments on the proposed algorithm to verify the effectiveness of each technology patent knowledge graph information component. Select to remove the features of different aspects of the structure semantic representation module (SE), namely relations, attributes and high-speed gate mechanism. These variants are represented as \(SE_{w/oRE}\), \(SE_{w/oAE}\) and \(SE_{w/oHW}\) respectively.

The experimental results are shown in Table 3. It can be seen that the performance of the data set is reduced after deleting the relationship or attribute feature. Therefore, when MIFEA deletes topological features, the algorithm loses this structural knowledge, so its results are worse. From these experiments, we can conclude that topology information plays an indispensable role in making alignment decisions. When the high-speed gate mechanism is deleted, the effect of the algorithm decreases the most, which shows that the high-speed gate mechanism plays an important role in the convergence speed and effectiveness of the neural model used for entity matching.

4.4 Operation Efficiency Analysis↩︎

In order to compare and analyze the efficiency of MIFEA methods, this paper will conduct experiments on different algorithms on a unified platform based on scientific and technological patent data sets. Each algorithm is run five times on the ZH-EN data set based on the scientific and technological patent data set to obtain the average running time. The efficiency of different algorithms is compared and analyzed through the running results.

The result of algorithm running time is shown in Figure 1. From the figure, we can see that the RNM algorithm is efficient, and its running time is only 11.9% of the JAPE running time. It achieves good results with less space and less time. This shows that the subgraph matching based on relationship and the probability calculation of adjacent entities are very suitable for the entity alignment task of knowledge graph. The running time of JAPE algorithm is the longest, which indicates that the efficiency of the embedding method based on TransE is not high, and the fusion of attribute information also increases the complexity of the algorithm, thus increasing the running time.

The algorithm MIFEA proposed in this paper also has a long running time, but as an information fusion algorithm, the efficiency of MIFEA is 6.4% higher than that of JAPE. The algorithm uses BERT’s model to obtain the deep semantic representation of attribute description information. Although it can effectively improve the entity alignment effect, the universality and fine-tuning time of the model need to be improved.

4.5 Proportion Analysis of Training Set↩︎

In order to study how the size of the training set affects the results of the algorithm, different numbers of pre-aligned entities are used as training data to further compare MIFEA with JAPE, TransEdge and AttGNN. The experimental results show that the more pre-aligned entities used, the better the results of the algorithm.

Pre-aligned entities of different proportions are used as training data, with a range of 10% to 50%. All remaining pre-aligned entities are used for testing. Figure 2 shows the results of three methods in two data sets Hits@1. The results show that with the increase of training data, these three methods perform better. In the ZH-EN dataset, MIFEA is always better than the other two algorithms. The higher the proportion of training data, the smaller the gap between the results of the algorithms. In JA-EN dataset, MIFEA is always better than TransEdge, and the result is close to AttGNN. Due to some differences in the semantics of Japanese and Chinese, the BERT pre-training model adopted by MIFEA has not fully played its role, and the embedded representation of attribute semantics has not shown advantages, so the results are not better than those of AttGNN.

5 Conclusion↩︎

This paper proposes a method of entity alignment of science and technology patents based on graph convolution network and information fusion, so as to find the mapping of equivalent entities in the knowledge graph of science and technology patents. A method based on graph convolution network and BERT is proposed to study how to make better use of various information of entities provided in scientific and technological patent data, including topological connection, relationship, attribute, entity label and text description. The graph convolution network is used to embed the structure information while fusing various information. A multi-level alignment method is proposed, which uses topology information, relationship and attribute information as the input of the graph convolution network at the same time. The text description information is used to realize the deep semantic representation by BERT, and the entity alignment is achieved by integrating the graph structure embedded representation and the text semantic representation. The experimental results show that the method proposed in this paper achieves the best performance on the scientific and technological patent data set. With the increasing complexity and scale of scientific and technological patent data sources, it is necessary to consider the implementation efficiency and accuracy of the method.

References↩︎