1 Introduction↩︎

Code intelligence that utilizes machine learning techniques to promote the productivity of software developers, has attracted increasing interest in both communities of software engineering and artificial intelligence [1]–[5]. To achieve code intelligence, one fundamental task is code representation learning (also known as code embedding), which aims to preserve the semantics of source code in distributed vectors [6]. It can support various downstream tasks about code intelligence, including code defect detection [7]–[9], code summarization [10], code retrieval [11], and code clone detection [12].

Current approaches to code representation borrow ideas from the successful deep learning methods in natural language processing, mainly attributed to the naturalness hypothesis in source code [19]. From our investigation, existing approaches mainly represent the source code from different views of code, including code token in plain text [20], Abstract Syntax Tree (AST) [21], and Control/Data Flow Graphs (CFGs/DFGs) of code [22], [23]. Recently, many attempts have been made to pre-train a masked language model for source code, such as CodeBERT [2], GraphCodeBERT [13], SynCoBERT [14], CodeGPT [1], PLBART [15], CoTexT [17], and CodeT5 [18]. Table 1 shows the contribution of our work when compared with current pre-trained language models for source code.

Despite much progress in code representation learning, most of them only consider a single view of source code independently, ignoring the consistency among different views [1], [2], [15], [18]. Usually, a program, accompanied by a corresponding natural-language comment (NL), can be parsed into multiple views, e.g., the source code tokens, AST, and CFG. We argue that these different views contain complementary semantics of the program. For example, the source code tokens (e.g., method name identifiers) and natural-language comments always reveal the lexical semantics of code, while the intermediate structures of code (e.g., AST and CFG) always reveal the syntactic and executive information of code. In addition, a program can also be transformed (or rewritten) into different variants that have equivalent functionality. We think that different variants of the same program reveal the functional information of code. That is, those different program variants with the same functionality are expected to represent the same semantics.

Inspired by the aforementioned insights, this paper proposes a novel Code-MVP for code representation, which aims to integrate multiple views of the code into a unified framework with multi-view contrastive pre-training. Concretely, we first extract multiple views of code using several compiler tools, and learn the complementary information among them under a multi-view contrastive learning framework. Meanwhile, inspired by the type checking in compilation process, we also introduce fine-grained type inference as an auxiliary task in the pre-training process to encourage the model to learn more fine-grained type information.

To summarize, the contributions of this paper are two-fold: (1) We are the first to represent source code from multiple views, including the code tokens, AST, CFG, and various program equivalents, under a unified multi-view contrastive pre-training framework. Meanwhile, we also introduce an auxiliary task of inferring type annotations for variables. (2) We extensively evaluate Code-MVP on three program comprehension tasks. Experimental results demonstrate the superiority of Code-MVP when compared with several state-of-the-art baselines. Specifically, Code-MVP achieves 2.4/2.3/1.1 gain on MRR/MAP/Accuracy metrics in natural language code retrieval, code similarity, and code defect detection tasks, respectively.

2 Multiple Views of Code↩︎

We borrow ideas from the way that computers process the source code in compilation, where a program would be converted into multiple views. Figure 1 shows the process of converting a program from source code to machine code. During this process, the compiler would automatically utilize some program analysis techniques to verify the correctness of source code, including lexical, syntax, and semantic analyses. In the lexical analysis, a program is treated as a sequence of tokens and checked for spelling problems. In the syntax analysis, syntactic rules of programs are defined by the context-free grammar [24]. Then the program could be parsed as an AST, based on which many program transformation heuristics can be applied to rewrite the program while maintaining the same desired functionality. In the semantic analysis, semantic rules of the program are defined by the attribute grammar [25]. Then the compiler could check the types of code tokens, and a decorated AST could be obtained. After the three stages above, a translator will convert the source code to its Intermediate Representation (IR), which is then considered as the basis for building Control/Data Flow Graphs (CFGs/DFGs) for further optimizations in the static analysis. Finally, the IR of the source code should be converted into machine code to execute through a code generator. Next, we introduce how we extract different views of the source code. Figure 2 illustrates multiple views of source code with an example.

2.0.0.1 Abstract Syntax Tree (AST).

An AST, which is composed of leaf nodes, non-leaf nodes and edges between them, contains rich syntactic structural information of source code. In the AST, an assignment statement y = 0 can be represented by a non-leaf node assignment that points to three leaf nodes (0, y, and =). In this paper, we parse a snippet of source code into an AST using a standard compiler tool tree-sitter.¹. To feed an AST into our model, we apply depth-first traversal to convert it into a sequence of AST tokens [26].

2.0.0.2 Control Flow Graph (CFG).

CFG, which represents the execution semantics of the program in the form of a graph, is one intermediate representation of programs. A CFG consists of basic blocks and directed edges between them, where each directed edge reflects the execution order of the two basic blocks in the program. We can easily traverse the CFG along directed edges to parse it into a token sequence, which reveals the execution semantics of the program. In this paper, we use a static analyzer Scalpel² [27] to construct the CFGs for Python code snippets.

2.0.0.3 Program Transformation (PT).

The program transformation operations aim to produce multiple variants for a given program that satisfy the same desired functionality [28]. These different variants of a program can help the model capture functional semantics. In this work, we employ the following program transformation heuristics on ASTs and rewrite one program into another equivalent variant.

• Function and Variable Renaming. We randomly take new names from a set of candidates, such as VAR_i, FUNC_i, to rename the names of variables and functions in a program. This heuristic will not change the AST structure of the program, except for the textual appearance of variable and function names in the AST.

• Loop Exchange. The for and while loops represent the same functionality in a program. We traverse the AST to identify the for and while loop nodes, and replace for loops with while loops or vice versa. We also modify the initialization, condition and afterthought simultaneously.

• Dead Code Insertion. We first traverse the AST to identify several basic blocks [29], and then randomly select a basic block and insert dead code snippets into it. Note that the dead code snippets are predefined and selected from a set of candidates.

3 Code-MVP↩︎

3.1 Tasks and Notations↩︎

We define the set of program samples in multiple views (i.e. NL, PL, AST, CFG, PT) as \(S = \{S^1, \ldots, S^{m} \}\), where \(m\) represents the number of views, \(s_i^a\in S^{a}\) represents a program in the view of \(a\). Given a program, the PL view denotes its textual appearance, the NL view denotes its corresponding natural-language comment, and the PT denotes the variants of this program based on program transformation. The AST and CFG are extracted from a program using several compiler tools. Code-MVP adopts two forms of input, i.e., single-view input \(x_i^a=\{{\texttt{<CLS>}}, s_i^a \}\) and dual-view input \(x_i^{ab}=\{{\texttt{<CLS>}}, s_i^a,{\texttt{<SEP>}}, s_i^b\}\), where \(a\) and \(b\) denote two different views of the program. Following [30], a special token \(\texttt{<CLS>}\) is appended at the beginning of each input sequence, and \(\texttt{<SEP>}\) is used to concatenate two sequences. Subsequently, the representation of \(\texttt{<CLS>}\) is used to represent the entire sequence, and \(\texttt{<SEP>}\) is used to split two views of sub-sequences. Given a set of programs with their corresponding multiple views, we aim to learn the code representation by utilizing the mutual information existing in different views. Our intuition is to learn complementary information from multiple views of code by pulling the code under different views together and pushing the dissimilar ones apart.

3.2 Framework Overview↩︎

Figure 3 shows a simple example of our multi-view contrastive pre-training framework. Given a program \(s_i\), we use the same program to construct a pair of positive samples (\(x_i^a = \{{\texttt{<CLS>}}, s_i^a\}\) vs \(x_i^b = \{{\texttt{<CLS>}}, s_i^b\}\)) in the form of views \(a\) and \(b\), as described above. We take \(x_i^a\) and \(x_i^b\) as the input of Code-MVP respectively. The last hidden representations of <CLS> tokens in the two inputs can be formulated as \(\boldsymbol{h}_i^a = \mathrm{\small Code-MVP}(x_i^a)\) and \(\boldsymbol{h}_i^b = \mathrm{\small Code-MVP}(x_i^b)\). We utilize a projection head (a two-layer MLP) to map hidden representations to a space, i.e., \(\boldsymbol{v}_i^a = f(\boldsymbol{h}_i^a)\), \(\boldsymbol{v}_i^b = f(\boldsymbol{h}_i^b)\). Then the multi-view contrastive objective can be performed. During the pre-training process, we also design other two pre-training tasks, i.e., fined-grained type inference (FGTI) task and multi-view masked language modeling (MMLM).

3.3 Multi-View Contrastive Learning↩︎

We train Code-MVP with paired data and unpaired data. Paired data refers to those program samples with paired NL, while unpaired data stands for those isolated program samples without paired NL. Next, we explain how we construct positive and negative samples for these two cases.

3.3.0.1 Multi-View Positive Sampling.

We design Single-View (for paired and unpaired data) and Dual-View (for paired data only, which needs the NL) methods to construct multi-view positive samples for the MVCL objective:

• Single-View. To bridge the gap between different views of a same program, we consider the view of a program \(x_i^a\) as a positive sample w.r.t another view \(x_i^b\). That is, (\(x_i^a = \{{\texttt{<CLS>}}, s_i^a\}\) vs \(x_i^b = \{{\texttt{<CLS>}}, s_i^b\}\)) forms an inter-view positive pair, since \(x_i^a\) and \(x_i^b\) are two different views of a same program \(x_i\).

• Dual-View. There are a total of \(C_{m}^{2}\) combinations for two views of a same program. For efficiency, we focus on the features of the program itself, and propose the NL-conditional dual-view contrastive pre-training strategy, freezing the position of NL. Concretely, we construct a NL-conditional inter-view positive pair by replacing the second view in the input \(\{{\texttt{<CLS>}}, s_i^{\rm NL},{\texttt{<SEP>}}, s_i^a\}\) to be \(\{{\texttt{<CLS>}}, s_i^{\rm NL},{\texttt{<SEP>}}, s_i^b\}\), where \(\forall {a,b \neq {\rm NL}}\).

It is worth mentioning that there are many combinations to construct positive pairs. Some combinations are not considered in this work, such as the AST vs PT of the same program, and the CFG vs PT of the same program. Simultaneously, for training efficiency and downstream applications, we comprehensively consider eight combinations. They are (1) single-view: (NL vs PL), (NL vs PT), (PL vs AST), (PL vs CFG), and (PL vs PT); and (2) dual-view: (NL-PL vs NL-AST), (NL-PL vs NL-CFG), and (NL-PL vs NL-PT).

3.3.0.2 Multi-View Negative Sampling.

Since the processes of unpaired data and paired data are similar, here we take the unpaired data as an example. We leverage in mini-batch and cross mini-batch sampling strategies [31] to construct intra-view and inter-view negative samples, respectively. Given a mini-batch of training data \(b_1 = [ x_1^a, \ldots, x_n^a ]\) in the view of \(a\) with size \(n\), we can easily get another positive mini-batch data \(b_2 = [ {x_1^b},\ldots, {x_n^b}]\) in the view of \(b\), where (\(x^a_i\) vs \(x_i^b\)) denotes an inter-view positive pair. For \(x_i^a\), the intra-view negative samples are \(\{x_j^a\}, \forall {i \neq j}\), and the inter-view negative samples are \(\{{x_j^b}\}, \forall {i \neq j}\). Finally, for each \(x_i\), we can get a set of \(2n-2\) negative samples.

For an input \(x_i^a\) with representation \(v_i^a\) under the view of \(a\), it has one positive sample \(x_i^b\) with representation \(v_i^b\) under the view of \(b\). It also has a negative sample set \(\mathbf{V^-} = \{\boldsymbol{v}_1^-, \ldots, \boldsymbol{v}_{2n-2}^-\}\) with size \(2n-2\), which consists of two types of negative sample subsets, e.g., intra-view negative sample set \(\mathbf{V_1^-}\) with size \(n-1\), where \(\boldsymbol{v}_j^a \in \mathbf{V_1^-}, \forall {j \neq i}\), and the inter-view negative sample set \(\mathbf{V_2^-}\) with size \(n-1\), where \(\boldsymbol{v}_j^b \in \mathbf{V_2^-}, \forall {j \neq i}\). We define the similarity of a pair of samples as the dot product of their representations. Then the loss function for a positive pair \((x_i^a, x_i^b)\) can be defined as: \[\small \label{eq:cl} l(x_i^a, x_i^b)\!=\!- {\rm ln} \frac{{\rm exp}(\boldsymbol{v}_i^a \cdot \boldsymbol{v}_i^b)}{ {\rm exp}(\boldsymbol{v}_i^a \cdot \boldsymbol{v}_i^b)\! +\! \sum_{k=1}^{2n-2}{\rm exp}(\boldsymbol{v}_i^a \cdot \boldsymbol{v}_k^-) }\,.\!\tag{1}\] We calculate the loss for the same pair twice with order switched, i.e., \((x_i^a, x_i^b)\) is changed to \((x_i^b, x_i^a)\) as the dot product with negative samples for \(x_i^a\) and \(x_i^b\) are different. Overall, the MVCL loss function is defined as follows: \[\label{eq:batch95cl} \mathcal{L}_{\rm MVCL}\!=\!-\frac{1}{|\mathcal{N}|} \sum^{|\mathcal{N}|}_i \left[ l(x_i^a,x_i^b)\!+\!l(x_i^b,x_i^a) \right]\,,\!\tag{2}\] where \(\mathcal{N}\) denotes the set of all program samples covering all different views.

3.4 Pre-Training with Type Inference↩︎

Figure 4 shows the other two pre-training tasks, including fine-grained type inference and multi-view masked language modeling.

3.4.0.1 Fine-Grained Type Inference.

Several previous works [14], [18] have proven the importance of symbolic properties in programming languages. Two concurrent works, SynCoBERT [14] and CodeT5 [18] let the model divide the code token types into identifier or non-identifier. Inspired by the type checking in compilation process, we propose a fine-grained type inference (FGTI) objective to capture the fine-grained type information of variables [27], [32]. First, we parse all source codes into ASTs. Then, we traverse the AST and use the type checker to obtain fine-grained identifier types. We employ BPE tokenizer [33] to tokenize tokens and let sub-tokens inherit the type information of the token. Finally, we define the loss function as follows: \[\label{eq:fgtp} \mathcal{L}_{\rm FGTI} = -\frac{1}{|\mathcal{Z}|} \sum^{|\mathcal{Z}|}_i \sum^{|\mathcal{T}|}_j \;Y_{ij}\;{\rm log}\;P_{ij}\,,\tag{3}\] where \(\mathcal{Z}\) denotes the set of all tokens that need to inference types, \(\mathcal{T}\) represents the set of all types contained in the pre-training corpus, \(Y_{ij}\) denotes the label of token \(i\) in type \(j\), and \(P_{ij}\) denotes the predicted probability of token \(i\) in type \(j\).

3.4.0.2 Multi-View Masked Language Modeling.

In addition to the multi-view contrastive learning objective and fine-grained type inference objective, we also extend the Masked Language Modeling (MLM) to the multi-view program corpus, named MMLM. Given a data point \(x\), we randomly select 15% of tokens in \(x\) and replace them with a special token \(\texttt{<MASK>}\), following the same settings in [30]. The MMLM objective aims to predict original tokens which are masked out. We calculate the MMLM loss as follows: \[\label{eq:mmlm} \mathcal{L}_{\rm MMLM} = -\frac{1}{|\mathcal{M}|} \sum^{|\mathcal{M}|}_i \sum^{|\mathcal{V}|}_j \;Y_{ij}\;{\rm log}\;P_{ij}\,,\tag{4}\] where \(\mathcal{M}\) denotes the set of masked tokens, \(\mathcal{V}\) represents the vocabulary, \(Y_{ij}\) denotes the label of the masked token \(i\) in class \(j\), and \(P_{ij}\) denotes the predicted probability of token \(i\) in class \(j\).

3.5 Overall Training Objective↩︎

The overall loss function in Code-MVP is the integration of several components we have defined before. \[\mathcal{L} = \mathcal{L}_{\rm MVCL} + \mathcal{L}_{\rm FGTI} + \mathcal{L}_{\rm MMLM} + \lambda \lVert \Theta\rVert^2\,,\] where \(\Theta\) contains all trainable parameters of the model, and \(\lambda\) is the coefficient of \(L_2\) regularizer.

4 Experimental Setup↩︎

We conduct experiments to answer the following research questions: (1) How effective is Code-MVP compared with the state-of-the-art baselines? (2) How do different components and different views affect our Code-MVP?

4.1 Pre-Training Dataset and Settings↩︎

Different programming languages often require different program analyzers. Existing program analysis tools rarely support multiple programming languages and multi-view program transformations. For convenience, we choose Python for our experiments, as it is very popular and used in many projects. We pre-train Code-MVP on the Python corpus of CodeSearchNet dataset [34], which consists of 0.5M bimodal Python functions with their corresponding natural-language comments, as well as 1.1M unimodal Python functions.

Code-MVP is built on the top of Transformer [35], and consists of a 12-layer encoder with 768 hidden sizes and 12 attention heads. The pre-training procedure is conducted on 8 NVIDIA V100 GPUs for 600K steps, with each mini-batch containing 128 sequences up to 512 tokens including special tokens. According to the length distribution of samples in the training corpus, we set the lengths of PL/AST/CFG/PT in unpaired data to 512, and set the lengths of NL and PL/AST/CFG/PT in paired data to 96 and 416 respectively. The learning rate of Code-MVP is set to \(1e\textit{-}4\) with a linear warm up over the first 30K steps and a linear decay. Code-MVP is trained with a dropout rate of 0.1 on all layers and attention weights. We initialize the parameters of Code-MVP by GraphCodeBERT [13] and utilize a BPE tokenizer [33].

4.2 Evaluation Tasks, Datasets and Metrics↩︎

We select several program comprehension tasks to evaluate Code-MVP, including natural language code retrieval, code similarity, and code defect detection. We pre-train Code-MVP on Python corpus, and choose several public Python datasets to evaluate it, as shown in Table 2.

4.2.0.1 Natural Language Code Retrieval.

This task aims to find the most relevant code snippet from a collection of candidates, given a natural language query. We choose three datasets to evaluate this task, including AdvTest [1], CoNaLa [36], and CoSQA [37]. We adopt the Mean Reciprocal Rank (MRR) metric to evaluate the performance of code retrieval. In AdvTest dataset, we set the learning rate as \(5e\text{-}5\), the batch size as 32, the maximum fine-tuning epoch as 20, the maximum length of both query and code sequence as 256. In CoNaLa and CoSQA datasets, we set the learning rate as \(5e\text{-}5\), the batch size as 32, the maximum fine-tuning epoch as 30, the maximum length of query and code sequence as 128. In AdvTest and CoSQA datasets, we save the optimal checkpoint on the validation set, and test it on the testing set. In CoNaLa dataset, we report the best results on the testing set.

4.2.0.2 Code Similarity.

This task is always categorized into two groups: code-to-code retrieval and code clone detection. We conduct experiments on the Python800 dataset [38], which is composed of 800 problems with each problem having 300 unique Python solution files. We remove those files not in UTF-8 encoding formats and randomly select 100 solutions for each problem. In code-to-code retrieval, the filtered dataset is split to 720/40/40 problems for training, validation, and testing. Given a program, this task aims to retrieve other programs that solve the same problem; we evaluate using Mean Average Precision (MAP). Regarding the task of code clone detection, we treat it as binary classification and evaluate it using the Accuracy score, following [38].

To train these two tasks, we set the learning rate as \(2e\text{-}5\), the batch size as 32, the epoch number as 20. In code-to-code retrieval, we set the maximum length of both query and code sequence as 256. In code clone detection, we set the maximum concatenation sequence length of the two code snippets to 512. We save the optimal checkpoint on the validation set, and test it on the testing set.

4.2.0.3 Code Defect Detection.

This task aims to identify whether a given piece of code snippet is vulnerable or not, which is usually treated as a binary classification task. We evaluate all models on the GREAT dataset [39], which is originally built from the ETH Py150 dataset [40]. We evaluate the performance of code defect detection using the Accuracy score. We randomly select 100K samples for training, 5K samples for validation and 5K samples for testing, respectively. We set the learning rate as \(5e\text{-}5\), the batch size as 32, the maximum fine-tuning epoch as 50, the maximum length of both query and code sequence as 256. We save the optimal checkpoint on the validation set, and test it on the testing set.

4.3 Baselines↩︎

We compare Code-MVP with various state-of-the-art models. RoBERTa [41] is a robustly optimized BERT [30], which is originally pre-trained on a large-scale natural-language corpus. We fine-tune it on source code datasets of downstream tasks. CodeBERT [2] is pre-trained on NL-PL pairs using both masked language modeling [30] and replaced token detection [42] objectives. GraphCodeBERT [13] is a pre-trained language model of source code which incorporates the data flow information of source code. PLBART [15] is based on the BART [43] architecture and pre-trained on Python and Java functions using denoising autoencoding. CodeT5 [18] is based on the T5 [44] architecture and employs denoising sequence-to-sequence pre-training on seven programming languages. SynCoBERT [14] incorporates AST by edge prediction and uses contrastive learning to maximize the mutual information among programs, documents, and ASTs.

5 Results and Analysis↩︎

5.1 Performance on Downstream Tasks (RQ1)↩︎

5.1.0.1 Natural Language Code Retrieval.

Table 3 shows the results of natural language code retrieval on three datasets. We can observe that Code-MVP outperforms all baseline models on all datasets. Specifically, it outperforms CodeT5 by 3.8 points on average. Compared to the previous state-of-the-art SynCoBERT, Code-MVP also performs better with an average improvement of 2.4 points. This significant performance improvement indicates that the code representation learned by Code-MVP preserves more code semantics. We attribute this improvement to our introduced multi-view contrastive pre-training strategy.

5.1.0.2 Code Similarity.

Table 4 presents the results for code similarity calculation, including code-to-code retrieval and code clone detection. We can see that Code-MVP significantly outperforms all baseline models on these two tasks. In the task of code-to-code retrieval, Code-MVP outperforms CodeT5 and SynCoBERT by 3.4 points and 2.3 points, respectively. In the task of code clone detection, Code-MVP achieves 1.5 and 1.3 points higher compared to GraphCodeBERT and SynCoBERT, respectively. These results show that Code-MVP can better identify those programs with the same semantics and distinguish those programs with different semantics.

5.1.0.3 Code Defect Detection.

Table 5 shows the experimental results of code defect detection. Code-MVP consistently outperforms all models. Specifically, it outperforms GraphCodeBERT and SynCoBERT by 1.8 and 1.1 points, respectively. These results indicate that Code-MVP can effectively preserve the semantics of programs, which is beneficial for code defect detection.

5.2 Ablation Study (RQ2)↩︎

We empirically study several simplified variants of Code-MVP to understand the contributions of each component, including the Multi-View Contrastive Learning (MVCL), Fine-Grained Type Inference (FGTI), Abstract Syntax Tree (AST), Program Transformation (PT), and Control Flow Graph (CFG). Taking the natural language code retrieval task as an example, Table 6 shows the experimental results of each variant on that task. The setting of w/o (MVCL, FGTI) indicates that these pre-training objectives are removed from Code-MVP respectively. The setting of w/o (AST, PT, CFG) indicates that different views of programs are removed from Code-MVP respectively. From Table 6, several meaningful observations can be drawn. (1) Both MVCL and FGTI effectively increase the performance, which confirms that the two proposed pre-training objectives can indeed improve the ability of the model for program comprehension. (2) Exploiting different views of programs can bring performance improvements to the model as arbitrarily discarding any view of programs degrades the performance. Additionally, the introduction of CFG brings more performance improvements, indicating the importance of execution information for program understanding.

6 Related Work↩︎

6.0.0.1 Pre-Trained Models for Source Code.

Benefiting from the strong power of pre-trained models in natural language processing [30], [41], [45]–[47], several recent works attempt to use the pre-training techniques on programs [48]. [49] proposed CuBERT which follows the architecture of BERT [30], and is pre-trained with a masked language modeling objective on a large-scale Python corpus. [2] proposed CodeBERT, which is pre-trained on NL-PL pairs in six programming languages, introducing the replaced token detection objective [42]. Furthermore, [13] proposed GraphCodeBERT, which incorporates the data flow of programs into the model pre-training process. [14] proposed SynCoBERT, which incorporates ASTs via edge prediction to enhance the structural information of programs. They also used contrastive learning to maximize the mutual information among programs, documents, and ASTs. [1] proposed CodeGPT for code completion, which is pre-trained using a unidirectional language modeling objective. [15] proposed PLBART based on BART [43], which is pre-trained on a large-scale corpus of Java and Python programs paired with their corresponding comments via denoising autoencoding. [18] proposed CodeT5 following the architecture of T5 [44]. It employs denoising sequence-to-sequence pre-training on seven programming languages. Recently, [50] conducted a thorough structural analysis aiming to provide an interpretation of pre-trained language models for source code (e.g., CodeBERT and GraphCodeBERT).

6.0.0.2 Program Analysis for Code Intelligence.

In addition to the lexical information of programs, many recent works attempt to leverage program analysis techniques to capture the structural and syntactic representations of programs [22]. [26] designed several strategies to feed the ASTs of programs into Transformer [35]. [51] proposed a graph matching network, which utilizes the CFG of the program to deal with the challenge of binary function similarity search. [52] proposed a deep graph matching and searching model based on graph neural networks [53]–[57] for code retrieval. They represented both natural language queries and code snippets based on the unified graph-structured data. [58] presented the program-derived semantic graph to capture the semantics of programs at multiple levels of abstraction. [59] presented inst2vec, which locally embeds individual statement in LLVM intermediate representations by processing a contextual flow graph with a context prediction objective [60].

6.0.0.3 Contrastive Learning on Programs.

Recently, several attempts have been made to leverage contrastive learning for better code semantics. ContraCode [61] and Corder [62] first utilized semantic-preserving program transformations such as identifier renaming, dead code insertion, to build positive instances. Then a contrastive learning objective is designed to maximize the mutual information among the positive and negative instances. [63] presented a self-supervised pre-training technique called BOOST based on contrastive learning. They inject real-world bugs to build hard negative pairs. In Code-MVP, we construct the positive pairs throughout the compilation process of programs, including lexical analysis, syntax analysis, semantic analysis, and static analysis. It is the first pre-trained model that integrates multi-views of programs for program comprehension.

7 Conclusion↩︎

In this paper, we have proposed Code-MVP, a novel approach to represent the source code with multi-view contrastive pre-training learning. We extract multiple code views with compiler tools and learn the complement among them under a contrastive learning framework. We also propose a fine-grained type inference task in the pre-training process. Comprehensive experiments on three downstream tasks over five datasets verify the effectiveness of Code-MVP when compared with several state-of-the-art baselines.

Acknowledgements↩︎

We would like to thank Gerasimos Lampouras and Ignacio Iacobacci from Huawei London Research Institute for their constructive comments on this paper. Jin Liu is supported by National Natural Science Foundation of China under Grant No. 61972290. Yao Wan is partially supported by National Natural Science Foundation of China under Grant No. 62102157. Hao Wu is supported by National Natural Science Foundation of China under Grant No. 61962061, and partially supported by Yunnan Provincial Foundation for Leaders of Disciplines in Science and Technology (202005AC160005).

References↩︎