1 Introduction↩︎

Source code is a formal language that describes the logic and behavior of a software system [6]. Source code can be written in different programming languages, domains, and styles, but it always has a common characteristic, i.e. it is structured as a graph, not a sequence. A graph is a mathematical object that consists of nodes and edges, where nodes represent entities and edges represent relations [7]. A graph can capture both the syntactic and semantic information of the source code, such as the tokens, statements, control flow, data flow, and program dependence to name a few. To realise that source code is a graph, not a sequence, we take a simple code task of identifying if a pair of source code given is a clone or not.

Code clone detection is the task of finding code fragments that are identical or similar in functionality, but may differ in syntax or structure. Code clone detection is important for software engineering, as it can help to improve code quality, reduce maintenance costs, and prevent bugs and vulnerabilities [1]. However, code clone detection is also challenging, as it requires a deep understanding of the semantics and logic of the code, which may vary across different programming languages, domains, and styles.

Existing methods for code clone detection can be broadly classified into two categories: sequence-based and graph-based. Sequence-based methods rely on textual similarity of the code, such as token sequences. Graph-based methods rely on structural similarity of the code, such as Abstract Syntax Tree (ASTs), or control flow graphs (CFGs) or Code Property Graphs (CPGs). Sequence-based methods are fast and scalable, but they may fail to detect clones that have different syntax or structure. Graph-based methods are more accurate and robust, but they may be slow and complex, especially for large-scale or cross-language code clone detection.

A python source code clone pair is presented in Listing [[motivational_expample_1], [motivational_expample_2]]. The two code snippets have the same semantic behavior: they print ‘A’ or ‘a’ depending on the case of the input. However, they differ in their syntactic forms. Further such examples can be viewed in Appendix 10.

S = input()

if S.isupper():
  print("A")
else:
  print("a")

alp=input()

if alp==alp.upper():
  print("A")
elif alp==alp.lower():
  print("a")

In this paper, we argue that source code is naturally a graph, not a sequence, and that graph-based methods are more suitable for code clone detection than sequence-based methods. We compare the performance of sequence-based and graph-based methods for code clone detection on two benchmark data-sets: BCB [4] and PoolC [5]. BCB is a data-set of Java code snippets where as PoolC is a data-set of Python code snippets. We use CodeBERT [2] as a representative sequence-based modelling approach, and CodeGraph [3] as a representative graph-based modeling approach. CodeBERT is a bimodal pre-trained model for programming language (PL) and natural language (NL) that learns general-purpose representations that support downstream NL-PL applications. CodeGraph is a graph-based model for semantic code clone detection based on a Siamese graph-matching network that uses attention mechanisms to learn code semantics from DFGs and CPGs.

We conduct various experiments to evaluate the accuracy, recall, precision, and F1-score of CodeBERT and CodeGraph on three experimental setups: (i) in-domain static source code analysis , (ii) cross-lingual generalization and semantic extraction, and (iii) zero-shot source code clone classification. We show that CodeGraph outperforms CodeBERT on all experimental setups in terms of all metrics. The main contributions of this paper are as follows:

• To best of our knowledge, we are the first one to demonstrate the superiority of graph-based methods over sequence-based methods for multilingual static source code analysis tasks, such as clone detection, by exploiting the natural graph structure of source code across programming languages.

• We provide novel insights on the generalization and cross-domain understanding of graph-based models, compared to sequence-based models, for source code analysis, as they leverage both the syntactic and semantic features of source code in various cross-domain settings.

• We show how mixing cross-lingual data-sets can improve the overall performance of the graph-based model by 4.5%, as it can learn from the commonalities and differences of different programming languages.

• We focus on the under-explored clone detection python data-set POOLC, along with the benchmark Java data-set BCB, and draw parallel comparisons on both of the data-sets.

• We provide highly efficient and scalable code for standard CPG representation generation using Tree-sitter [8] and Micrsoft’s DFG [9], along with the re-implemented code for the sequence and graph-based models.

The rest of this paper is organized as follows: Section 2 reviews the related work on source code clone detection. Section 3 introduces to the various code representations, like Abstract syntax tree, Data flow graph etc and the background knowledge on graph neural networks and natural language processing. Section 4 describes the details of how the experiments are carried out, and lays down the research questions. Section 5 presents the results and its discussion. Section 6 discusses the limitations and future research directions. Section 7 concludes the paper.

2 Related Work↩︎

2.1 What is static source code analysis?↩︎

Static code analysis is a valuable technique for improving software quality and security without actually compiling the code. It can find errors that are hard to detect at run time, improve the quality and maintainability of the code, and reduce the cost and time of testing and debugging. This is basically a form of white-box testing. According to [10] the cost of fixing a defect increases exponentially as it moves from the coding phase to the testing phase to the maintenance phase. Therefore, having a static tools to analyse and fix a source code as soon as possible helps save lot of resources and efforts.

2.2 What are applications of static code analysis?↩︎

There are various different applications for static code analysis. Security Vulnerability detection, is one of the major static code analysis, which can help developers identify and fix security vulnerabilities before they are exploited by the attackers as extensively stated by [11]. Another important area of static code analysis is in helping developers find inefficient algorithms and improve the resource utilization, response time, and other throughput of the software. Clone Detection is another application that can help identify similar functional code fragments which may indicate code duplication, plagiarism or reuse. This can improve quality, maintainability and the security of the code by eliminating redundant and inconsistent or outdated code [12].

2.3 What is source code clone detection?↩︎

Source code clone detection is the process of finding code fragments that have similar functionalities or structures, which could indicate that the code is a duplicate, plagiarised or reused, which may be done purposefully, negligently or accidentally by a developer as said by [12]. Clone detection is very harmful to the quality of the entire source code [12]. A broad categorization on various types of code clones is given by [12].

• Textual Similarity

  • Type I: Changes in White-spaces, comments, layouts.

  • Type II: Renaming of variable names, or changes in types and identifiers.

  • Type III: Addition or removal of statements.

• Functional Similarity

  • Type IV: Complete change in syntax, but functionally same behavior.

2.4 Types of Clone detection approaches?↩︎

According to [12] there are multiple techniques to detect a source code clones, like Text based, token based, tree based, Program dependency graph based, metrics based, or hybrid approaches. Here we deal and compare token based vs tree based clone detection approach. As we know that, to detection semantically same code clones, the model should not just rely on difference of syntax, but also understand the semantics of the structure. This becomes hard, if we pass the source code to the model as sequence rather than a Tree like Abstract Syntax Tree which naturally holds the syntatical information of a source code.

2.5 Sequence based modeling↩︎

There has been various sequence based modelling approaches used by source code clone detection like, CodeBERT [2], UNIXCODER [13], ContraBERT [14]. Here in sequence modeling the source code is tokenized as a piece of words (or source code). This tokenized pieces of words in a sequence is learnt by the model to understand a fragment of code. This helps the model learn the semantics, by taking the code in a sequential manner.

We use CodeBERT [2] which is a bimodal pre-trained model for programming language (PL) and natural language (NL) that learns general-purpose representations that support downstream NL-PL applications such as natural language code search, code documentation generation, etc1. CodeBERT is developed with a Transformer-based neural architecture, and is trained with a hybrid objective function that incorporates the pre-training task of replaced token detection, which is to detect plausible alternatives sampled from generators 1, along side with Masked Language modelling. In this study, we use CodeBERT as a pre-trained model for our sequence model for source code clone detection.

2.6 Graph based modeling↩︎

On the other side, clone detection as a graph modelling approach, we have models like TBCCD [15], FA-AST [16], HOLMES [17], DG-IVHFS [18], CodeGraph4CCDetector [3]. These types of graph models first construct a tree or a graph like, abstract syntax tree, Control flow graph etc from the source code. This helps to retain the structural information of the code, regardless of it being moved from its location or variables being replaced. This ideally should help the model concentrate more on the semantics, rather than the structural learning, as it is already baked into its structure.

We use CodeGraph4CCDetector [3] as our graph-based model, from here on referred as CodeGraph. This model is reported to have state of the art results on the BCB [4] data-set. This is a Siamese graph matching network which basically takes in two source code snippets and output a similarity score between them. The input for this is the Code Property Graph, which is essentially graph having various nodes and edges. This helps the network capture the source codes syntactical and semantical information. The node representation of this CodeGraph uses attention mechanism on a node level to extract out a node representation, before combining it to graph level representation. The major advantage of a graph level over the sequence level is, this can handle code snippets of different lengths and structures, as long as the hardware memory can load it.

3 Methods↩︎

This section describes the methods and models that we employ for our experiments on the sequence and graph representations of source code. We organize this section into four parts. First, we present how we use byte pair tokenizers to represent source code as a sequence of tokens. Second, we illustrate how we use code property graphs (CPGs) to represent source code as a graph. Third, we introduce CodeBERT [2], the sequence-based model that we employ for code clone detection. Fourth, we present CodeGraph [3], the graph-based model that we employ for code clone detection.

3.1 Code Tokenization↩︎

We apply the default Byte Pair Encoding, BPE tokenizer of CodeBERT [2] to represent the source code as a sequence of tokens. Figure 1b shows an example of how the BPE tokenizer splits the source code in Figure 1a into word and subword tokens.

3.2 Code Representations↩︎

We use a graph representation of source code that consists of nodes and edges connecting source code tokens. To generate this representation, we apply the following steps: First, we use a lexical parser, to produce the abstract syntax tree (AST) of the source code. Second, we extract the data flow information from the AST. Third, we merge the AST and DFG, to form one graph which we call it as Code Property Graph. This is further pruned and standardized across source code languages to make the standard CPG.

3.2.1 Abstract Syntax Tree (AST)↩︎

We use Tree-sitter [8], a lexical parser, to generate the abstract syntax tree (AST) of the source code for any language. We currently use it for Java and Python, but it supports 141 different languages. Figure 1c shows the AST generated from the sample source code in Figure 1a.

3.2.2 Data Flow Graph (DFG)↩︎

We use Microsoft’s Data Flow Generator (DFG) [9] to generate a data flow graph (DFG). This DFG generator takes the AST from the previous step and adds the data flow edges to it to form the DFG. Figure 1d shows the DFG graph for the same example source code in Figure 1a. We can see that there is a data flow edge between the integer literal ‘5’ and the variable ‘num’, as ‘5’ is assigned to ‘num’. There is also a data flow edge between the second occurrence of ‘num’ and the print statement, as ‘num’ is used as an argument.

3.2.3 Standard Code Property Graph (CPG)↩︎

We standardize the DFG to a code property graph (CPG), which is our final graph representation of source code. We perform two main operations to standardize the DFG across languages. First, we prune the graph from nodes that do not add value to the model’s understanding, such as opening and closing brackets that are implicitly understood when there is a method call. Second, we standardize the node type labels across languages so that the model can recognize them consistently across languages. For example, in Figure 1e we see that the root node ‘module’ and ‘integer’ node are standardized and replaced with ‘_program’ and ‘_integer’ as standard node types.

The major impact of a standard CPG can be seen in Figure 2, where two programs that print hello, name in Java and Python are written. The programs look different as raw code, but they have the same functionality and semantics. The standard CPGs look very similar in both cases, as shown in Figure 2c and 2d. We provide more examples of standard CPGs in Appendix 10. This supports our claim that this type of code representation is more suitable than the sequence of code for identifying code clones.

3.3 Sequence-based Model: CodeBERT↩︎

In order to model a sequence model for source code clone detection, we use CodeBERT [2] as a pre-trained model. We fine-tune CodeBERT on the source code clone detection labelled data-set. The fine tuning task is a binary classification task where the source code pair is passed sequentially through the CodeBERT which acts as an encoder, and the 2 representation vectors coming out from this encoder, is concatenated and passed to a shallow 2 layer MLP classifier to give the final output, if the pair is a clone or a no clone. The major advantage of using this state of the art encoder CodeBERT is that it can help capture both the syntactic and semantic information of the PL code, by leveraging the large-scale pre-training data of multiple languages.

3.4 Graph-based Model: CodeGraph↩︎

We use CodeGraph4CCDetector [3] as the graph based model for our source code clone detection, it is from here on refered to as CodeGraph model. This model initially is used by its authors on BCB data-set for its classification, and hence we keep the pipeline as it is. However, we trained our own word2vec model [19], using gensim [20] to keep it consistent with respect to the sequence model. We call this model as Code2Vec model, which helps to generate the source code token embedding for our source code. We train this Code2Vec model using the source tokens which are tokenized by the CodeBERT’s [2] tokenizer, which is a Byte Pair encoding tokenizer. This helps in two ways, Firstly, it helps to keep it consistent with the comparison of the sequence model, and secondly it helps to retain the word meanings of the human written source code variable names etc.

Once we get the tokenized vector format of each graph nodes using the Code2Vec model on every node of CPG, we then use the CodeGraph architecture as it is. Here similar to the sequence model, we pass the code pair sequentially to the CodeGraph model, which then generates the graph level representation. This representation is the taken to a shallow LSTM layer which helps to perform a binary classification on this graph level representation.

4 Experimental Design↩︎

Based on our proposed methodology, we conduct experimentation on the following research questions (RQs):

• RQ1: Will a graph-based model that leverages both structural and semantic information surpass a sequence-based model in an in-domain static source code analysis?

• RQ2: Will a graph-based model trained on multiple source code languages outperform a sequence-based model in cross-lingual generalization and semantic extraction?

• RQ3: Will a graph-based model excel over a sequence-based model in the domain adaptation of zero-shot source code clone classification?

4.1 Experiment Data↩︎

For our experimental setups, we perform clone detection on two publicly available data-sets. The first one is Big Clone Bench (BCB), which is a java language data-set that was originally introduced by [4]. We used the version of BCB that was filtered according to FA-AST [16]. BCB contains 9,134 java methods, which generate over 2M combinations of clone and non-clone code pairs. The second one is PoolC, which consists of over 6M python code snippets that were extracted from hugging face [5].

We applied various parameters to limit the data-set for the experimentation phase. We restricted the number of lines to be between 5 and 100, the maximum number of characters to be 2000, and the maximum number of nodes in the graph to be 100. The details of how the distribution changed before and after applying the thresholds are given in Appendix 11. Table 1 shows the summary of the average counts for the filtered files according to each data-set (BCB, PoolC, and their combination, Mix_1).

We randomly sampled pairs of clone and non-clone from the filtered files set to form the data-set. Table 2 summarizes the data-set pairs according to each data-set.

4.2 Experimental Setup↩︎

We chose the state-of-the-art sequence model and graph model, namely CodeBERT [2] and CodeGraph [3], respectively, to conduct various experiments. To answer the research questions, we designed the experiments around them as follows.

• Experiment 1: We train and evaluate Sequence and Graph models independently on each of the data-sets, namely BCB and PoolC, to compare their performance within the same domain as baselines.

• Experiment 2: We train and evaluate Sequence and Graph models on Mix_1 Data-set, which is a mixture of data from both domains, to examine their cross-domain learning and generalization capabilities.

• Experiment 3: We train Sequence and Graph models on BCB data-set and test them on PoolC data-set, and vice versa, to assess their cross-domain zero-shot performance.

4.3 Model Hyper-parameters↩︎

We use the same machines with Intel® Xeon® Gold 5222 and one Quadro RTX 6000 to train both the sequence and graph models (CodeBERT and CodeGraph, respectively) in order to maintain a consistent experimentation environment. The maximum batch size that CodeBERT can run on a single RTX 6000 is 16 code pairs, or 32 code snippets per batch. We also set the batch size of CodeGraph to the same value. The other hyper-parameters used for training these models are given in Appendix 12.

5 Results↩︎

5.1 Experiment 1↩︎

We train the sequence and graph models (CodeBERT and CodeGraph, respectively) on two datasets: BCB and PoolC. This leads to four model trainings and evaluations, as shown in Table ¿tbl:table:exp951? on page . We select the best-performing epochs for each model, which are the 3^rd epoch for CodeBERT and the 2^nd epoch for CodeGraph. We find that CodeGraph consistently outperforms CodeBERT on both datasets, demonstrating that CodeGraph has a better learning capability on the source code than CodeBERT under limited data and constrained environment conditions. We highlight statistically significant experimental results in the tables based on bootstrap testing [21] with p value below 0.05 for statistical significance, which compares CodeBERT and CodeGraph.

5.2 Experiment 2↩︎

We use the same model architectures from Experiment 1, but we train them on a cross-lingual data-set (Mix_1) that combines both the BCB and PoolC data sets. We then evaluate these models on the Mix_1 data-set as well as the individual BCB and PoolC data-sets. The results are shown in Table ¿tbl:table:exp952? on page .

The evaluation results on the Mix_1 dataset for both CodeBERT and CodeGraph are intermediate between the single-language models trained in Experiment 1. This is further confirmed by the evaluation results on the individual BCB and PoolC data-sets, where we observe that cross-lingual training improves the performance of CodeGraph on both data-sets, from 83.90 to 87.64 on the PoolC data-set and from 98.87 to 99.42 on the BCB data-set, indicating that CodeGraph generalizes better on the source code with cross-lingual training. On the other hand, we observe that cross-lingual training does not improve the performance of CodeBERT as much as CodeGraph, decreasing it by 0.55 on the BCB data-set and increasing it by only 0.19 on the PoolC data-set.

5.3 Experiment 3↩︎

We test the domain adaptation of the pre-trained models from Experiment 1, i.e., CodeBERT and CodeGraph, on a different source code language than the one they were trained on. For instance, we evaluate CodeBERT trained on BCB on PoolC, and vice versa. We repeat the same procedure with CodeGraph without changing the environment. The results of this experiment are shown in Table ¿tbl:table:exp953? on page .

This experiment simulates the domain adaptation from Python source code to Java source code and vice versa. The results show that CodeBERT performs very poorly on a different domain, with F1 scores of 33.71 and 36.56 for PoolC and BCB evaluation, respectively. This indicates that the model has over-fitted on the domain and cannot generalize well to a new domain. We observe the same trend with other epochs. On the other hand, CodeGraph performs much better than CodeBERT on a different domain, with F1 scores of 53.67 and 46.44 for PoolC and BCB evaluation, respectively. This demonstrates that CodeGraph has a better domain adaptation capability than CodeBERT in a zero-shot learning setting, although it does not achieve state-of-the-art performance. This suggests that representing source code as a graph rather than a sequence is a promising direction for future research.

5.4 Discussion↩︎

We analyze the false predictions made by both the models, CodeBERT and CodeGraph, and find that most of them are false positives, especially from CodeBERT. When we examine these examples from CodeBERT, we notice that the model predicts them as false positives with high confidence, whereas the CodeGraph model either predicts them as true negatives or false positives with low confidence. This indicates that adjusting the classification threshold for CodeGraph could improve its overall performance. However, for CodeBERT, we observe that the model is confused by the very similar keywords in the code pair. We provide a detailed analysis of this in Appendix 13.2.

We also analyze the false negatives for CodeGraph on the PoolC data-set, which are the most frequent among all the models and data-sets. We find that these false negatives are mainly due to the large size differences between the code pairs in the PoolC data-set. The examples we inspect are clones of type IV, but they have one code snippet much longer than the other. This makes it difficult for CodeGraph to recognize them as clones and it predicts them as non-clones instead. We provide some detailed explanation and examples of these false negatives in Appendix 13.3.

6 Limitations and Future Research↩︎

6.1 Limitations↩︎

We note the following limitations and concerns in the study:

• The experimental setup data-set size is drastically reduced in order to time-bound the experiments for this research project. Majorly, there are 2 cuts in the data-set size, Firstly, source code files are filtered to only those which have less than or equal to 100 nodes. Secondly, the sample size of the data-set clone and no clone pair is restricted to 50K data-points only. Refer the Appendix 11 to check the filtering criterion and Section 4.1 to understand the data-point split samples.

• The BCB data-set was significantly reduced after applying the thresholding criterion, resulting in only 2K files out of the original 9K. This led to a over-sampled distribution of the training pairs, which consisted of 50K data-points. As shown in the Results Section 5, this caused the BCB data-set model to over-fit the data, unlike the PoolC data-set model.

• A potential limitation of this study is the discrepancy in the number of trainable model parameters between the sequence model (CodeBERT) and the graph model (CodeGraph). The sequence model has 125M parameters, which is 125 times more than the graph model’s 1.1M parameters. This could raise the question of whether the graph model’s superior performance is due to its inherent advantages or its lower complexity. However, this also suggests that there is room for further improvement on the graph model by increasing its number of parameters.

6.2 Future Research↩︎

Some possible directions for the future research based on the limitations are as follows:

• To evaluate the impact of data-set size on the performance of the models, future research could use the complete and more diverse data-set that include source code files with more than 100 nodes and data-set samples itself going upwards of a million samples. This would help to test the generalizability and robustness of the models across different domains and languages at a larger scale.

• To help reduce over fitting problem on the Java data-set (i.e. BCB data-set), future research could use samples from other data-sets, like CodeForces codeForces?, Google Code Jam [22], which can yield in more diversified data-set for Java language.

• To explore the potential of the graph model (CodeGraph), future research could increase its number of trainable parameters and compare its performance with the sequence model (CodeBERT) under the same complexity level. This would help to determine whether the bigger graph model would still have inherent advantages over the sequence model or not. conversely, one can reduce the parameters on sequence model and check its impact.

• PoolC data-set false Negatives are majorly due to the code size length differences. This can be solved if trained with longer snippet of codes.

• Train a mixture model on various source code languages, not just limiting to two, such as JavaScript, SQL, HTML, etc., and evaluate its generalization ability on different domains together. Moreover, cross-domain example pairs could be generated from Code Forces codeForces?, which is an online platform for competitive programming that supports multiple languages.

7 Conclusion↩︎

In this paper, we have shown that graph-based methods are superior to sequence-based methods for source code clone detection. We have used the state-of-the-art models CodeBERT [2] and CodeGraph [3] to conduct various experiments on two benchmark data-sets: BCB [4] and PoolC [5]. We have demonstrated that graph models can better capture the structural and semantic information of the source code than sequence models in a series of 3 experimental setups, and that they can generalize better across different source code languages and domains. We have also provided efficient and scalable code for generating standard CPG representations of source code, along with the re-implemented code for the sequence and graph-based models. Our work has important implications for future research on source code analysis, as it suggests that representing source code as a graph rather than a sequence is a promising direction for enhancing the performance and generalization of static source code analysis models.

8 Acknowledgements↩︎

We are deeply grateful to our supervisor, Dr. Julia Ive, for her constant guidance, support, and feedback throughout this research project. She has inspired and motivated us with her expertise and experience, and we have learned a lot from her. We also thank Vishal Yadav, Mashhood Alam, and other peers as well as the reviewers for their valuable comments and suggestions that enhanced the quality of our paper. Moreover, we acknowledge the Queen Mary University and organizations that supported this work, as well as the authors of the data-sets and models that we used in our experiments. Lastly, we thank our families for their love and encouragement, and the almighty for his blessings and grace.

9 Data Availability↩︎

We have made the data and source code that we used in this paper publicly accessible. Source code is available for replication at: https://github.com/Ataago-AI/clone-detection, and the filtered data-sets can be downloaded from here: https://drive.google.com/drive/folders/1phx8k_JB8HC_HW3nhZLec9BKjRxNCN2b?usp=drive_link

10 Code Representations↩︎

10.1 Python Standard Code Property Graphs Example pairs↩︎

10.2 Java Standard Code Property Graphs Example pairs↩︎

11 Data-set↩︎

Data set is filtered based on various parameters like number of lines, number of characters, and number of nodes. Given below are the charts how the data looks before and after filtering.

11.1 Java Data : BCB↩︎

Given in Figure 7 are the original and filtered distributions for Java dataset from BigCloneBench BCB dataset [16].

11.2 Python Data : PoolC↩︎

Given in Figure 8 are the original 8 (a) and filtered 8 (c) distributions for Python dataset from PoolC dataset [5].

11.3 Java and Python Data : mix 1↩︎

Given in Figure 9 we have the distribution for the mixture of BCB and PoolC dataset. This dataset is made by randomly sampling the filtered datasets of BCB and PoolC examples from each of train, valid, and test splits. This results in total 19K files for source code from both Java and Python together, which results in total 25K Java and 25K python labelled pairs.

12 Training↩︎

12.1 Model hyper-parameters↩︎

12.2 CodeBERT : Sequence Model↩︎

12.3 CodeGraph : Graph Model↩︎

13 Result Analysis↩︎

13.1 Confusion Matrix↩︎

13.1.1 BCB Dataset↩︎

Given in Figure 12 are the confusion matrix for CodeBERT and CodeGraph models.

13.1.2 PoolC Dataset↩︎

Given in Figure 13 are the confusion matrix for CodeBERT and CodeGraph models.

13.2 False Positive Analysis↩︎

13.2.1 BCB Dataset↩︎

• There are 18 False positive from both the models combined. Here we observe that the prediction confidence from CodeBERT is very high above 0.9, where as CodeGraph has prediction confidence on a lower side at 0.5 to 0.6. As observed from Code Pair Listings [[code:1.1] & [code:1.2]], [[code:2.1] & [code:2.2]]. This suggest that adjusting the classification threshold for CodeGraph can help reduce the False positives which are common in both.

• The False Positives from CodeGraph, but True Negative from CodeBERT is seen to be consistently having less confidence, which is below 0.7. Although these False positives are only predicted by CodeGraph, and CodeBERT very strongly predicts them as True Negative. Examples of Code Pair Listing [[code:bcb:1300030] & [code:bcb:20955454]] following this trend.

• The False Positives from CodeBERT, but True Negative from CodeGraph, is seen to have a consistent prediction with confidence less than 0.9. This can be misleading as this is a higher confidence from CodeBERT. on the same side, CodeGraph doesn’t have very strong prediction either, but it is atleast consistently predicting them as TN. Example of Code Pair Listing [[code:bcb:5510183] & [code:bcb:9356670]]

public PhoneDurationsImpl(URL url) throws IOException {
    BufferedReader reader;
    String line;
    phoneDurations = new HashMap();
    reader = new BufferedReader(new InputStreamReader(url.openStream()));
    line = reader.readLine();
    while (line != null) {
        if (!line.startsWith("***")) {
            parseAndAdd(line);
        }
        line = reader.readLine();
    }
    reader.close();
}

public static String getMyGlobalIP() {
    try {
        URL url = new URL(IPSERVER);
        HttpURLConnection con = (HttpURLConnection) url.openConnection();
        BufferedReader in = new BufferedReader(new InputStreamReader(con.getInputStream()));
        String ip = in.readLine();
        in.close();
        con.disconnect();
        return ip;
    } catch (Exception e) {
        return null;
    }
}

public static LinkedList<String> read(URL url) throws IOException {
    LinkedList<String> data = new LinkedList<String>();
    HttpURLConnection con = (HttpURLConnection) url.openConnection();
    BufferedReader br = new BufferedReader(new InputStreamReader(con.getInputStream()));
    String input = "";
    while (true) {
        input = br.readLine();
        if (input == null) break;
        data.add(input);
    }
    br.close();
    return data;
}

protected Reader getText() throws IOException {
    BufferedReader br = new BufferedReader(new InputStreamReader(url.openStream()));
    String readLine;
    do {
        readLine = br.readLine();
    } while (readLine != null && readLine.indexOf("</table><br clear=all>") < 0);
    return br;
}

public PhoneDurationsImpl(URL url) throws IOException {

    BufferedReader reader;
    String line;
    phoneDurations = new HashMap();
    
    reader = new BufferedReader(new InputStreamReader(url.openStream()));
    line = reader.readLine();
    
    while (line != null) {
        if (!line.startsWith("***")) {
            parseAndAdd(line);
        }
        
        line = reader.readLine();
    }
    
    reader.close();
}

public void alterarQuestaoMultiplaEscolha(QuestaoMultiplaEscolha q) throws SQLException {
    PreparedStatement stmt = null;
    String sql = "UPDATE multipla_escolha SET texto=?, gabarito=? WHERE id_questao=?";
    try {
        for (Alternativa alternativa : q.getAlternativa()) {
            stmt = conexao.prepareStatement(sql);
            stmt.setString(1, alternativa.getTexto());
            stmt.setBoolean(2, alternativa.getGabarito());
            stmt.setInt(3, q.getIdQuestao());
            stmt.executeUpdate();
            conexao.commit();
        }
    } catch (SQLException e) {
        conexao.rollback();
        throw e;
    }
}

public static String getMyGlobalIP() {
    try {
        URL url = new URL(IPSERVER);
        HttpURLConnection con = (HttpURLConnection) url.openConnection();
        BufferedReader in = new BufferedReader(new InputStreamReader(con.getInputStream()));
        String ip = in.readLine();
        in.close();
        con.disconnect();
        return ip;
    } catch (Exception e) {
        return null;
    }
}

private FTPClient loginToSharedWorkspace() throws SocketException, IOException {
    FTPClient ftp = new FTPClient();
    ftp.connect(mSwarm.getHost(), mSharedWorkspacePort);
    if (!ftp.login(SHARED_WORKSPACE_LOGIN_NAME, mWorkspacePassword)) {
        throw new IOException("Unable to login to shared workspace.");
    }
    ftp.setFileType(FTPClient.BINARY_FILE_TYPE);
    return ftp;
}

13.2.2 PoolC Dataset↩︎

• There are 344 total False positives predicted by both the models combine. But here we observe that the confidence is following similar trend with BCB dataset, CodeBERT is having higher prediction confidence of Fasle positive, where as CodeGraph has a lower prediction confidence, with max being at 0.71. Here we see that the positive prediction is majorly coming from confusion of syntactically same keywords present in both, for example having extensive usage of if and for loops. This can be seen in the example Code Pair Listing [[code:poolc:5931] & [code:poolc:6679]].

• The False positive from CodeGraph, but True Negative from CodeBERT is seen to have consistently lower prediction score from CodeGraph, max being 0.78 and average being 0.58. This again suggests that the CodeGraph is understanding the semantics, and with a high classification threshold, should improve significantly. Here the rational behind why the Graph model seem to get them wrong, is it seems to confused on the syntactic structure of the codes. They might not have same keywords, but the structure syntactically is dominating. A code pair Listing [[code:poolc:4851] & [code:poolc:7175]].

• The false positives from CodeBERT, but True negatives from CodeGraph, seems to have stronger prediction of True negatives from Graph models, again showing the Graph models are superior to learn the structural and symatic information with an average score of 0.81. Looking at what might be going wrong with Sequence model would mostly be the keywords having similar names in sequence, but not syntactically similar, as observed in code pair Listing [[code:poolc:2388] & [code:poolc:1287]].

n=int(input())
x=list(map(int,input().split()))
m=10**15
for i in range(101):
    t=x[:]
    s=sum(list(map(lambda x:(x-i)**2,t)))
    m=min(m,s)
print(m)

a,b,c,d = map(int, input().split())

ans = -10**18+1
for i in [a,b]:
  for j in [c,d]:
    if ans < i*j: ans = i*j
print(ans)

import sys
x=int(input())

n=1
while(100*n<=x):
    if(x<=105*n):
        print(1)
        sys.exit()
    n+=1
print(0)

s = str(input())
t = str(input())
revise = 0
len_str = len(s)
for i in range(len_str):
    if s[i] != t[i]:
        revise += 1
print(int(revise))

K, N= map(int, input().split())
A = list(map(int, input().split()))
max=K-(A[N-1]-A[0])
for i in range(N-1):
    a=A[i+1]-A[i]
    if max<a:
        max=a
print(K-max)

x = float(input())
if 1 >= x >= 0:
    if x == 1:
        print(0)
    elif x == 0:
        print(1)

13.3 False Negative Analysis↩︎

13.3.1 BCB Dataset↩︎

• There is zero overlap of False Negative between both the models. This is also due to the fact that there are very low false negative overall, due to the model tending to overfit on the dataset.

• The False Negatives from the CodeGraph, but True Positives from CodeBERT, shows consistently lower confidence at an average of 0.7. This shows that we can tweak the classification threshold to handle these lower confidence scores. On further inspection of these cases, which where just 14, shows that this prediction of false negatives are more cause of variation in parameters, which seems to mislead the model to not detect them as clone. Morover we can argue these are Type IV clones, which would be better identified, given more context. An example can be seen in the Code Pair Listing [[code:bcb:23677124] & [code:bcb:23677129]]

• The False Negatives from the CodeBERT, but True Positives from CodeGraph, have a strong very high confidence. This is not helpful, as it is clearly seen that the sequence model is predicting them wrongly as Negataives with a conf average of 0.99. This is a very intersting case, as all the 52 of these cases seems to have the code very different syntactically, but semantically they are same. This shows how the graph model has an edge over the sequence model. This can be seen in the Code Pair Listing [[code:bcb:3257108] & [code:bcb:21044331]]

public FTPClient sample1c(String server, int port, String username, String password) throws SocketException, IOException {
        FTPClient ftpClient = new FTPClient();
        ftpClient.setDefaultPort(port);
        ftpClient.connect(server);
        ftpClient.login(username, password);
        return ftpClient;
}

public FTPClient sample3b(String ftpserver, String proxyserver, int proxyport, String username, String password) throws SocketException, IOException {
        FTPHTTPClient ftpClient = new FTPHTTPClient(proxyserver, proxyport);
        ftpClient.connect(ftpserver);
        ftpClient.login(username, password);
        return ftpClient;
}

public static String getMD5(String s) {

    try {
        MessageDigest m = MessageDigest.getInstance("MD5");
        m.update(s.getBytes(), 0, s.length());
        s = new BigInteger(1, m.digest()).toString(16);
    } 
    catch (NoSuchAlgorithmException ex) {
        ex.printStackTrace();
    }
    return s;
}

private static byte[] getKey(String password) throws UnsupportedEncodingException, NoSuchAlgorithmException {
    MessageDigest messageDigest = MessageDigest.getInstance(Constants.HASH_FUNCTION);
    messageDigest.update(password.getBytes(Constants.ENCODING));
    byte[] hashValue = messageDigest.digest();
    int keyLengthInbytes = Constants.ENCRYPTION_KEY_LENGTH / 8;
    byte[] result = new byte[keyLengthInbytes];
    System.arraycopy(hashValue, 0, result, 0, keyLengthInbytes);
    return result;
}

13.3.2 PoolC Dataset↩︎

• There are 34 False Negatives predicted by both the models combined. Here we see that the average score from Graph model is 0.64 where as 0.87 is the average score from the sequence model. This shows how the sequence model is very confidently wrong, which is not a good sign although, when examples are examined for this case, we find the clone pairs distinctly have a difference in length in the code, which seems to be the reason for these wrong predictions. Code Pair Listing [[code:poolc:7785] & [code:poolc:4401]] demonstrates this.

• The False Negatives from CodeGraph, but True positives from CodeBERT, shows a consistently lower score of confidence with average confidence being 0.63. where as the true positives as well from codeBERT have lower confidence average of 0.78. Here the CodeGraph is marignaly doing wrong, and this seems to be due to the code length again. the difference in the size of code snippet is larger. This can be seen again from the Code Pair Listing [[code:poolc:5696] & [code:poolc:3165]].

• The False Negatives from CodeBERT, but True Positives from CodeGraph, are around 83. This cases seems to be strongly False at an average score of 0.81 for the CodeBERT, which is not a good sign of the model predicting them wrong confidently, again, the reason looks the same as the snippet sizes being very different. Simialarly average confidence of True positive from CodeGraph is 0.61, which is again not that confident. This can be seen with the Code Pair Listing [[code:poolc:6612] & [code:poolc:8682]].

while True: 

    a = input()
    
    if a == '0':
        break
    
    print(sum(map(int,*a.split())))

n = int(input())
res = 0
while n != 0:
 res = n dropped = n
 while dropped//10 != 0:
  dropped = dropped//10
  res += dropped print(res)
 res = 0
 n = int(input())

while True:
    
    n = input()
    
    if n == "0":
        break
        
    print(sum([int(i) for i in n]))

while True:
    num = input()
    if int(num) == 0:
        break
    sum = 0
    for i in num:
        a = int(i)
        sum += a
    print(sum)

H,A = map(int,input().split())
cnt = 0
while True:
    if H <= 0:
        print(cnt)
        break
    else:
        H -= A
        cnt += 1

h,a = map(int, input().split())
an, bn = divmod(h,a)
if bn == 0:
    print(an)
else:
    print(an+1)

References↩︎