CoCoAST: Representing Source Code via Hierarchical Splitting and Reconstruction of Abstract Syntax Trees

Abstract

Recently, machine learning techniques especially deep learning techniques have made substantial progress on some code intelligence tasks such as code summarization, code search, clone detection, etc. How to represent source code to effectively capture the syntactic, structural, and semantic information is a key challenge. Recent studies show that the information extracted from abstract syntax trees (ASTs) is conducive to code representation learning. However, existing approaches fail to fully capture the rich information in ASTs due to the large size/depth of ASTs. In this paper, we propose a novel model CoCoAST that hierarchically splits and reconstructs ASTs to comprehensively capture the syntactic and semantic information of code without the loss of AST structural information. First, we hierarchically split a large AST into a set of subtrees and utilize a recursive neural network to encode the subtrees. Then, we aggregate the embeddings of subtrees by reconstructing the split ASTs to get the representation of the complete AST. Finally, we combine AST representation carrying the syntactic and structural information and source code embedding representing the lexical information to obtain the final neural code representation. We have applied our source code representation to two common program comprehension tasks, code summarization and code search. Extensive experiments have demonstrated the superiority of CoCoAST. To facilitate reproducibility, our data and code are available https://github.com/s1530129650/CoCoAST.

Appendices

Appendix

A Code Summarization

1.1 A.1 Hyperparameters

Table  13 summarizes the hyperparameters used in our experiments. $len_{code}$ and $len_{sum}$ are the sequence length of code and summary, respectively. Each covers at least 90% of the training set. $vocab_{code}$, $vocab_{sum}$, and $vocab_{ast}$ are the vocabulary size of code, summary, and AST. $len_{pos}$ refers to the clipping distance in relative position. $d_{Emb}$ is the dimension of the embedding layer. heads and layers indicate the number of layers and heads in Transformer, respectively. $d_{ff}$ is the hidden layer dimension of feed-forward in Transformer. $d_{RvNN}$ and $activate_f$ are dimension of hidden state and activate function in RvNN, respectively. $layer_{ff}$ is the number of the feed-forward layer in the copy component.

The values of these hyperparameters are set according to the related work (Ahmad et al. 2020; Zhang et al. 2019). We adjust the sizes of $d_{Emb}$, $d_{ff}$, and $d_{RvNN}$ empirically. The batch size is set according to the computing memory.

Table 13 Hyperparameters in our experiments on code summarization

1.2 A.2 Runtime of Different AST Representation Models

Tables 14 and 15 show the time cost for differenct approaches on the two datasets. The 2nd column is the training time of one epoch. The 3rd column is the total training time. And the last column is the inference time per function. On the Funcom dataset, the baselines have different training time. Most of them range from 7 to 69 hours (except that Hybrid-DRL takes 633 hours). The inference time per function is 0.0073s for CASTS and 0.001 to 0.0139s for other baselines. Our approach has comparable time cost as the baselines. Similar performance can also be observed on TL-CodeSum. During training, HybridDrl firstly trains hybrid code representation by applying LSTM and Tree-LSTM to code token and AST. Then it generates summary based on actor-critic reinforcement learning. Therefore, it takes long time for HybridDrl to train the model.

Table 14 Time cost of different AST representation models in Funcom on code summariztion

Table 15 Time cost of different AST representation models in TL-CodeSum on code summarization

1.3 A.3 Experimental Results on Deduplicated Dataset

In our experiment, we find the existence of code duplication in TL-CodeSum: around $20\%$ code snippets in the testing set can be found in the training set. Thus, we remove the duplicated samples from the testing set and re-evaluate all approaches. Table  16 shows the result of different models in the rest testings set without duplicated samples. Our model still outperforms all baselines.

Table 16 Performance of different models on deduplicated TL-CodeSum dataset

1.4 A.4 Evaluation Metrics

We provide the details of the evaluation metrics we used in the experiments.

1.4.1 A.4.1 BLEU

BLEU measures the average n-gram precision between the reference sentences and generated sentences, with brevity penalty for short sentences. The formula to compute BLEU-1/2/3/4 is:

$$\begin{aligned} {\text {BLEU-N}}= BP \cdot \exp \sum _{n=1}^{N} \omega _{n} \log p_{n}, \end{aligned}$$

(14)

where $p_n$ (n-gram precision) is the fraction of n-grams in the generated sentences which are present in the reference sentences, and $\omega _{n}$ is the uniform weight 1/N. Since the generated summary is very short, high-order n-grams may not overlap. We use the +1 smoothing function (Lin and Och 2004). BP is brevity penalty given as:

$$\begin{aligned} BP=\left\{ \begin{array}{cl} 1 &{} \text{ if } c>r \\ e^{(1-r / c)} &{} \text{ if } c \le r \end{array}\right. \end{aligned}$$

(15)

Here, c is the length of the generated summary, and r is the length of the reference sentence.

1.4.2 A.4.2 ROUGE-L

Based on longest common subsequence (LCS), ROUGE-L is widely used in text summarization. Instead of using only recall, it uses F-score which is the harmonic mean of precision and recall values. Suppose A and B are generated and reference summaries of lengths c and r respectively, we have:

$$\begin{aligned} \left\{ \begin{array}{cl} P_{ROUGE-L}=\frac{L C S(A, B)}{c}\\ R_{ROUGE-L}=\frac{L C S(A, B)}{r}\\ \end{array}\right. \end{aligned}$$

(16)

$F_{ROUGE-L}$, which indicates the value of ${\text {ROUGE-L}}$, is calculated as the weighted harmonic mean of $P_{ROUGE-L}$ and $R_{ROUGE-L}$:

$$\begin{aligned} F_{ROUGE-L}=\frac{\left( 1+\beta ^{2}\right) P_{ROUGE-L} \cdot R_{ROUGE-L}}{R_{ROUGE-L}+\beta ^{2} P_{ROUGE-L}} \end{aligned}$$

(17)

$\beta$ is set to 1.2 as in Zhang et al. (2020); Wan et al. (2018).

1.4.3 A.4.3 METEOR

METEOR is a recall-oriented metric that measures how well the model captures the content from the references in the generated sentences and has a better correlation with human judgment. Suppose m is the number of mapped unigrams between the reference and generated sentence with lengths c and r respectively. Then, precision, recall and F are given as:

$$\begin{aligned} P=\frac{m}{c},\,\, R=\frac{m}{r},\,\, F=\frac{P R}{\alpha P+ (1-\alpha )R} \end{aligned}$$

(18)

The sequence of mapping unigrams between the two sentences is divided into the fewest possible number of “chunks”. This way, the matching unigrams in each “chunk” are adjacent (in two sentences) and the word order is the same. The penalty is then computed as:

$$\begin{aligned} \text{ Pen }=\gamma \cdot \text{ frag } ^{\beta } \end{aligned}$$

(19)

where $\text{ frag }$ is a fragmentation fraction: $\text {frag}=ch/m$, where ch is the number of matching chunks and m is the total number of matches. The default values of $\alpha , \beta , \gamma$ are 0.9, 3.0 and 0.5 respectively.

1.4.4 A.4.4 CIDER

CIDER is a consensus-based evaluation metric used in image captioning tasks. The notions of importance and accuracy are inherently captured by computing the TF-IDF weight for each n-gram and using cosine similarity for sentence similarity. To compute CIDER, we first calculate the TF-IDF weighting $g_k(s_i)$ for each n-gram $\omega _k$ in reference sentence $s_i$. Here $\omega$ is the vocabulary of all n-grams. Then we use the cosine similarity between the generated sentence and the reference sentences to compute ${\text {CIDER}}_n$ score for n-grams of length n. The formula is given as:

$$\begin{aligned} {\text {CIDER}}_{n}\left( c_{i}, s_{i}\right) ==\frac{<\varvec{g}^{\varvec{n}}\left( c_{i}\right) , \varvec{g}^{\varvec{n}}\left( s_{i}\right) >}{\left\| \varvec{g}^{n}\left( c_{i}\right) \right\| \left\| \varvec{g}^{n}\left( s_{i}\right) \right\| } \end{aligned}$$

(20)

where $\varvec{g}^{\varvec{n}}(s_i)$ is a vector formed by $g_k(s_i)$ corresponding to all the n-grams (n varying from 1 to 4). $c_i$ is the $i^{th}$ generated sentence. Finally, the scores of various n-grams can be combined to calculate CIDER as follows:

$$\begin{aligned} {\text {CIDER}}\left( c_{i}, s_{i}\right) =\sum _{n=1}^{N} w_{n} {\text {CIDER}}_{n}\left( c_{i}, s_{i}\right) \end{aligned}$$

(21)

1.5 A.5 Human evaluation

Table 17 Statistics significance p-value of CoCoAST over other methods in human evaluation

We conduct a human evaluation to evaluate the effectiveness of the summaries generated by our approach CoCoAST and the other three approaches. The shows that CoCoAST outperforms the others in all three aspects: similarity, naturalness, and informativeness. we confirmed the dominance of our approach using Wilcoxon signed-rank tests for human evaluation. The result shown in Table  17 reflects that the improvement of CoCoAST over other approaches is statistically significant with all p-values smaller than 0.05 at 95% confidence level (except CodeAstnn in the naturalness).

B Code Search

1.1 B.1 Hyperparameters

Table  18 summarizes the hyperparameters used in our experiments on code search. The values of these hyperparameters are set according to the related work (Zhang et al. 2019). We adjust the sizes of $d_{Emb}$, $d_{ff}$, and $d_{RvNN}$ empirically. The batch size is set according to the computing memory.

Table 18 Hyperparameters in our experiments on code search

C Full AST

Fig. 8

Full AST

D Full Rules

Fig. 9

Full Splitting Rules. The Body and Block are treated the same in implementation