Detecting bursty terms in computer science research

Burst detection

The problem of tracking topics in time-ordered corpora was formalised by a DARPA-sponsored initiative under the name Topic Detection and Tracking (TDT). Early research focused on segmenting a corpus into topics, finding the first mention of each topic and then tracking and plotting their popularity over time (Allan et al. 1998). As computer hardware improved, it became common to use Latent Dirichlet Allocation (LDA) for this kind of topic modelling (Blei et al. 2003). A typical method involves splitting the corpus into time steps, finding topics in each chunk and then linking them together across time steps based on some measure of similarity (Griffiths and Steyvers 2004; Steyvers et al. 2004; Mei and Zhai 2005). The prevalence of each topic can then be tracked over time and bursty periods identified. However, this comes with a number of disadvantages, such as the lack of interpretability of the results and the difficulty in coherently linking LDA topics together between subsequent time-steps.

The opposite approach is to first identify the bursty terms in a dataset, and then cluster them together into topics, using, for instance, Kleinberg (2002)’s burst detection algorithm. Originally developed to detect topics in email chains, Kleinberg’s method assumes that terms in documents are emitted by a two-state automaton. The automaton may spontaneously transition from a non-bursty state to a bursty state, or vice versa. Variants of this have been applied across several domains: Diao et al. (2012) and Mathioudakis and Koudas (2010) used it to detect bursty topics in Twitter data, the latter in real time, while Fung et al. (2005) and Takahashi et al. (2012) applied it to news streams.

However, when it comes to scientific literature, there are a few reasons why Kleinberg’s method is a less natural fit. He and Parker (2010) point out that, unlike Tweets and news articles, scientific papers tend to enter the world in batches, such as when a new edition of a journal or the proceedings of a conference is published. This violates Kleinberg’s underlying assumption that new items enter the dataset in a continuous fashion. It also forces us to impose longer time steps, such as years rather than seconds. This causes a second problem: the quantity of data available.

While there are several large open-access corpora of scientific abstracts, such as PubMed (biomedicine), arXiv (physics and computer science), Semantic Scholar (assorted) and DBLP (computer science), all of them cover short intervals, relative to the size of the time steps. Even in the best case scenario, we are likely to have less than a hundred years worth of usable data—which means approximately a hundred time steps. There has also been a vast change in the underlying landscape over the span of the dataset, because science in general (Bornmann and Mutz 2015) and computer science in particular (Wu et al. 2018), have both seen strong and sustained growth in the last century. By contrast, unless one collects many years of Twitter data, the size and characteristics of the dataset do not change substantially over time.

Several burst detection methods from other domains have been proposed for use on scientific documents. For instance, Stroup et al. (1989) take inspiration from epidemiology, and Zhang and Shasha (2006) take inspiration from gamma rays. However, of particular interest to us is He and Parker (2010)’s work that takes a popular technique from stock market analysis and applies it to PubMed data. This is an attractive idea: a great deal of work has been done in analysing stocks, because some people are highly motivated to predict what prices will do in later time steps.

Moving average convergence divergence

The basic item in the toolkit of the stock market analyst is the moving average. While a moving average necessarily lags behind real time data, it can smooth out random fluctuations to reveal underlying trends. The simple moving average (SMA) of a time series is the sum of its values in a set interval (called the span of the SMA), divided by the width of that interval. More advanced methods use exponential moving averages (EMAs), which assign more weight to more recent data.

For a given span, n, the exponential moving average of a time series, y(t), is (Investopedia 2019):

$$\begin{aligned} \text {EMA}(t_i) = \text {EMA}(t_{i-1}) + \frac{2}{n+1}(y(t_i)-\text {EMA}(t_{i-1})) \end{aligned}$$

(1)

Buy and sell signals for stocks can be generated by taking two EMAs with different spans and seeing where they cross. The moving average with the longer span responds more slowly to new data, so when there is a sudden change in the price of the stock, the shorter moving average will cross it in an upwards or downwards direction (Murphy 1999) (Fig. 2b). Moving Average Convergence Divergence (MACD) takes this idea a step further (Appel 2005). The long EMA is subtracted from the short EMA to give the MACD line. This MACD line is then itself averaged, to create a fourth time series called the signal line (Fig. 2c). The difference between the MACD and signal lines is called the histogram of the data, and can be thought of as an approximate measure of curve acceleration. When the histogram is positive, the price of the stock is accelerating upwards. When it is negative, the reverse is happening.

Open image in new window

Using a controlled vocabulary rather than free text has advantages in that it ensures the terms in the vocabulary will be meaningful, unique and free of spelling errors. However, a controlled vocabulary will necessarily lag behind the forefront of scientific development because new terms can only be added after they have risen to prominance. Additionally, keywords are not available for all datasets, limiting the scope of the method. For this reason, we use the free text of abstracts and titles.

Prediction

As well as detecting historically bursty terms, we would like to be able to predict whether they will become more or less prevalent in the future. Some related work has been done on this task. Prabhakaran et al. (2016) took 2.4 million abstracts from Web of Science and used an implementation of LDA to identify 500 topics. They then used a logistic regression classifier to predict whether their topics would rise or fall in popularity over subsequent time steps, yielding an accuracy of 70%. Balili et al. (2017) looked at a slightly different task; they took 21.2 million PubMed abstracts, clustered their MeSH keywords based on co-occurrence in article annotations and then trained a gradient-boosted trees classifier to predict whether individual clusters would survive or dissolve. Finally, in their 2011 paper, He and Parker took a database of approximately 100,000 Californian grant abstracts which were pre-labelled with “project terms” (keywords from a defined vocabulary of biomedical concepts (RePORT 2018) that were automatically assigned to grant applications) (He and Parker 2011). They calculated the MACD and histogram values for each term, then used various classifiers to predict whether the histogram itself would rise or fall in the future. While this is not exactly the same as predicting whether term prevalence would increase, their best classifier had an accuracy of 88%.

Dataset

We use a corpus gathered from DBLP, which is a large computer science bibliography hosted by Trier University in Germany. DBLP is considered to be reasonably comprehensive in its coverage of the field: Cavacini (2015) compared a number of bibliographies and found that DBLP had the greatest number of unique computer science articles indexed. It is freely available to download, either directly (DBLP 2019), or via Semantic Scholar (Allen Institute for Artificial Intelligence 2015).

After downloading, cleaning, and filtering out foreign language abstracts, we had a dataset of 2.6 million articles spanning the years 1988-2017. For each article, we combined title and abstract to form a document, then used Python’s Natural Language Toolkit (NLTK) (Bird et al. 2009) to tokenise, lemmatise and remove a short list of standard English stopwords.

From there, we read in the dataset year by year and formed a vocabulary of uni-, bi- and tri-gram terms. We calculated the document frequency of each term in each year, which left us with a 31 (year) \(\times\) 4.1 million (term) matrix. We filtered again, removing the terms that did not occur in more than 0.02% of abstracts for at least three consecutive years. This substantially reduced the amount of noise in the dataset from digitization errors and very rare bi- and tri-gram terms. The final size of our vocabulary was approximately 70,000 terms.

Normalisation

Over the 31 year span of the dataset, the number of documents published each year has risen substantially (Fig. 3). The mean length of titles has increased, while the mean length of abstracts has fluctuated (Fig. 4). This presents a problem; terms later in the dataset will be more likely to be flagged as bursts because of the underlying increase of size of the dataset. Therefore, we normalised the document frequency counts twice, first by dividing the data for each year by the total number of documents in that year, and then dividing by the mean number of tokens per document. This means that each element in the year-term matrix can be viewed as a normalised measure of prevalence.

Open image in new window

Open image in new window

Applying MACD

The first parameter we chose was the length of the moving average spans, (\(n_1\), \(n_2\), \(n_3\)). In stock market analysis, it is common to use (12, 26, 9) months (Murphy 1999). However, our dataset has just 31 time steps, so moving averages of this length would leave us with too little data with which to work. Therefore, after some experimentation, we used (6, 12, 3) years.

He and Parker (2010) used the raw value of the histogram as their metric for burstiness. However, when we attempted to apply it to our dataset, we discovered that there were some issues with scale. Common terms, such as “data”, often showed large numerical shifts on the histogram that are still insignificant when compared to their historical baseline level. Therefore, we introduced a scaling factor.

Initially, we experimented with using the mean or median value of each term’s historical prevalence. However, this biased the metric in favour of new terms which did not exist in the dataset before becoming popular. Then we tried the historical maximum, but found that this produced variable results; the prevalence over time is not generally smooth, so anomalous spikes occur frequently. Finally, we decided on the square root of the historical maximum since this produces more consistent results than the other metrics.

Predicting the future prevalence of terms using MACD features

In order to train a supervised classifier, we need to create a training set, (X, Y), where X is a matrix of terms and features and Y is a binary class indicating whether each term rose or fell in prevalence after a number of years. First, we choose how far in the future we wanted to predict (e.g. 3 years) and call this the prediction interval I. Then, for each year, \(y_i\), we:

1. 1.

   Take the last 20 years of data (\(\mathcal {D}(y_{i-20}, y_i)\)).

    
2. 2.

   Apply our burst detection method to \(\mathcal {D}(y_{i-20}, y_i)\). Select all terms above a burstiness threshold.

    
3. 3.

   Extract time series features such as the MACD and histogram values, the standard deviation, min and max. This forms the X part of our dataset (Fig. 5).

    
4. 4.

   Take a smoothed value of term prevalence during \(y_{i+I}\) and calculate whether it is above or below prevalence during \(y_i\). This forms the Y part of our dataset.

    
5. 5.

   Append X and Y onto the data for the previous year.

    

Both He and Parker (2011) and Balili et al. (2017) used a tree-based method to predict whether their clusters or terms would rise or fall in popularity. We follow them, using a random forest classifier, tested via 10-fold cross validation. A process diagram for the full methodology can be found in Fig. 6.

Open image in new window

Open image in new window

Burst detection

As an initial evaluation, we applied the burst detection method to our dataset and sorted the burstiness scores in descending order. Some terms with high burstiness were surprising, such as “novel” and “state [of the] art”. However, when we plotted them on a graph over time, we found that they have indeed become more popular over the span of the dataset (Fig. 7).

Open image in new window

Social media is another interesting example (Fig. 8). “Social network” began to climb in popularity around 1998, with a growth curve that mirrors “social media”, 5 years later. “Twitter” and “facebook” climb together, but “facebook” reaches a plateau earlier. We also find some cases where the orthography of words has changed over time. For instance, “web site”, peaked in 2001, then was gradually replaced by “website” (Fig. 9).

Open image in new window

Open image in new window

There are, however, some issues with the terms detected. For example, the top 30 bursty terms over the span of the dataset are:

deep, neural network, neural, convolutional, reserved, right reserved, science bv right, bv right, bv right reserved, convolutional neural, convolutional neural network, elsevier bv, deep learning, elsevier bv right, right, elsevier science bv, science bv, elsevier science, spl, bv, elsevier, cnn, iot, learning, deep neural, deep neural network, elsevier ltd, xml, internet thing, elsevier ltd right

To investigate how bursty terms have developed over time, we sort the clusters by year, based on when the bulk of the activity occurred, then manually choose a sample that is fairly evenly spread over the span of the dataset. For each of the 52 chosen clusters, we choose a single representative term, or a term and an acronym, such as [recurrent neural, rnn] for [recurrent neural network, recurrent neural, recurrent, network rnn, rnn, neural network rnn], then track the term over time and display it on a graph (Fig. 10).

Open image in new window

In Fig. 10, we notice that many of the later bursts seem to involve deep learning in some way. The earlier ones are more diverse. There are a number of different growth patterns. “Security” undergoes a nearly linear increase, as does “dataset”. Others peak twice, such as “neural networks” and “virtual reality”. This ties in neatly with what we know of the history of these two ideas: the first consumer VR headsets were in the headlines in the 90s (Kahaner 1994), and a number of neural network breakthroughs happened in a similar time period (Rumelhart et al. 1986). Some of the terms reach peak popularity, then persist, such as “data mining” and “gene expression”. Others fall out of favour fairly swiftly, such as “intranet” and “web 2.0”. We note that this does not mean that these concepts are no longer used, only that these particular terms no longer find their way into titles and abstracts.

Prediction

As described in “Predicting the future prevalence of terms using MACD features” section, we aim to predict whether a given term will rise or fall in popularity after a time interval I. Since the classes (rise, fall) are unbalanced, we subsampled the majority class, and trained a random forest classifier on the data. We chose the number of trees and the maximum depth by considering the training/testing error, then experimented with a range of prediction intervals (1–5 years) and burstiness thresholds (0.0006–0.0016).

The results are shown in Fig. 11, while the effect of the burstiness threshold on dataset size is shown in more detail in Table 2.

Open image in new window

Table 2

The effect of changing the burstiness threshold on classifier accuracy

Burst threshold	Dataset size	Accuracy	F1
0.0006	56136	0.71 ± 0.01	0.72 ± 0.01
0.0008	25570	0.75 ± 0.01	0.75 ± 0.01
0.0010	12978	0.79 ± 0.01	0.79 ± 0.01
0.0012	7450	0.82 ± 0.01	0.82 ± 0.01
0.0014	4886	0.82 ± 0.02	0.83 ± 0.02
0.0016	3332	0.83 ± 0.02	0.84 ± 0.02

A prediction interval of 3 was chosen and the error in each measurement is the standard deviation of the ten folds

Increasing the burstiness threshold increases the performance of the classifier substantially. However, thresholding this way comes at the cost of the amount of data available. At the highest threshold, there are just over 3000 terms, some of which describe the same concept (e.g. “convolutional neural”, “convolutional neural network”). The prediction interval also matters. When we vary both parameters together, we see that performance is highest when the prediction is made 3 years into the future.

Limitations