Extracting significant pattern histories from timestamped texts using MapReduce

Abstract

This paper provides valuable clues for trend analysis in text mining that one can have texts attached with timestamps as tags and then observe the frequency distribution of the patterns over equally spaced time intervals to predict the trend. Observing frequency distributions (histories) of significant patterns plays an important role for trend analysts. To have the computation of extracting these frequency distributions from a huge amount of texts with timestamps over long time periods scalable, this paper proposes a novel approach based on Hadoop MapReduce programming model that improves our previous work based on external memory approach to reduce the computation time from several days to several hours. The history of a significant pattern is the frequency distribution of that pattern over equally spaced time intervals; a significant pattern is one maximal repeat of consecutive words within texts. Note that the length of one significant pattern can be as long as that of one sentence if that sentence appears twice. To solidify the contribution of this study, the experimental resources included the titles and abstracts (total 12 GB) of 14,473,242 articles from 1990 to 2014 (25 years) downloaded from PubMed, a well-known web site for biomedical literature. Experimental results show that the scale of computation time can be reduced from days to hours employing six computing nodes within one personal computer cluster. Notably, these pattern histories, over two decades in length, not only provide clues that can be analyzed for trend variations within these articles, but also have the potential to reveal revolutions in article writing that might be valuable to the linguist who engages in corpus analysis in the future.

Appendix: Extraction of candidate significant pattern histories

The history of one significant pattern is a series of frequency distributions of that pattern over consecutive, equal time-intervals. The processes of extracting candidate significant patterns with the statistics of their tags herein are derived from the previous work in [2] that was characters-based. The main idea that underlies the extraction of candidate significant pattern histories is to scan sorted word-suffixes that are assigned timestamps and then to extract the longest common prefix words as candidate significant patterns, if they exist, between two adjacent word-suffixes. Meanwhile, a stack is used to accumulate the history of that pattern via push/pop operations.

Fig. 12

The pseudo code for extracting candidate significant pattern history

Figure 12 presents the pseudocode for extracting candidate significant patterns and their corresponding histories. The input contains consecutive and sorted word-suffixes that are labeled with timestamps, and the output contains the histories of candidate significant patterns with right or left boundary verification. For the purposes of describing the extraction processes in detail, the key terms that are used in the pseudocodes are as follows:

• ParseSuffix: a subroutine to extract word-suffixes and their timestamp within one input line.

• Record: an object for storing information about one candidate significant pattern.

  • Length: the number of words within a candidate significant pattern.

  • AddHistory: a method for adding the history of one pattern to that of a candidate significant pattern.

  • SubHistory: a method for subtracting the history of one pattern from that of a candidate significant pattern.

  • Boundary: a timestamp for the pattern whose history is computed twice.

• PatternStack: a stack for keeping track the histories of one pattern and its sub-patterns.

• CommonPrefixWordsAndHistory: a method that returns a record with the longest common prefix words of two adjacent word-suffixes and their histories.

• TopStackRecord: the record on the top of the stack PatternStack

A simple example of scanning sorted word-suffixes is presented below to illustrate the extraction process. Figure 13, in which each of $W_\alpha$, $W_\beta$ and so on, represents one word, includes nine sorted word-suffixes with individually attached timestamps, such as “2012–3”. For simplicity, only those common prefix words between two adjacent word-suffixes are presented; when the last sorted word-suffixes are reached, one extra line (the tenth line) was added with an empty string to force the patterns that remain in the PatternStack, if any exist, to pop out. Table 6 presents the histories of the four candidate significant patterns that are extracted according to Fig. 13.

Fig. 13

9 Sorted word-suffixes

Table 6 The histories of four candidate significant patterns that are extracted from Fig. 13

Fig. 14

Push the pattern “$W_\alpha$” when at Line 2

Fig. 15

Push the “$W_\alpha W_\beta W_\gamma$” pattern when at Line 4

Fig. 16

Push the “$W_\alpha W_\beta W_\gamma W_\delta W_\eta W_\rho$” pattern when at Line 7

To understand the extraction processes more precisely, temporary contents within PatternStack are presented below, and the ten sorted word-suffixes are scanned, as shown in Fig. 13. First, as shown in Fig.14 scanning at line 2 found—pattern “$W_\alpha$”, which is the longest common prefix-word that is derived from the first and the second lines, and this pattern was pushed into PatternStack because its length (${=}1$) was greater than that (${=}\hbox { -}1$) of the NullStr of the current top record in PatternStack. This action corresponds to the pseudocode at line (S7) in Fig. 12. Similarly, as shown in Figs. 15 and 16, when scanning lines 4 and 7 revealed two patterns, “$W_\alpha W_\beta , W_\gamma$” and “$W_\alpha W_\beta W_\gamma W_\delta W_\eta W_\rho$”, which were consecutively pushed into PatternStack. However, as shown in Fig. 17, Scanning line 8 revealed that the length (${=}2$) of pattern “$W_\alpha W_\beta$” was less than that (${=}6$) of pattern “$W_\alpha W_\beta W_\gamma W_\delta W_\eta W_\rho$” on the top record within PatternStack. This situation met the conditions of line (S12) and caused the record that contained the pattern “$W_\alpha W_\beta W_\gamma W_\delta W_\eta W_\rho$” and its history to be Popped. The history of “$W_\alpha W_\beta W_\gamma W_\delta W_\eta W_\rho$” was accumulated with that of the top record for pattern “$W_\alpha W_\beta W_\gamma$” because the latter was a sub-pattern of the former; the history of the pattern on the top of PatternStack needed to be removed along with the boundary history that was marked “2012–1” because that boundary history was counted twice at line 6. Furthermore, a pop operation ran once again and output the pattern “$W_\alpha W_\beta W_\gamma$” with its history, which was accumulated with that of the pattern “$W_\alpha W_\beta$”, which was freshly generated, and then the record with its history was pushed onto PatternStack after removing the boundary overlapping “$W_\alpha W_\beta$” at line 7. Note that the above operations were associated with the pseudocode from (S12) to (S29) in Fig. 12. When line 10 was scanned, an extra line was added to force the two patterns, “$W_\alpha$” and “$W_\alpha W_\beta$”, and their histories, to be popped from PatternStack as shown in Fig. 18.

Fig. 17

Line 8(1): pop the “$W_\alpha W_\beta W_\gamma W_\delta W_\eta W_\rho$” and its history

Fig. 18

Line 8(2): pop the “$W_\alpha W_\beta W_\gamma$” and its history