Background

The suffix array (\({{\mathsf {S}}}{{\mathsf {A}}}\)) [1] is one of the most important data structures in string processing. It enables efficient pattern searching in strings, as well as solving many other string problems [2,3,4]. More space-efficient solutions for such problems are possible by replacing the suffix array with an index based on the Burrows–Wheeler transform (\(\mathsf {BWT}\)) [5]. Many applications require additional data structures—most commonly, the longest common prefix (\(\mathsf {LCP}\)) [6] array and the document array (\({{\mathsf {D}}}{{\mathsf {A}}}\)) [7]—on top of \({{\mathsf {S}}}{{\mathsf {A}}}\) or \(\mathsf {BWT}\). These structures, possibly stored in compressed form, serve as a basis for building modern compact full-text indices, which allow to efficiently pre-process and query strings in compact space.

There are several internal memory algorithms designed for constructing the suffix array and additional data structures when the input consists of a single string [8, 9]. While less emphasis has been put on specialized algorithms for string collections, in many applications the input is composed by many strings, and a common approach is concatenating all strings into a single one and using a standard construction algorithm. However, this approach may deteriorate either the theoretical bounds or the practical behavior of construction algorithms due to, respectively, the resulting alphabet size or unnecessary string comparisons [10,11,12].

Textual documents and webpages are examples of widespread large string collections. In Bioinformatics, important problems on collections of sequences may be solved rapidly with a small memory footprint using the aforementioned data structures, for example, finding suffix-prefix overlaps for sequence assembly [13], clustering cDNA sequences [14], finding repeats [15] and sequence matching [16].

In this paper we present gsufsort, an open source tool that takes a string collection as input, constructs its (generalized) suffix array and additional data structures, like the \(\mathsf {BWT}\), the \(\mathsf {LCP}\) array, and the \({{\mathsf {D}}}{{\mathsf {A}}}\), and writes them directly to disk. This way, applications that rely on such data structures may either read them from disk or may easily include gsufsort as a component. Large collections, with up to \(2^{64}-d-2\) total letters in d strings, may be handled provided that there is enough memory. This tool is an extension of previous results [10], with new implementations of procedures to obtain the \(\mathsf {BWT}\) and the generalized suffix array (\(\mathsf {GSA}\)) from \({{\mathsf {S}}}{{\mathsf {A}}}\) during output to disk, and with the implementation of a lightweight alternative to compute \({{\mathsf {D}}}{{\mathsf {A}}}\).

Implementation

gsufsort is implemented in ANSI C and requires a single Make command to be compiled. It may receive a collection of strings in fasta, fastq or raw ASCII text formats and computes \({{\mathsf {S}}}{{\mathsf {A}}}\) and related data structures, according to input parameters. gsufsort optionally supports gzipped input data using zlibFootnote 1 and kseqFootnote 2 libraries. Setting command-line arguments allows selecting which data structures are computed and written on disk, and which construction algorithm is used (see below). Additionally, a function for loading pre-constructed data structures from disk is also provided.

Given a collection of d strings \(T^1, T^2, \dots , T^d\) from an alphabet \(\Sigma =[1,\sigma ]\) of ASCII symbols, having lengths \(n_1, n_2, \dots , n_d\), the strings are concatenated into a single string \(T[0,N-1]=T^1\$ T^2\$ \cdots \$ T^d \$\#\) using the same separator $ and an end-marker #, such that $ and # do not occur in any string \(T^i\), and \(\#<\) $ \(< \alpha\) for any other symbol \(\alpha \in \Sigma\). The total length of T is \(\sum _{i=1}^{d} (n_i +1) +1 = N\).

Before giving details on gsufsort implementation, we briefly recall some data structures definitions. For a string S of length n let the suffix starting at position i be denoted \(S_i\), \(0\le i\le n-1\). The suffix array \({{\mathsf {S}}}{{\mathsf {A}}}\) of a string S of length n is an array with a permutation of \([0,n-1]\) that gives the suffixes of S in lexicographic order. The length of the longest common prefix of strings R and S is denoted by \({\mathsf {lcp}} (R,S)\). The \(\mathsf {LCP}\) array for S gives the \({\mathsf {lcp}}\) between consecutive suffixes in the order of \({{\mathsf {S}}}{{\mathsf {A}}}\), that is \(\mathsf {LCP} [0]=0\) and \(\mathsf {LCP} [i]={\mathsf {lcp}} (S_{SA[i]},S_{SA[i-1]})\), \(0< i\le n-1\). For a suffix array of a collection of strings, the position i of the document array \({{\mathsf {D}}}{{\mathsf {A}}}\) gives the string to which suffix \(T_{{{\mathsf {S}}}{{\mathsf {A}}} [i]}\) belongs. For the last suffix \(T_{N-1}=\#\) we have \({{\mathsf {D}}}{{\mathsf {A}}} [0]=d+1\). The generalized suffix array gives the order of the suffixes of every string in a collection, that is, the \(\mathsf {GSA}\) is as an array of N pairs of integers (a, b) where each entry (a, b) represents the suffix \(T^a_b\), with \(1 \le a \le d\) and \(0 \le b \le n_a-1\).

gsufsort uses algorithm gSACA-K [10] to construct \({{\mathsf {S}}}{{\mathsf {A}}}\) for the concatenated string \(T[0,N-1]\), which breaks ties between equal suffixes from different strings \(T^i\) and \(T^j\) by their ranks, namely i and j. gSACA-K can also compute \(\mathsf {LCP}\) and \({{\mathsf {D}}}{{\mathsf {A}}}\) during \({{\mathsf {S}}}{{\mathsf {A}}}\) construction, such that \(\mathsf {LCP}\) values do not exceed separator symbols. gSACA-K runs in O(N) time using \(O(\sigma )\) working space.

The \(\mathsf {BWT}\) is calculated during the output to disk according to its well-known relation to \({{\mathsf {S}}}{{\mathsf {A}}}\) [3]

$$\begin{aligned} \mathsf {BWT} [i]=T[({{\mathsf {S}}}{{\mathsf {A}}} [i]-1) \bmod N]. \end{aligned}$$

The generalized suffix array (\(\mathsf {GSA}\)) can be computed by gsufsort from \({{\mathsf {S}}}{{\mathsf {A}}}\) and \({{\mathsf {D}}}{{\mathsf {A}}}\) during the output to disk, using the identity

$$\begin{aligned} \mathsf {GSA} [i] = {\left\{ \begin{array}{ll} ({{\mathsf {D}}}{{\mathsf {A}}} [i], {{\mathsf {S}}}{{\mathsf {A}}} [i]-{{\mathsf {S}}}{{\mathsf {A}}} [{{\mathsf {D}}}{{\mathsf {A}}} [i]]{-1}) &{} \text{ if } \, {{\mathsf {D}}}{{\mathsf {A}}} [i]>1 \\ ({{\mathsf {D}}}{{\mathsf {A}}} [i], {{\mathsf {S}}}{{\mathsf {A}}} [i]) &{} \text{ otherwise }. \end{array}\right. } \end{aligned}$$
(1)

We also provide a lightweight version (gsufsort-light) for the computation of \({{\mathsf {D}}}{{\mathsf {A}}}\). It uses less memory at the price of being slightly slower. It computes a bitvector \({\mathsf {B}} [0,N-1]\) with O(1) rank support [4] such that \(B[i]=1\) if \(T[i]=\$,\) and \(B[i]=0\) otherwise. The values in \({{\mathsf {D}}}{{\mathsf {A}}}\) are obtained on-the-fly while \({{\mathsf {D}}}{{\mathsf {A}}}\) (or \(\mathsf {GSA}\)) is written to disk, through the identity

$$\begin{aligned} {{\mathsf {D}}}{{\mathsf {A}}} [i] = \mathsf {rank_1}({{\mathsf {S}}}{{\mathsf {A}}} [i])+1. \end{aligned}$$

Results

We compared our tool and mkESA. mkESA [17] is a fast suffix array construction software designed for bioinformatics applications.

We ran both versions of our tool, gsufsort and gsufsort-light, to build arrays \(\mathsf {GSA}\) and \(\mathsf {LCP}\), while mkESAFootnote 3 was run to build arrays \({{\mathsf {S}}}{{\mathsf {A}}}\) and \(\mathsf {LCP}\) for the concatenation of all strings (using the same symbol as separators). The experiments were conducted on a single core of a machine with GNU/Linux (Debian 8, kernel 3.16.0-4, 64 bits) with an Intel Xeon E5-2630 2.40-GHz, 384 GB RAM and 13 TB SATA storage. The sources were compiled by GNU GCC version 4.8.4 with option -O3.

The collections we used in our experiments are described in Table 1. They comprise real DNAs, real proteins, documents, random DNA and random protein, and differ by their alphabet size and also by the maximum and average \({\mathsf {lcp}}\), which offer an approximation for suffix sorting difficulty.

Table 1 Collections
Full size table

The results are shown in Table 2. The data shows a clear time/memory tradeoff for DNA sequences, gsufsort being faster while using approximately 1.25 more memory, gsufsort-light using slightly less memory then mkESA but taking more time. On proteins, gsufsort-light is only marginally slower than gsufsort but faster than mkESA. The authors of mkESA reported a 32% gain on a large protein dataset using 16 threads [17], but larger \({\mathsf {lcp}}\) values seem not to favor mkESA when compared to gsufsort-light, which is 47.9% faster on proteins and 12.9% faster on DNA.

Table 2 Algorithms’ running times and memory usage on different datasets collections
Full size table

The memory ratio (bytes/N) of gsufsort and gsufsort-light is constant, 21 and 17 bytes per input symbol respectively, corresponding to the space of the input string T (N bytes) plus the space for arrays \({{\mathsf {S}}}{{\mathsf {A}}}\) and \(\mathsf {LCP}\) (8N bytes each) and, only for gsufsort, the space for \({{\mathsf {D}}}{{\mathsf {A}}}\) (4N bytes).

We have also evaluated the performance of gsufsort, gsufsort-light and mkESA on collections of random DNA and random protein sequences. The collections have a growing number of 1MB sequences. The running time in seconds and the peak memory usage in GB are shown in Fig. 1 (logarithmic scale). Using random sequences reduces the variation due to \({\mathsf {lcp}}\) among collections. We can see a perfectly steady behavior of mkESA. While still O(N), gsufsort displays a deviation due to larger constants.

Fig. 1
figure 1

Running time in seconds and peak memory in GB (in logarithmic scale) on an random DNA and protein collections

Full size image

Conclusions

We have introduced gsufsort, a fast, portable, and lightweight tool for constructing the suffix array and additional data structures for string collections. gsufsort may be used to pre-compute indexing structures and write them to disk, or may be included as a component in different applications. As an additional advantage, gsufsort is not restricted to biological sequences, as it can process collections of strings over ASCII alphabets.

Availability and requirements

  • Project name: gsufsort

  • Project home page: http://www.github.com/felipelouza/gsufsort

  • Operating system(s): Platform independent

  • Programming language: ANSI C

  • Other requirements: make, zlib (optional)

  • License: GNU GPL v-3.0.