pblat: a multithread blat algorithm speeding up aligning sequences to genomes
The blat is a widely used sequence alignment tool. It is especially useful for aligning long sequences and gapped mapping, which cannot be performed properly by other fast sequence mappers designed for short reads. However, the blat tool is single threaded and when used to map whole genome or whole transcriptome sequences to reference genomes this program can take days to finish, making it unsuitable for large scale sequencing projects and iterative analysis. Here, we present pblat (parallel blat), a parallelized blat algorithm with multithread and cluster computing support, which functions to rapidly fine map large scale DNA/RNA sequences against genomes.
The pblat algorithm takes advantage of modern multicore processors and significantly reduces the run time with the number of threads used. pblat utilizes almost equal amount of memory as when running blat. The results generated by pblat are identical with those generated by blat. The pblat tool is easy to install and can run on Linux and Mac OS systems. In addition, we provide a cluster version of pblat (pblat-cluster) running on computing clusters with MPI support.
pblat is open source and free available for non-commercial users. It is easy to install and easy to use. pblat and pblat-cluster would facilitate the high-throughput mapping of large scale genomic and transcript sequences to reference genomes with both high speed and high precision.
KeywordsSequence alignment Genome annotation Parallel computing Cluster computing
General parallel file system
Message passing interface
Blat  is a sequence alignment tool designed to map DNA, RNA and protein sequences to reference genomes. It is commonly used to locate sequences in a reference genome, find homologous sequences from genomes of closely related species, identify exon-intron boundaries from mRNA sequences and determine gene structures, and to help assemble and annotate genome and transcriptome sequences . Although many fast sequence aligners, like BWA  and Bowtie , have been developed to map short sequence reads generated by high-throughput sequencing, they are not capable of mapping long reads or sequences with abundant gaps or spliced sequences from discontinues genomic regions . In contrast, blat is an ideal tool for such applications with its high sensitivity and precision [6, 7].
However, with the increasing quantity of sequences generated by high throughput sequencing projects, blat cannot meet the speed requirements needed for large-scale analysis and regularly updated annotations. For example, when used to map the whole transcriptome sequences of vertebrates to a reference genome, it would take days to finish using blat. This is due to the blat algorithm being single threaded and, thus, not taking full advantage of modern multicore processors. One might use the GNU parallel  tool to execute multiple instances of blat in parallel using one or more computers. However, each blat process would load a copy of the whole reference genome and build and store the index of the genome into memory, which might exceed the available physical memory of conventional computers if multiple blat processes run simultaneously.
To overcome these limitations, we present pblat (parallel blat), which functions to speed up blat alignments by implementing multiple thread and cluster computing support. With pblat, all threads share the same memory copy of the whole reference genome and the index. As such, pblat utilizes almost the same amount of memory as blat. The run time is reduced with the number of threads used, and the output results of pblat are identical with that of blat. The cluster version of pblat (pblat-cluster) runs on computer clusters with MPI (Message Passing Interface) support, which is able to help reduce the run time of blat from days to minutes.
pblat extends the blat algorithm with multiple thread support by employing POSIX threads via the C programming language. pblat employs data-level parallelism. The input query file in FASTA format is virtually divided into the same number of parts as the number of threads specified using the ‘-threads’ command line option. Each part is comprised of the same number of query sequences to load balance among threads. Each thread takes one part of the input sequences and performs the blat algorithm to align the sequences to the genome. Only one copy of the genome sequences is loaded into memory and all the threads share this memory to query the input sequences against the genome. This makes the memory consumption of pblat keep to the minimum no matter how many threads are used. The outputs of each thread are written to an independent temporary file. After all the threads finish, all of the temporary output files are combined to form the final output file. This ensures that the order of output records corresponds to the order of query sequences in the input file no matter how many threads are used. All of the global variables and static variables in the original blat program are localized to ensure all the variables and subroutines are thread safe.
The cluster version of pblat extends the multithread version of pblat with MPI support. The master process of the pblat-cluster distributes the virtual input query file parts to all of the spawned processes in each computing node via MPI. The spawned processes running in the same computing node are automatically merged and switched to the multithread mode of pblat, sharing the same memory copy of the reference genome and index to minimize the memory requirement per computing node. After all the spawned processes in each computing node finish, the master process combines all of the output files generated by each process to form the final output file.
Performance evaluation of pbalt
We evaluated the performance of pblat using different number of threads and compared to the results of the original blat. All analyses were performed on a Linux machine with 4 Intel Xeon E7–4830 @ 2.13GHz CPUs (32 cores in total, 2 threads per core) and 128G memory. We employed the nucleotide sequences of all human protein-coding transcripts with length range from 1 kb to 5 kb in the GENCODE release 26 . We aligned these sequences to the GRCh38.p10 human reference genome sequences with blat and pblat. The test data consisted of 42,286 transcript sequences. The mean length of the test transcripts was 2440 and the median length was 2226. The blat and pblat analyses, with 2, 4, 8, 16, 24, 32, 40, 48, 56 and 64 threads, were executed to map all the transcripts to the genome.
To evaluate the performance of pblat on real long-read sequencing data with sequencing errors leading to abundant mismatches and gaps, we adopted the mouse single-cell full length transcript RNA-seq data sequenced on the long-read single-molecule Oxford Nanopore MinION sequencer (ONT) . The dataset was downloaded from NCBI Sequence Read Archive (SRA) with accession number SRR5286960. For a single murine B1a cell, 104,990 reads were generated by ONT with the maximum length 6296 bp, mean length 1868 bp and median length 1787 bp. Fast spliced read aligners including HISAT2 , STAR  etc., were not compatible with such long reads with high error rates . In the original study , the blat was employed to align these nanopore long reads to the mouse genome and successfully helped identify novel transcript isoforms. We replicated the aligning step using blat and pblat with 8 and 16 threads. The blat program took 935 min to align all these reads to GRCm38 genome (primary assembly), while pblat with 8 threads used 149 min and pblat with 16 threads took 86 min to finish. The speedup for pblat with 8 and 16 threads relative to blat was 6.28 and 10.87, respectively. The speedup was consistent with results in the last analysis. These results showed pblat could significantly accelerate aligning long sequencing reads generated by the Oxford Nanopore and PacBio SMRT (Single Molecule Real-Time) sequencing platforms.
Performance evaluation of pblat-cluster
We then compared the time consumption when running pblat with 12 threads and that when running pblat-cluster with 12 nodes (1 thread per node). The pblat with 12 threads took 44 min to align all the test transcripts to the reference genome. The pblat-cluster with 12 nodes used 39.8 min. As expected, pblat-cluster is faster than pblat using the same number of threads, because for pblat-cluster the threads run on different computing nodes, with each node using its own memory of genomes and indexes, they do not compete to access the memory.
pblat is open source and the source code is easy to compile by simply typing ‘make’ to generate the binary program on Linux or Mac OS systems. pblat is also easy to use, and all the command-line options are the same as blat, with only the addition of the ‘-threads’ option to specify the number of threads to use. Pipelines employing blat could directly switch to using pblat. pblat enables users to take advantage of both the high precision of blat and the high speed available with other popular sequence aligners. Overall, pblat facilitates the rapid analysis and annotation of whole genome and whole transcriptome sequences.
Availability and requirements
Project name: pblat
Operating systems: Linux, Mac OS X
Programming language: C
License: The source code and executables are freely available for academic, nonprofit and personal use. Commercial licensing information is available on the Kent Informatics website (http://www.kentinformatics.com).
Any restrictions to use by non-academics: license needed
We appreciate the early users of pblat for their valuable feedback towards improving the program. Part of the analysis was performed on the Computing Platform of the Center for Life Science, Peking University.
This work was supported by the National Key Research and Development Program of China (2017YFC1201200, 2016YFB0201700, 2017YFC0908400) and the International Collaboration Program for Proteome Biological Big Data and Standard System (2014DFB30030). MW was supported in part by the Postdoctoral Fellowship of Peking-Tsinghua Center for Life Sciences.
Availability of data and materials
LK and MW conceived the project. LK and MW designed the program. MW implemented the software and performed evaluations. MW and LK wrote the manuscript. Both authors read and approved the final manuscript.
Ethics approval and consent to participate
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
- 8.Tange O: GNU Parallel - The Command-Line Power Tool. ;The USENIX Magazine 2011(2):42–47.Google Scholar
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.