COBS: A Compact Bit-Sliced Signature Index

  • Timo BingmannEmail author
  • Phelim Bradley
  • Florian Gauger
  • Zamin Iqbal
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11811)


We present COBS, a COmpact Bit-sliced Signature index, which is a cross-over between an inverted index and Bloom filters. Our target application is to index k-mers of DNA samples or q-grams from text documents and process approximate pattern matching queries on the corpus with a user-chosen coverage threshold. Query results may contain a number of false positives which decreases exponentially with the query length. We compare COBS to seven other index software packages on 100 000 microbial DNA samples. COBS’ compact but simple data structure outperforms the other indexes in construction time and query performance with Mantis by Pandey et al. in second place. However, unlike Mantis and other previous work, COBS does not need the complete index in RAM and is thus designed to scale to larger document sets.


  1. 1.
    Almodaresi, F., Pandey, P., Patro, R.: Rainbowfish: a succinct colored de Bruijn graph representation. In: 17th International Workshop on Algorithms in Bioinformatics (WABI). LIPIcs, vol. 88, pp. 18:1–18:15. Schloss Dagstuhl, August 2017. preprint bioRxiv:138016Google Scholar
  2. 2.
    Almodaresi, F., Sarkar, H., Srivastava, A., Patro, R.: A space and time-efficient index for the compacted colored de Bruijn graph. Bioinformatics 34(13), i169–i177 (2018)CrossRefGoogle Scholar
  3. 3.
    Bingmann, T.: NVMe “disk” bandwidth and latency for batched block requests, March 2019. Online Article,
  4. 4.
    Bloom, B.H.: Space/time trade-offs in hash coding with allowable errors. Commun. ACM 13(7), 422–426 (1970)CrossRefGoogle Scholar
  5. 5.
    Bradley, P., den Bakker, H.C., Rocha, E.P.C., McVean, G., Iqbal, Z.: Ultrafast search of all deposited bacterial and viral genomic data. Nat. Biotechnol. 37, 152–159 (2019)CrossRefGoogle Scholar
  6. 6.
    Brin, S., Page, L.: The anatomy of a large-scale hypertextual web search engine. Comput. Networks ISDN Syst. 30(1–7), 107–117 (1998)CrossRefGoogle Scholar
  7. 7.
    Broder, A.Z., Mitzenmacher, M.: Network applications of Bloom filters: a survey. Internet Math. 1(4), 485–509 (2003)MathSciNetCrossRefGoogle Scholar
  8. 8.
    Chikhi, R., Holub, J., Medvedev, P.: Data structures to represent sets of \(k\)-long DNA sequences. Computing Research Repository (CoRR), arXiv:1903.12312:1–16, March 2019
  9. 9.
    Collet, Y.: xxHash: extremely fast non-cryptographic hash algorithm, 2014. Git repository. Accessed July 2019
  10. 10.
    Cook, C.E., Lopez, R., Stroe, O., Cochrane, G., Brooksbank, C., Birney, E., Apweiler, R.: The European Bioinformatics Institute in 2018: tools, infrastructure and training. Nucleic Acids Res. 47(D1), D15–D22 (2019)CrossRefGoogle Scholar
  11. 11.
    Crainiceanu, A., Lemire, D.: Bloofi: multidimensional bloom filters. Inf. Syst. 54, 311–324 (2015)CrossRefGoogle Scholar
  12. 12.
    Faloutsos, C., Christodoulakis, S.: Signature files: an access method for documents and its analytical performance evaluation. ACM Trans. Inf. Syst. (TOIS) 2(4), 267–288 (1984)CrossRefGoogle Scholar
  13. 13.
    Gauger, F.: Engineering a compact bit-sliced signature index for approximate search on genomic data. Master Thesis. Karlsruhe Institute of Technology, Germany, February 2018Google Scholar
  14. 14.
    Gog, S., Beller, T., Moffat, A., Petri, M.: From theory to practice: plug and play with succinct data structures. In: Gudmundsson, J., Katajainen, J. (eds.) SEA 2014. LNCS, vol. 8504, pp. 326–337. Springer, Cham (2014). Scholar
  15. 15.
    Goodwin, B., et al.: BitFunnel: revisiting signatures for search. In: 40th ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 605–614. ACM, August 2017Google Scholar
  16. 16.
    Harris, R.S., Medvedev, P.: Improved representation of sequence Bloom trees. bioRxiv, pp. 501452, December 2018Google Scholar
  17. 17.
    Harrison, P.W., et al.: The european nucleotide archive in 2018. Nucleic Acids Res. D47(1), D84–D88 (2019)CrossRefGoogle Scholar
  18. 18.
    Heinz, S., Zobel, J., Williams, H.E.: Burst tries: a fast, efficient data structure for string keys. ACM Trans. Inf. Syst. (TOIS) 20(2), 192–223 (2002)CrossRefGoogle Scholar
  19. 19.
    Holley, G., Wittler, R., Stoye, J.: Bloom filter trie: an alignment-free and reference-free data structure for pan-genome storage. Algorithms Mol. Biol. 11(1), 3 (2016)CrossRefGoogle Scholar
  20. 20.
    Iqbal, Z., Caccamo, M., Turner, I., Flicek, P., McVean, G.: De novo assembly and genotyping of variants using colored de Bruijn graphs. Nat. Genet. 44(2), 226 (2012)CrossRefGoogle Scholar
  21. 21.
    Iqbal, Z., Turner, I., McVean, G.: High-throughput microbial population genomics using the cortex variation assembler. Bioinformatics 29(2), 275–276 (2012)CrossRefGoogle Scholar
  22. 22.
    Krugel, J.: Approximate Pattern Matching with Index Structures. Ph.D. thesis, Technische Universität München, Germany, February 2016Google Scholar
  23. 23.
    Marçais, G., Kingsford, C.: A fast, lock-free approach for efficient parallel counting of occurrences of \(k\)-mers. Bioinformatics 27(6), 764–770 (2011)Google Scholar
  24. 24.
    Mitzenmacher, M., Upfal, E.: Probability and Computing: Randomized Algorithms and Probabilistic Analysis. Cambridge University Press, Cambridge (2005)CrossRefGoogle Scholar
  25. 25.
    Mohamadi, H., Khan, H., Birol, I.: ntCard: a streaming algorithm for cardinality estimation in genomics data. Bioinformatics 33(9), 1324–1330 (2017)Google Scholar
  26. 26.
    Muggli, M.D., et al.: Succinct colored de Bruijn graphs. Bioinformatics 33(20), 3181–3187 (2017). preprint bioRxiv:040071CrossRefGoogle Scholar
  27. 27.
    Navarro, G., Baeza-Yates, R.A., Sutinen, E., Tarhio, J.: Indexing methods for approximate string matching. IEEE Bull. Tech. Committee Data Eng. 24(4), 19–27 (2001)Google Scholar
  28. 28.
    Pandey, P., Almodaresi, F., Bender, M.A., Ferdman, M., Johnson, R., Patro, R.: Mantis: a fast, small, and exact large-scale sequence-search index. Cell Systems, June 2018. preprint bioRxiv:217372Google Scholar
  29. 29.
    Pandey, P., Bender, M.A., Johnson, R., Patro, R.: A general-purpose counting filter: making every bit count. In: ACM International Conference on Management of Data, pp. 775–787. ACM (2017)Google Scholar
  30. 30.
    Pandey, P., Bender, M.A., Johnson, R., Patro, R.: Squeakr: an exact and approximate k-mer counting system. Bioinformatics 34(4), 568–575 (2018). preprint bioRxiv:122077CrossRefGoogle Scholar
  31. 31.
    Raman, R., Raman, V., Srinivasa Rao, S.: Succinct indexable dictionaries with applications to encoding \(k\)-ary trees and multisets. In: 13th ACM-SIAM Symposium on Discrete Algorithms (SODA), pp. 233–242. SIAM, January 2002Google Scholar
  32. 32.
    Solomon, B., Kingsford, C.: Fast search of thousands of short-read sequencing experiments. Nat. Biotechnol. 34(3), 300–312 (2016)CrossRefGoogle Scholar
  33. 33.
    Solomon, B., Kingsford, C.: Improved search of large transcriptomic sequencing databases using split sequence Bloom trees. J. Comput. Biol. 25(7), 755–765 (2018)MathSciNetCrossRefGoogle Scholar
  34. 34.
    Sun, C., Harris, R.S., Chikhi, R., Medvedev, P.: AllSome sequence Bloom trees. J. Computat. Biol. 25(5), 467–479 (2018)CrossRefGoogle Scholar
  35. 35.
    Turner, I., Garimella, K.V., Iqbal, Z., McVean, G.: Integrating long-range connectivity information into de Bruijn graphs. Bioinformatics 34(15), 2556–2565 (2018)CrossRefGoogle Scholar
  36. 36.
    Ukkonen, E.: Approximate string-matching with \(q\)-grams and maximal matches. Theoret. Comput. Sci. 92(1), 191–211 (1992)Google Scholar
  37. 37.
    Wong, H.K.T., Liu, H.-F., Olken, F., Rotem, D., Wong, L.: Bit transposed files. In 11th International Conference on Very Large Data Bases (VLDB), pp. 448–457. VLDB Endowment, August 1985Google Scholar
  38. 38.
    Ye, Y., Belazzougui, D., Qian, C., Zhang, Q.: Memory-efficient and ultra-fast network lookup and forwarding using othello hashing. IEEE/ACM Trans. Networking 26(3), 1151–1164 (2018)CrossRefGoogle Scholar
  39. 39.
    Ye, Y., et al.: SeqOthello: querying RNA-seq experiments at scale. Genome Biol. 19(1), 167 (2018). preprint bioRxiv:258772CrossRefGoogle Scholar
  40. 40.
    Zobel, J., Moffat, A.: Inverted files for text search engines. ACM Comput. Surveys (CSUR) 38(2), 6 (2006)CrossRefGoogle Scholar
  41. 41.
    Zobel, J., Moffat, A., Ramamohanarao, K.: Inverted files versus signature files for text indexing. ACM Trans. Database Syst. (TODS) 23(4), 453–490 (1998)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Timo Bingmann
    • 1
    Email author
  • Phelim Bradley
    • 2
  • Florian Gauger
    • 1
  • Zamin Iqbal
    • 2
  1. 1.Institute of Theoretical InformaticsKarlsruhe Institute of TechnologyKarlsruheGermany
  2. 2.European Molecular Biology LaboratoryEuropean Bioinformatics InstituteCambridgeUK

Personalised recommendations