Skip to main content

A Bulk-Parallel Priority Queue in External Memory with STXXL

  • Conference paper
  • First Online:
Experimental Algorithms (SEA 2015)

Part of the book series: Lecture Notes in Computer Science ((LNTCS,volume 9125))

Included in the following conference series:

Abstract

We propose the design and an implementation of a bulk-parallel external memory priority queue to take advantage of both shared-memory parallelism and high external memory transfer speeds to parallel disks. To achieve higher performance by decoupling item insertions and extractions, we offer two parallelization interfaces: one using “bulk” sequences, the other by defining “limit” items. In the design, we discuss how to parallelize insertions using multiple heaps, and how to calculate a dynamic prediction sequence to prefetch blocks and apply parallel multiway merge for extraction. Our experimental results show that in the selected benchmarks the priority queue reaches 64% of the full parallel I/O bandwidth of SSDs and 49% of rotational disks, or the speed of sorting in external memory when bounded by computation.

This paper is a short version of the technical report [6].

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Similar content being viewed by others

References

  1. Alistarh, D., Kopinsky, J., Li, J., Shavit, N.: The SprayList: A scalable relaxed priority queue. Tech. Rep. MSR-TR-2014-16, Microsoft Research, September 2014

    Google Scholar 

  2. Arge, L.: The buffer tree: A technique for designing batched external data structures. Algorithmica 37(1), 1–24 (2003)

    Article  MATH  MathSciNet  Google Scholar 

  3. Arge, L., Goodrich, M.T., Nelson, M., Sitchinava, N.: Fundamental parallel algorithms for private-cache chip multiprocessors. In: SPAA, pp. 197–206. ACM (2008)

    Google Scholar 

  4. Beckmann, A., Dementiev, R., Singler, J.: Building a parallel pipelined external memory algorithm library. In: IPDPS 2009, pp. 1–10. IEEE (2009)

    Google Scholar 

  5. Bingmann, T., Fischer, J., Osipov, V.: Inducing suffix and LCP arrays in external memory. In: ALENEX 2013, pp. 88–102. SIAM (2013)

    Google Scholar 

  6. Bingmann, T., Keh, T., Sanders, P.: A bulk-parallel priority queue in external memory with STXXL, April 2015. see ArXiv e-print arXiv:1504.00545

  7. Brodal, G.S., Katajainen, J.: Worst-case efficient external-memory priority queues. In: Arnborg, S. (ed.) SWAT 1998. LNCS, vol. 1432, pp. 107–118. Springer, Heidelberg (1998)

    Chapter  Google Scholar 

  8. Chiang, Y.J., Goodrich, M.T., Grove, E.F., Tamassia, R., Vengroff, D.E., Vitter, J.S.: External-memory graph algorithms. In: SODA 1995, pp. 139–149. SIAM (1995)

    Google Scholar 

  9. Dementiev, R., Kettner, L., Sanders, P.: STXXL: Standard template library for XXL data sets. Software & Practice and Experience 38(6), 589–637 (2008)

    Article  Google Scholar 

  10. Dementiev, R., Sanders, P.: Asynchronous parallel disk sorting. In: SPAA 2003, pp. 138–148. ACM (2003)

    Google Scholar 

  11. Deo, N., Prasad, S.: Parallel heap: An optimal parallel priority queue. The Journal of Supercomputing 6(1), 87–98 (1992)

    Article  MATH  Google Scholar 

  12. Hutchinson, D.A., Sanders, P., Vitter, J.S.: Duality between prefetching and queued writing with parallel disks. SIAM Journal on Computing 34(6) (2005)

    Google Scholar 

  13. Keh, T.: Bulk-parallel priority queue in external memory, Bachelor Thesis, Karlsruhe Institute of Technology, Germany (2014)

    Google Scholar 

  14. Petersen, L.H.: External Priority Queues in Practice. Master’s thesis, Aarhus Universitet, Datalogisk Institut, Denmark (2007)

    Google Scholar 

  15. Pinotti, M.C., Pucci, G.: Parallel priority queues. IPL 40(1), 33–40 (1991)

    MATH  MathSciNet  Google Scholar 

  16. Rihani, H., Sanders, P., Dementiev, R.: Multiqueues: Simpler, faster, and better relaxed concurrent priority queues. arXiv preprint arXiv:1411.1209 (2014)

  17. Sanders, P.: Randomized priority queues for fast parallel access. Journal of Parallel and Distributed Computing 49(1), 86–97 (1998)

    Article  MATH  Google Scholar 

  18. Sanders, P.: Fast priority queues for cached memory. JEA 5, 7 (2000)

    Article  Google Scholar 

  19. Singler, J., Sanders, P., Putze, F.: MCSTL: the multi-core standard template library. In: Kermarrec, A.-M., Bougé, L., Priol, T. (eds.) Euro-Par 2007. LNCS, vol. 4641, pp. 682–694. Springer, Heidelberg (2007)

    Chapter  Google Scholar 

  20. Varman, P.J., Scheufler, S.D., Iyer, B.R., Ricard, G.R.: Merging multiple lists on hierarchical-memory multiprocessors. Journal of Parallel and Distributed Computing 12(2), 171–177 (1991)

    Article  MATH  Google Scholar 

  21. Vitter, J.S., Shriver, E.A.: Algorithms for parallel memory, i: Two-level memories. Algorithmica 12(2–3), 110–147 (1994)

    Article  MATH  MathSciNet  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Timo Bingmann .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2015 Springer International Publishing Switzerland

About this paper

Cite this paper

Bingmann, T., Keh, T., Sanders, P. (2015). A Bulk-Parallel Priority Queue in External Memory with STXXL. In: Bampis, E. (eds) Experimental Algorithms. SEA 2015. Lecture Notes in Computer Science(), vol 9125. Springer, Cham. https://doi.org/10.1007/978-3-319-20086-6_3

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-20086-6_3

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-20085-9

  • Online ISBN: 978-3-319-20086-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics