Skip to main content
SpringerLink
Log in

Challenging SQL-on-Hadoop Performance with Apache Druid

  • Conference paper
  • First Online:
Business Information Systems (BIS 2019)

Part of the book series: Lecture Notes in Business Information Processing ((LNBIP,volume 353))

Included in the following conference series:

Abstract

In Big Data, SQL-on-Hadoop tools usually provide satisfactory performance for processing vast amounts of data, although new emerging tools may be an alternative. This paper evaluates if Apache Druid, an innovative column-oriented data store suited for online analytical processing workloads, is an alternative to some of the well-known SQL-on-Hadoop technologies and its potential in this role. In this evaluation, Druid, Hive and Presto are benchmarked with increasing data volumes. The results point Druid as a strong alternative, achieving better performance than Hive and Presto, and show the potential of integrating Hive and Druid, enhancing the potentialities of both tools.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 69.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 89.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. IBM, Zikopoulos, P., Eaton, C.: Understanding Big Data: Analytics for Enterprise Class Hadoop and Streaming Data, 1st edn. McGraw-Hill Osborne Media (2011)

    Google Scholar 

  2. Ward, J.S., Barker, A.: Undefined by data: a survey of big data definitions. CoRR, abs/1309.5821 (2013)

    Google Scholar 

  3. Madden, S.: From databases to big data. IEEE Internet Comput. 16(3), 4–6 (2012)

    Article  Google Scholar 

  4. Krishnan, K.: Data Warehousing in the Age of Big Data, 1st edn. Morgan Kaufmann Publishers Inc., San Francisco (2013)

    Google Scholar 

  5. Costa, C., Santos, M.Y.: Evaluating several design patterns and trends in big data warehousing systems. In: Krogstie, J., Reijers, H.A. (eds.) CAiSE 2018. LNCS, vol. 10816, pp. 459–473. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-91563-0_28

    Chapter  Google Scholar 

  6. Rodrigues, M., Santos, M.Y., Bernardino, J.: Big data processing tools: an experimental performance evaluation. Wiley Interdisc. Rev. Data Min. Knowl. Discov. 9, e1297 (2019)

    Article  Google Scholar 

  7. Cuzzocrea, A., Bellatreche, L., Song, I.-Y.: Data warehousing and OLAP over big data: current challenges and future research directions. In: Proceedings of the Sixteenth International Workshop on Data Warehousing and OLAP, New York, USA, pp. 67–70 (2013)

    Google Scholar 

  8. Yang, F., Tschetter, E., Léauté, X., Ray, N., Merlino, G., Ganguli, D.: Druid: a real-time analytical data store. In: Proceedings of the 2014 ACM SIGMOD International Conference on Management of Data, pp. 157–168 (2014)

    Google Scholar 

  9. Santos, M.Y., et al.: Evaluating SQL-on-Hadoop for big data warehousing on not-so-good hardware. In: ACM International Conference Proceeding Series, vol. Part F1294, pp. 242–252 (2017)

    Google Scholar 

  10. Costa, E., Costa, C., Santos, M.Y.: Partitioning and bucketing in hive-based big data warehouses. In: Rocha, Á., Adeli, H., Reis, L.P., Costanzo, S. (eds.) WorldCIST’18 2018. AISC, vol. 746, pp. 764–774. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-77712-2_72

    Chapter  Google Scholar 

  11. Chambi, S., Lemire, D., Godin, R., Boukhalfa, K., Allen, C.R., Yang, F.: Optimizing druid with roaring bitmaps. In: ACM International Conference Proceeding Series, 11–13 July 2016, pp. 77–86 (2016)

    Google Scholar 

  12. Correia, J., Santos, M.Y., Costa, C., Andrade, C.: Fast online analytical processing for big data warehousing. Presented at the IEEE 9th International Conference on Intelligent Systems (2018)

    Google Scholar 

  13. O’Neil, P.E., O’Neil, E.J., Chen, X.: The Star Schema Benchmark (SSB) (2009)

    Google Scholar 

  14. Kimball, R., Ross, M.: The Data Warehouse Toolkit: The Definitive Guide to Dimensional Modeling. Wiley, Hoboken (2013)

    Google Scholar 

  15. LLAP - Apache Hive - Apache Software Foundation. https://cwiki.apache.org/confluence/display/Hive/LLAP. Accessed 07 Nov 2018

  16. Druid Integration - Apache Hive - Apache Software Foundation. https://cwiki.apache.org/confluence/display/Hive/Druid+Integration. Accessed 07 Nov 2018

  17. Ultra-fast OLAP Analytics with Apache Hive and Druid - Part 1 of 3, Hortonworks, 11 May 2017. https://hortonworks.com/blog/apache-hive-druid-part-1-3/. Accessed 07 Nov 2018

Download references

Acknowledgements

This work is supported by COMPETE: POCI-01-0145- FEDER-007043 and FCT – Fundação para a Ciência e Tecnologia within Project UID/CEC/00319/2013 and by European Structural and Investment Funds in the FEDER component, COMPETE 2020 (Funding Reference: POCI-01-0247-FEDER-002814).

Author information

Authors and Affiliations

  1. ALGORITMI Research Centre, University of Minho, Guimarães, Portugal

    José Correia, Carlos Costa & Maribel Yasmina Santos

  2. NATIXIS, on Behalf of Altran Portugal, Porto, Portugal

    José Correia

  3. Centre for Computer Graphics - CCG, Guimarães, Portugal

    Carlos Costa

Authors
  1. José Correia
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Carlos Costa
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Maribel Yasmina Santos
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to José Correia .

Editor information

Editors and Affiliations

  1. Poznań University of Economics and Business, Poznań, Poland

    Witold Abramowicz

  2. ETSI Informática, University of Seville, Seville, Spain

    Rafael Corchuelo

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Correia, J., Costa, C., Santos, M.Y. (2019). Challenging SQL-on-Hadoop Performance with Apache Druid. In: Abramowicz, W., Corchuelo, R. (eds) Business Information Systems. BIS 2019. Lecture Notes in Business Information Processing, vol 353. Springer, Cham. https://doi.org/10.1007/978-3-030-20485-3_12

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-20485-3_12

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-20484-6

  • Online ISBN: 978-3-030-20485-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation