Advertisement

Challenging SQL-on-Hadoop Performance with Apache Druid

  • José CorreiaEmail author
  • Carlos Costa
  • Maribel Yasmina Santos
Conference paper
Part of the Lecture Notes in Business Information Processing book series (LNBIP, volume 353)

Abstract

In Big Data, SQL-on-Hadoop tools usually provide satisfactory performance for processing vast amounts of data, although new emerging tools may be an alternative. This paper evaluates if Apache Druid, an innovative column-oriented data store suited for online analytical processing workloads, is an alternative to some of the well-known SQL-on-Hadoop technologies and its potential in this role. In this evaluation, Druid, Hive and Presto are benchmarked with increasing data volumes. The results point Druid as a strong alternative, achieving better performance than Hive and Presto, and show the potential of integrating Hive and Druid, enhancing the potentialities of both tools.

Keywords

Big Data Big Data Warehouse SQL-on-Hadoop Druid OLAP 

Notes

Acknowledgements

This work is supported by COMPETE: POCI-01-0145- FEDER-007043 and FCT – Fundação para a Ciência e Tecnologia within Project UID/CEC/00319/2013 and by European Structural and Investment Funds in the FEDER component, COMPETE 2020 (Funding Reference: POCI-01-0247-FEDER-002814).

References

  1. 1.
    IBM, Zikopoulos, P., Eaton, C.: Understanding Big Data: Analytics for Enterprise Class Hadoop and Streaming Data, 1st edn. McGraw-Hill Osborne Media (2011)Google Scholar
  2. 2.
    Ward, J.S., Barker, A.: Undefined by data: a survey of big data definitions. CoRR, abs/1309.5821 (2013)Google Scholar
  3. 3.
    Madden, S.: From databases to big data. IEEE Internet Comput. 16(3), 4–6 (2012)CrossRefGoogle Scholar
  4. 4.
    Krishnan, K.: Data Warehousing in the Age of Big Data, 1st edn. Morgan Kaufmann Publishers Inc., San Francisco (2013)Google Scholar
  5. 5.
    Costa, C., Santos, M.Y.: Evaluating several design patterns and trends in big data warehousing systems. In: Krogstie, J., Reijers, H.A. (eds.) CAiSE 2018. LNCS, vol. 10816, pp. 459–473. Springer, Cham (2018).  https://doi.org/10.1007/978-3-319-91563-0_28CrossRefGoogle Scholar
  6. 6.
    Rodrigues, M., Santos, M.Y., Bernardino, J.: Big data processing tools: an experimental performance evaluation. Wiley Interdisc. Rev. Data Min. Knowl. Discov. 9, e1297 (2019)CrossRefGoogle Scholar
  7. 7.
    Cuzzocrea, A., Bellatreche, L., Song, I.-Y.: Data warehousing and OLAP over big data: current challenges and future research directions. In: Proceedings of the Sixteenth International Workshop on Data Warehousing and OLAP, New York, USA, pp. 67–70 (2013)Google Scholar
  8. 8.
    Yang, F., Tschetter, E., Léauté, X., Ray, N., Merlino, G., Ganguli, D.: Druid: a real-time analytical data store. In: Proceedings of the 2014 ACM SIGMOD International Conference on Management of Data, pp. 157–168 (2014)Google Scholar
  9. 9.
    Santos, M.Y., et al.: Evaluating SQL-on-Hadoop for big data warehousing on not-so-good hardware. In: ACM International Conference Proceeding Series, vol. Part F1294, pp. 242–252 (2017)Google Scholar
  10. 10.
    Costa, E., Costa, C., Santos, M.Y.: Partitioning and bucketing in hive-based big data warehouses. In: Rocha, Á., Adeli, H., Reis, L.P., Costanzo, S. (eds.) WorldCIST’18 2018. AISC, vol. 746, pp. 764–774. Springer, Cham (2018).  https://doi.org/10.1007/978-3-319-77712-2_72CrossRefGoogle Scholar
  11. 11.
    Chambi, S., Lemire, D., Godin, R., Boukhalfa, K., Allen, C.R., Yang, F.: Optimizing druid with roaring bitmaps. In: ACM International Conference Proceeding Series, 11–13 July 2016, pp. 77–86 (2016)Google Scholar
  12. 12.
    Correia, J., Santos, M.Y., Costa, C., Andrade, C.: Fast online analytical processing for big data warehousing. Presented at the IEEE 9th International Conference on Intelligent Systems (2018)Google Scholar
  13. 13.
    O’Neil, P.E., O’Neil, E.J., Chen, X.: The Star Schema Benchmark (SSB) (2009)Google Scholar
  14. 14.
    Kimball, R., Ross, M.: The Data Warehouse Toolkit: The Definitive Guide to Dimensional Modeling. Wiley, Hoboken (2013)Google Scholar
  15. 15.
    LLAP - Apache Hive - Apache Software Foundation. https://cwiki.apache.org/confluence/display/Hive/LLAP. Accessed 07 Nov 2018
  16. 16.
    Druid Integration - Apache Hive - Apache Software Foundation. https://cwiki.apache.org/confluence/display/Hive/Druid+Integration. Accessed 07 Nov 2018
  17. 17.
    Ultra-fast OLAP Analytics with Apache Hive and Druid - Part 1 of 3, Hortonworks, 11 May 2017. https://hortonworks.com/blog/apache-hive-druid-part-1-3/. Accessed 07 Nov 2018

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.ALGORITMI Research CentreUniversity of MinhoGuimarãesPortugal
  2. 2.NATIXIS, on Behalf of Altran PortugalPortoPortugal
  3. 3.Centre for Computer Graphics - CCGGuimarãesPortugal

Personalised recommendations