Skip to main content
SpringerLink
Log in

Distributed Graph Analytics with Datalog Queries in Flink

  • Conference paper
  • First Online:
Software Foundations for Data Interoperability and Large Scale Graph Data Analytics (SFDI 2020, LSGDA 2020)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1281))

Abstract

Large-scale, parallel graph processing has been in demand over the past decade. Succinct program structure and efficient execution are among the essential requirements of graph processing frameworks. In this paper, we present Cog, which executes Datalog programs on the Apache Flink distributed dataflow system. We chose Datalog for its compact program structure and Flink for its efficiency. We implemented a parallel semi-naive evaluation algorithm exploiting Flink’s delta iteration to propagate only the tuples that need to be further processed to the subsequent iterations. Flink’s delta iteration feature reduces the overhead present in acyclic dataflow systems, such as Spark, when evaluating recursive queries, hence making it more efficient. We demonstrated in our experiments that Cog outperformed BigDatalog, the state-of-the-art distributed Datalog evaluation system, in most of the tests.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Alexandrov, A., et al.: The stratosphere platform for big data analytics. VLDB J. 23(6), 939–964 (2014)

    Article  Google Scholar 

  2. Arni, F., Ong, K., Tsur, S., Wang, H., Zaniolo, C.: The deductive database system LDL++. Theory Pract. Log. Program. 3(1), 61–94 (2003)

    Article  MathSciNet  Google Scholar 

  3. Bancilhon, F.: Naive evaluation of recursively defined relations. In: Brodie, M.L., Mylopoulos, J. (eds.) On Knowledge Base Management Systems, pp. 165–178. Springer, New York (1986). https://doi.org/10.1007/978-1-4612-4980-1_17

    Chapter  Google Scholar 

  4. Bancilhon, F., Ramakrishnan, R.: An amateur’s introduction to recursive query processing strategies. In: Readings in Artificial Intelligence and Databases, pp. 376–430. Elsevier (1989)

    Google Scholar 

  5. Begoli, E., Camacho-Rodríguez, J., Hyde, J., Mior, M.J., Lemire, D.: Apache calcite: a foundational framework for optimized query processing over heterogeneous data sources. In: Proceedings of the 2018 International Conference on Management of Data, pp. 221–230 (2018)

    Google Scholar 

  6. Carbone, P., Katsifodimos, A., Ewen, S., Markl, V., Haridi, S., Tzoumas, K.: Apache flink: stream and batch processing in a single engine. Bull. IEEE Comput. Soc. Tech. Comm. Data Eng. 36(4), 28–38 (2015)

    Google Scholar 

  7. Ceri, S., Gottlob, G., Tanca, L.: What you always wanted to know about datalog (and never dared to ask). IEEE Trans. Knowl. Data Eng. 1(1), 146–166 (1989)

    Article  Google Scholar 

  8. Ewen, S., Tzoumas, K., Kaufmann, M., Markl, V.: Spinning fast iterative data flows. Proc. VLDB Endow. 5(11), 1268–1279 (2012)

    Article  Google Scholar 

  9. Fan, Z., Zhu, J., Zhang, Z., Albarghouthi, A., Koutris, P., Patel, J.: Scaling-up in-memory Datalog processing: observations and techniques. arXiv preprint arXiv:1812.03975 (2018)

  10. Gévay, G.E., Rabl, T., Breß, S., Madai-Tahy, L., Markl, V.: Labyrinth: compiling imperative control flow to parallel dataflows. arXiv preprint arXiv:1809.06845 (2018)

  11. Gonzalez, J.E., Xin, R.S., Dave, A., Crankshaw, D., Franklin, M.J., Stoica, I.: GraphX: graph processing in a distributed dataflow framework. In: 11th USENIX Symposium on Operating Systems Design and Implementation OSDI 14). pp. 599–613 (2014)

    Google Scholar 

  12. Gu, J., et al.: RaSQL: greater power and performance for big data analytics with recursive-aggregate-SQL on Spark. In: Proceedings of the 2019 International Conference on Management of Data, pp. 467–484 (2019)

    Google Scholar 

  13. Hajiyev, E., Verbaere, M., de Moor, O.: codeQuest: scalable source code queries with datalog. In: Thomas, D. (ed.) ECOOP 2006. LNCS, vol. 4067, pp. 2–27. Springer, Heidelberg (2006). https://doi.org/10.1007/11785477_2

    Chapter  Google Scholar 

  14. Malewicz, G., et al.: Pregel: a system for large-scale graph processing. In: Proceedings of the 2010 ACM SIGMOD International Conference on Management of data, pp. 135–146 (2010)

    Google Scholar 

  15. Murray, D.G., McSherry, F., Isaacs, R., Isard, M., Barham, P., Abadi, M.: Naiad: a timely dataflow system. In: Proceedings of the Twenty-Fourth ACM Symposium on Operating Systems Principles, pp. 439–455 (2013)

    Google Scholar 

  16. Ryzhyk, L., Budiu, M.: Differential datalog. In: Datalog 2.0 - 3rd International Workshop on the Resurgence of Datalog in Academia and Industry, CEUR-WS (2019)

    Google Scholar 

  17. Seo, J., Park, J., Shin, J., Lam, M.S.: Distributed sociaLite: a datalog-based language for large-scale graph analysis. Proc. VLDB Endow. 6(14), 1906–1917 (2013)

    Article  Google Scholar 

  18. Shkapsky, A., Yang, M., Interlandi, M., Chiu, H., Condie, T., Zaniolo, C.: Big data analytics with datalog queries on spark. In: SIGMOD, pp. 1135–1149 (2016)

    Google Scholar 

  19. Subotić, P., Jordan, H., Chang, L., Fekete, A., Scholz, B.: Automatic index selection for large-scale datalog computation. Proc. VLDB Endow. 12(2), 141–153 (2018)

    Article  Google Scholar 

  20. Wang, J., Balazinska, M., Halperin, D.: Asynchronous and fault-tolerant recursive datalog evaluation in shared-nothing engines. Proc. VLDB Endow. 8(12), 1542–1553 (2015)

    Article  Google Scholar 

  21. Zaharia, M., Chowdhury, M., Franklin, M.J., Shenker, S., Stoica, I., et al.: Spark: cluster computing with working sets. HotCloud 10(10–10), 95 (2010)

    Google Scholar 

  22. Zhang, Q., et al.: Optimizing declarative graph queries at large scale. In: Proceedings of the 2019 International Conference on Management of Data, pp. 1411–1428 (2019)

    Google Scholar 

  23. Apache Giraph. http://giraph.apache.org/. Accessed 12 Apr 2020

  24. Cog. https://github.com/imran-4/cog. Accessed 12 Apr 2020

  25. Gelly: Flink Graph API. https://ci.apache.org/projects/flink/flink-docs-stable/dev/libs/gelly/. Accessed 12 Apr 2020

  26. GTGraph. http://www.cse.psu.edu/~kxm85/software/GTgraph/. Accessed 12 Apr 2020

Download references

Acknowledgments

This work was funded by the German Ministry for Education and Research as BIFOLD (01IS18025A and 01IS18037A). We would like to thank Jorge-Arnulfo Quiané-Ruiz for helpful comments on a draft of this paper.

Author information

Authors and Affiliations

  1. Technische Universität Berlin, Berlin, Germany

    Muhammad Imran, Gábor E. Gévay & Volker Markl

Authors
  1. Muhammad Imran
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Gábor E. Gévay
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Volker Markl
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Muhammad Imran .

Editor information

Editors and Affiliations

  1. University of Technology Sydney, Sydney, NSW, Australia

    Lu Qin

  2. The University of New South Wales, Sydney, NSW, Australia

    Wenjie Zhang

  3. University of Technology Sydney, Sydney, NSW, Australia

    Ying Zhang

  4. The University of New South Wales, Sydney, NSW, Australia

    You Peng

  5. National Institute of Informatics, Tokyo, Japan

    Hiroyuki Kato

  6. The University of New South Wales, Sydney, NSW, Australia

    Wei Wang

  7. Osaka University, Osaka, Japan

    Chuan Xiao

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Imran, M., Gévay, G.E., Markl, V. (2020). Distributed Graph Analytics with Datalog Queries in Flink. In: Qin, L., et al. Software Foundations for Data Interoperability and Large Scale Graph Data Analytics. SFDI LSGDA 2020 2020. Communications in Computer and Information Science, vol 1281. Springer, Cham. https://doi.org/10.1007/978-3-030-61133-0_6

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-61133-0_6

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-61132-3

  • Online ISBN: 978-3-030-61133-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation