Sprouter: Dynamic Graph Processing over Data Streams at Scale

  • Tariq Abughofa
  • Farhana ZulkernineEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11030)


Graph data is becoming dominant for many applications such as social networks, targeted advertising, and web indexing. As a result of that, advances in machine learning and data mining techniques depend tightly on the ability to process this data structure efficiently and reliably. Despite the importance of processing dynamic graphs in real-time, it remains a challenge to maintain such graphs and process them over data streams. We propose Sprouter, an end-to-end framework which enable storing enormous graph data, allows updates in real-time, and supports efficient complex analytics in addition to OLTP queries. We demonstrate that our framework can ingest and process streaming data efficiently using a scalable multi-cluster distributed architecture, apply incremental graph updates, and store the dynamic graph for fast query performance. Experiments showed the system is able to update graphs having up to 100 million edges in under 50 s in a moderate underlying cluster. As we use all open source tools, the framework can be easily extended in the future with other equivalent software.


Graph processing Stream data processing Big data management and analytics 


  1. 1.
    Abughofa, T., Zulkernine, F.: Towards online graph processing with spark streaming. In: IEEE International Conference on Big Data, pp. 2787–2794. IEEE (2017)Google Scholar
  2. 2.
    Choudhury, S., et al.: NOUS: Construction and querying of dynamic knowledge graphs. In: 2017 IEEE 33rd International Conference on Data Engineering (ICDE), pp. 1563–1565. IEEE (2017)Google Scholar
  3. 3.
    Dave, A., Jindal, A., Li, L.E., Xin, R., Gonzalez, J., Zaharia, M.: Graphframes: an integrated API for mixing graph and relational queries. In: Proceedings of the Fourth International Workshop on Graph Data Management Experiences and Systems, p. 2. ACM (2016)Google Scholar
  4. 4.
    Gonzalez, J.E., Xin, R.S., Dave, A., Crankshaw, D., Franklin, M.J., Stoica, I.: Graphx: Graph processing in a distributed dataflow framework. In: OSDI, vol. 14, pp. 599–613 (2014)Google Scholar
  5. 5.
    Iyer, A.P., Li, L.E., Das, T., Stoica, I.: Time-evolving graph processing at scale. In: Proceedings of the Fourth International Workshop on Graph Data Management Experiences and Systems, p. 5. ACM (2016)Google Scholar
  6. 6.
    Malewicz, G., et al.: Pregel: a system for large-scale graph processing. In: Proceedings of the 2010 ACM SIGMOD International Conference on Management of Data, pp. 135–146. ACM (2010)Google Scholar
  7. 7.
    Sharma, A., Jiang, J., Bommannavar, P., Larson, B., Lin, J.: Graphjet: real-time content recommendations at twitter. Proc. VLDB Endow. 9(13), 1281–1292 (2016)CrossRefGoogle Scholar
  8. 8.
    Yin, S., et al.: Node-grained incremental community detection for streaming networks. In: IEEE 28th International Conference on Tools with Artificial Intelligence, pp. 585–592. IEEE (2016)Google Scholar
  9. 9.
    Zaharia, M., et al.: Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing. In: Proceedings of the 9th USENIX Conference on Networked Systems Design and Implementation, p. 2. USENIX Association (2012)Google Scholar
  10. 10.
    Zaharia, M., Das, T., Li, H., Hunter, T., Shenker, S., Stoica, I.: Discretized streams: fault-tolerant streaming computation at scale. In: Proceedings of the 24th ACM Symposium on Operating Systems Principles, pp. 423–438. ACM (2013)Google Scholar
  11. 11.
    Zaharia, M., et al.: Apache spark: a unified engine for big data processing. Commun. ACM 59(11), 56–65 (2016)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  1. 1.Queen’s UniversityKingstonCanada

Personalised recommendations