Deterministic Model for Distributed Speculative Stream Processing

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11019)


Users of modern distributed stream processing systems have to choose between non-deterministic computations and high latency due to a need in excessive buffering. We introduce a speculative model based on MapReduce-complete set of operations that allows us to achieve determinism and low-latency. Experiments show that our prototype can outperform existing solutions due to low overhead of optimistic synchronization.


Data streams Distributed processing Drifting state 


  1. 1.
    Akidau, T., et al.: Millwheel: fault-tolerant stream processing at internet scale. Proc. VLDB 6(11), 1033–1044 (2013)CrossRefGoogle Scholar
  2. 2.
    Akidau, T., et al.: The dataflow model: a practical approach to balancing correctness, latency, and cost in massive-scale, unbounded, out-of-order data processing. Proc. VLDB 8(12), 1792–1803 (2015)CrossRefGoogle Scholar
  3. 3.
    Apache storm, October 2017.
  4. 4.
    Arasu, A., Babu, S., Widom, J.: The CQL continuous query language: semantic foundations and query execution. The VLDB J. 15(2), 121–142 (2006)CrossRefGoogle Scholar
  5. 5.
    Babu, S., Srivastava, U., Widom, J.: Exploiting k-constraints to reduce memory overhead in continuous queries over data streams. ACM Trans. Database Syst. 29(3), 545–580 (2004). Scholar
  6. 6.
    Carbone, P., Ewen, S., Fóra, G., Haridi, S., Richter, S., Tzoumas, K.: State management in apache flink®: consistent stateful distributed stream processing. Proc. VLDB 10(12), 1718–1729 (2017)CrossRefGoogle Scholar
  7. 7.
    Carbone, P., Katsifodimos, A., Ewen, S., Markl, V., Haridi, S., Tzoumas, K.: Apache flink: stream and batch processing in a single engine. Bull. IEEE Comput. Soc. Tech. Committee Data Eng. 36(4) (2015)Google Scholar
  8. 8.
    Chintapalli, S., et al.: Benchmarking streaming computation engines: storm, flink and spark streaming. In: 2016 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), pp. 1789–1792, May 2016Google Scholar
  9. 9.
    Cranor, C., Johnson, T., Spataschek, O., Shkapenyuk, V.: Gigascope: a stream database for network applications. In: Proceedings of the 2003 ACM SIGMOD International Conference on Management of Data, SIGMOD 2003, pp. 647–651. ACM, New York (2003).
  10. 10.
    Jacques-Silva, G., et al.: Consistent regions: guaranteed tuple processing in IBM streams. Proc. VLDB Endow. 9(13), 1341–1352 (2016)CrossRefGoogle Scholar
  11. 11.
    Li, C.W., Gu, Y., Yu, G., Hong, B.: Aggressive complex event processing with confidence over out-of-order streams. J. Comput. Sci. Technol. 26(4), 685–696 (2011). Scholar
  12. 12.
    Li, J., Tufte, K., Shkapenyuk, V., Papadimos, V., Johnson, T., Maier, D.: Out-of-order processing: a new architecture for high-performance stream systems. Proc. VLDB Endow. 1(1), 274–288 (2008)CrossRefGoogle Scholar
  13. 13.
    Murray, D.G., McSherry, F., Isaacs, R., Isard, M., Barham, P., Abadi, M.: Naiad: a timely dataflow system. In: Proceedings of the Twenty-Fourth ACM Symposium on Operating Systems Principles, SOSP 2013, pp. 439–455. ACM, New York (2013).
  14. 14.
    Noghabi, S.A., et al.: Samza: stateful scalable stream processing at LinkedIn. Proc. VLDB Endow. 10(12), 1634–1645 (2017)CrossRefGoogle Scholar
  15. 15.
    Srivastava, U., Widom, J.: Flexible time management in data stream systems. In: Proceedings of the PODS, pp. 263–274. ACM, New York (2004)Google Scholar
  16. 16.
    Stonebraker, M., Çetintemel, U., Zdonik, S.: The 8 requirements of real-time stream processing. SIGMOD Rec. 34(4), 42–47 (2005)CrossRefGoogle Scholar
  17. 17.
    Tucker, P.A., Maier, D., Sheard, T., Fegaras, L.: Exploiting punctuation semantics in continuous data streams. IEEE Trans. Knowl. Data Eng. 15(3), 555–568 (2003). Scholar
  18. 18.
    Wei, M., Liu, M., Li, M., Golovnya, D., Rundensteiner, E.A., Claypool, K.: Supporting a spectrum of out-of-order event processing technologies: from aggressive to conservative methodologies. In: Proceedings of the 2009 ACM SIGMOD International Conference on Management of Data, pp. 1031–1034. ACM, New York (2009)Google Scholar
  19. 19.
    Zacheilas, N., Kalogeraki, V., Nikolakopoulos, Y., Gulisano, V., Papatriantafilou, M., Tsigas, P.: Maximizing determinism in stream processing under latency constraints. In: Proceedings of the 11th ACM International Conference on Distributed and Event-based Systems, DEBS 2017, pp. 112–123. ACM, New York (2017)Google Scholar
  20. 20.
    Zaharia, M., Das, T., Li, H., Shenker, S., Stoica, I.: Discretized streams: an efficient and fault-tolerant model for stream processing on large clusters. In: Proceedings of the 4th USENIX Conference on Hot Topics in Cloud Ccomputing, HotCloud 2012, p. 10. USENIX Association, Berkeley (2012)Google Scholar
  21. 21.
    Zaharia, M., et al.: Apache spark: a unified engine for big data processing. Commun. ACM 59(11), 56–65 (2016)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  1. 1.JetBrains ResearchSt. PetersburgRussia
  2. 2.Saint Petersburg State UniversitySt. PetersburgRussia

Personalised recommendations