Advertisement

Cost of Fault-Tolerance on Data Stream Processing

  • Valerio Vianello
  • Marta Patiño-Martínez
  • Ainhoa Azqueta-Alzúaz
  • Ricardo Jimenez-Péris
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11339)

Abstract

Data streaming engines process data on the fly in contrast to databases that first, store the data and then, they process it. In order to process the increasing amount of data produced every day, data streaming engines run on top of a distributed system. In this setting failures will likely happen. Current distributed data streaming engines like Apache Flink provide fault tolerance. In this paper we evaluate the impact on performance of fault tolerance mechanisms of Flink during regular operation (when there are no failures) on a distributed system and the impact on performance when there are failures. We use the Intel HiBench for conducting the evaluation.

Keywords

Data streaming Fault tolerance Evaluation HiBench 

Notes

Acknowledgments

This research has been partially funded by the European Commission under projects CloudDBAppliance, CrowdHealth and BigDataStack (grants H2020-732051, H2020-727560 and H2020-779747), the Madrid Regional Council, FSE and FEDER, project Cloud4BigData (grant S2013TIC2894), the Ministry of Economy and Competitiveness (MINECO) under project CloudDB (grant TIN2016-80350).

References

  1. 1.
    Apache flink. https://flink.apache.org/. Accessed 11 May 2018
  2. 2.
    Apache hadoop. http://hadoop.apache.org/. Accessed 11 May 2018
  3. 3.
    Apache kafka. https://kafka.apache.org/. Accessed 11 May 2018
  4. 4.
    Balazinska, M., Balakrishnan, H., Madden, S., Stonebraker, M.: Fault-tolerance in the borealis distributed stream processing system. ACM Trans. Database Syst. 33(1), 3:1–3:44 (2008)CrossRefGoogle Scholar
  5. 5.
    Flink an overview of end-to-end exactly-once processing in apache flink. https://flink.apache.org/features/2018/03/01/end-to-end-exactly-once-apache-flink.html. Accessed 11 May 2018
  6. 6.
  7. 7.
  8. 8.
    Gulisano, V., Jiménez-Peris, R., Patiño-Martínez, M., Soriente, C., Valduriez, P.: Streamcloud: an elastic and scalable data streaming system. IEEE Trans. Parallel Distrib. Syst. 23(12), 2351–2365 (2012)CrossRefGoogle Scholar
  9. 9.
    Hibench, a big data benchmark suite. https://github.com/intel-hadoop/HiBench. Accessed 11 May 2018
  10. 10.
    Huang, S., Huang, J., Dai, J., Xie, T., Huang, B.: The HiBench benchmark suite: characterization of the MapReduce-based data analysis. In: 22nd International Conference on Data Engineering Workshops, pp. 41–51 (2010).  https://doi.org/10.1109/icdew.2010.5452747
  11. 11.
    Kwon, Y., Balazinska, M., Greenberg, A.: Fault-tolerant stream processing using a distributed, replicated file system. Proc. VLDB Endow. 1(1), 574–585 (2008).  https://doi.org/10.14778/1453856.1453920CrossRefGoogle Scholar
  12. 12.
    Rabbitmq. https://www.rabbitmq.com/. Accessed 11 May 2018

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Valerio Vianello
    • 1
  • Marta Patiño-Martínez
    • 1
  • Ainhoa Azqueta-Alzúaz
    • 1
  • Ricardo Jimenez-Péris
    • 2
  1. 1.Universidad Politécnica de MadridMadridSpain
  2. 2.LeanXcaleMadridSpain

Personalised recommendations