A Mechanism for Stream Program Performance Recovery in Resource Limited Compute Clusters

  • Miyuru Dayarathna
  • Toyotaro Suzumura
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7826)


Replication, the widely adapted technique for crash fault tolerance introduces additional infrastructural costs for resource limited clusters. In this paper we take a different approach for maintaining stream program performance during crash failures. It is based on the concepts of automatic code generation. Albatross, the middleware we introduce for this task maintains the same performance level during crash failures based on predetermined priority values assigned to each stream program. Albatross constructs different versions of the input stream programs (sample programs) with different levels of performance characteristics, and assigns the best performing programs for normal operations. During node failure or node recovery, potential use of a different version of sample program is evaluated in order to bring the performance of each job back to its original level. We evaluated effectiveness of this approach with three different real world stream computing applications on System S distributed stream processing platform. We show that our approach is capable of maintaining stream program performance even if half of the nodes of the cluster has been crashed using both Apnoea, and Regex applications.


stream computing data-intensive computing reliability highly availability performance auto-scaling autonomic computing automatic code generation 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Andrade, H., Gedik, B., Wu, K.-L., Yu, P.S.: Scale-up strategies for processing high-rate data streams in systems. In: IEEE 25th International Conference on Data Engineering, ICDE 2009, March 29-April 2, pp. 1375–1378 (2009)Google Scholar
  2. 2.
    Babcock, B., Datar, M., Motwani, R.: Load shedding in data stream systems. In: Data Streams, vol. 31, pp. 127–147. Springer US (2007)Google Scholar
  3. 3.
    Catley, C., et al.: A framework to model and translate clinical rules to support complex real-time analysis of physiological and clinical data. In: Proceedings of the 1st ACM International Health Informatics Symposium, IHI 2010, pp. 307–315. ACM, New York (2010)Google Scholar
  4. 4.
    Dayarathna, M., Suzumura, T.: Hirundo: a mechanism for automated production of optimized data stream graphs. In: Proceedings of the Third Joint WOSP/SIPEW International Conference on Performance Engineering, ICPE 2012, pp. 335–346. ACM, New York (2012)CrossRefGoogle Scholar
  5. 5.
    Furht, B., Escalante, A.: Handbook of Cloud Computing. Springer-Verlag New York, Inc. (2010)Google Scholar
  6. 6.
    Gedik, B., et al.: Spade: the system s declarative stream processing engine. In: SIGMOD 2008, pp. 1123–1134. ACM, New York (2008)Google Scholar
  7. 7.
    Gu, Y., Zhang, Z., Ye, F., Yang, H., Kim, M., Lei, H., Liu, Z.: An empirical study of high availability in stream processing systems. In: Middleware 2009, pp. 23:1–23:9. Springer-Verlag New York, Inc., New York (2009)Google Scholar
  8. 8.
    Hwang, J.-H., Cetintemel, U., Zdonik, S.: Fast and highly-available stream processing over wide area networks, pp. 804–813 (April 2008)Google Scholar
  9. 9.
    Hwang, J.-H., et al.: High-availability algorithms for distributed stream processing. In: Proceedings of the 21st International Conference on Data Engineering, ICDE 2005, pp. 779–790. IEEE Computer Society, Washington, DC (2005)Google Scholar
  10. 10.
    IBM. Ibm infosphere streams version 1.2.1: Installation and administration guide (October 2010)Google Scholar
  11. 11.
    Ishii, A., Suzumura, T.: Elastic stream computing with clouds. In: 2011 IEEE International Conference on Cloud Computing (CLOUD), pp. 195–202 (July 2011)Google Scholar
  12. 12.
    Khandekar, R., Hildrum, K., Parekh, S., Rajan, D., Wolf, J., Wu, K.-L., Andrade, H., Gedik, B.: COLA: Optimizing stream processing applications via graph partitioning. In: Bacon, J.M., Cooper, B.F. (eds.) Middleware 2009. LNCS, vol. 5896, pp. 308–327. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  13. 13.
    Kleiminger, W., Kalyvianaki, E., Pietzuch, P.: Balancing load in stream processing with the cloud. In: Data Engineering Workshops (ICDEW), pp. 16–21 (April 2011)Google Scholar
  14. 14.
    Logothetis, D., Trezzo, C., Webb, K.C., Yocum, K.: In-situ mapreduce for log processing. In: USENIXATC 2011, Berkeley, CA, USA, p. 9. USENIX Association, Berkeley (2011)Google Scholar
  15. 15.
    Tanenbaum, A.S., Steen, M.V.: Distributed Systems. Pearson Education, Inc. (2007)Google Scholar
  16. 16.
    White, T.: Hadoop: The Definitive Guide. O’Reilly Media, Inc. (2010)Google Scholar
  17. 17.
    Wolf, J., et al.: Soda: an optimizing scheduler for large-scale stream-based distributed computer systems. In: Middleware 2008, pp. 306–325. Springer-Verlag New York, Inc., New York (2008)CrossRefGoogle Scholar
  18. 18.
    Zhang, Z., et al.: A hybrid approach to high availability in stream processing systems. In: Distributed Computing Systems (ICDCS), pp. 138–148 (June 2010)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Miyuru Dayarathna
    • 1
  • Toyotaro Suzumura
    • 1
    • 2
  1. 1.Department of Computer ScienceTokyo Institute of TechnologyMeguro-kuJapan
  2. 2.IBM ResearchTokyoJapan

Personalised recommendations