Moderated Resource Elasticity for Stream Processing Applications

  • Michael BorkowskiEmail author
  • Christoph Hochreiner
  • Stefan Schulte
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10659)


In stream processing, elasticity is often realized by adapting the system scale and topology according to the volume of input data. However, this volume is often fluctuating, with a high degree of noise, which can trigger a high amount of scaling operations. Since these scaling operations introduce additional overhead and cost, systems employing such approaches are at risk of spending a significant amount of time scaling up and down, nullifying the positive effects of scalability.

To overcome this, we propose an approach for moderating the scaling behavior of stream processing applications by reducing the number of scaling operations, while still providing quick responses to changes in input data volume. Contrary to existing approaches, instead of using linear smoothing techniques, we show how to employ non-linear filtering techniques from the field of signal processing to pre-process the raw volume measurements, mitigating superfluous scaling operations, and effectively reducing the number of such operations by up to 94%.


Stream processing Elasticity TVD EKF 



This work is partially supported by the Commission of the European Union within the CREMA H2020-RIA project (Grant agreement no. 637066) and by TU Wien research funds.


  1. 1.
    Abadi, D.J., et al.: The design of the borealis stream processing engine. In: CIDR, vol. 5, pp. 277–289 (2005)Google Scholar
  2. 2.
    Beloglazov, A., Buyya, R.: Energy efficient allocation of virtual machines in cloud data centers. In: International Conference on Cluster, Cloud and Grid Computing, pp. 577–578. IEEE (2010)Google Scholar
  3. 3.
    Buyya, R., Ranjan, R., Calheiros, R.N.: InterCloud: utility-oriented federation of cloud computing environments for scaling of application services. In: Hsu, C.-H., Yang, L.T., Park, J.H., Yeo, S.-S. (eds.) ICA3PP 2010 Part I. LNCS, vol. 6081, pp. 13–31. Springer, Heidelberg (2010). CrossRefGoogle Scholar
  4. 4.
    Castro Fernandez, R., et al.: Integrating scale out and fault tolerance in stream processing using operator state management. In: International Conference on Management of Data, pp. 725–736. ACM (2013)Google Scholar
  5. 5.
    Corradi, A., Fanelli, M., Foschini, L.: VM consolidation: a real case based on OpenStack Cloud. Future Gener. Comput. Syst. 32, 118–127 (2014)CrossRefGoogle Scholar
  6. 6.
    Dustdar, S., et al.: Principles of elastic processes. Internet Comput. 15(5), 66–71 (2011)CrossRefzbMATHGoogle Scholar
  7. 7.
    Figueiredo, M.A.T., et al.: On total variation denoising: a new majorization-minimization algorithm and an experimental comparison with wavalet denoising. In: International Conference on Image Processing, pp. 2633–2636. IEEE (2006)Google Scholar
  8. 8.
    Ghahremani, E., Kamwa, I.: Dynamic state estimation in power system by applying the extended Kalman filter with unknown inputs to phasor measurements. Trans. Power Syst. 26(4), 2556–2566 (2011)CrossRefGoogle Scholar
  9. 9.
    Gong, Z., Gu, X., Wilkes, J.: PRESS: predictive elastic resource scaling for cloud systems. In: International Conference on Network and Service Management (CNSM), pp. 9–16. IEEE (2010)Google Scholar
  10. 10.
    Gulisano, V., et al.: StreamCloud: an elastic and scalable data streaming system. Trans. Parallel Distrib. Syst. 23(12), 2351–2365 (2012)CrossRefGoogle Scholar
  11. 11.
    Heinze, T., et al.: Online parameter optimization for elastic data stream processing. In: Symposium on Cloud Computing, pp. 276–287. ACM, New York (2015)Google Scholar
  12. 12.
    Hochreiner, C., et al.: Elastic stream processing for distributed environments. Internet Comput. 19(6), 54–59 (2015)CrossRefGoogle Scholar
  13. 13.
    Hochreiner, C., et al.: Elastic stream processing for the Internet of Things. In: International Conference on Cloud Computing (CLOUD), pp. 100–107 (2016)Google Scholar
  14. 14.
    Islam, S., et al.: How a consumer can measure elasticity for cloud platforms. In: 3rd International Conference on Performance Engineering, pp. 85–96. ACM/SPEC (2012)Google Scholar
  15. 15.
    Kalman, R.E., Bucy, R.S.: New results in linear filtering and prediction theory. J. Basic Eng. 83(3), 95–108 (1961)MathSciNetCrossRefGoogle Scholar
  16. 16.
    Khan, A., et al.: Workload characterization and prediction in the cloud: a multiple time series approach. In: Network Operations and Management Symposium (NOMS), pp. 1287–1294. IEEE (2012)Google Scholar
  17. 17.
    Little, M.A., Jones, N.S.: Sparse Bayesian step-filtering for high-throughput analysis of molecular machine dynamics. In: International Conference on Acoustics Speech and Signal Processing (ICASSP), pp. 4162–4165. IEEE (2010)Google Scholar
  18. 18.
    Mao, M., Humphrey, M.: Auto-scaling to minimize cost and meet application deadlines in cloud workows. In: International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1–12. IEEE (2011)Google Scholar
  19. 19.
    Mencagli, G., Vanneschi, M., Vespa, E.: A cooperative predictive control approach to improve the reconfiguration stability of adaptive distributed parallel applications. Trans. Auton. Adapt. Syst. (TAAS) 9(1), 2 (2014)Google Scholar
  20. 20.
    Micchelli, C.A., Shen, L., Xu, Y.: Proximity algorithms for image models: denoising. Inverse Probl. 27(4), 45009–45038 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
  21. 21.
    Rudin, L.I., Osher, S., Fatemi, E.: Nonlinear total variation based noise removal algorithms. Phys. D: Nonlinear Phenom. 60(1–4), 259–268 (1992)MathSciNetCrossRefzbMATHGoogle Scholar
  22. 22.
    Selesnick, I.: Total variation denoising (an MM algorithm). In: NYU Polytechnic School of Engineering Lecture Notes (2012)Google Scholar
  23. 23.
    Shen, Z., et al.: CloudScale: elastic resource scaling for multi-tenant cloud systems. In: 2nd Symposium on Cloud Computing, pp. 5–18. ACM (2011)Google Scholar
  24. 24.
    Thomas, P.J., Midgley, P.A.: Image-spectroscopy-I. The advantages of increased spectral information for compositional EFTEM analysis. Ultramicroscopy 88(3), 179–186 (2001)CrossRefGoogle Scholar
  25. 25.
    Xu, J., et al.: T-storm: traffic-aware online scheduling in storm. In: 34th International Conference on Distributed Computing Systems (ICDCS), pp. 535–544. IEEE (2014)Google Scholar
  26. 26.
    Zhang, Q., Cheng, L., Boutaba, R.: Cloud computing: state-of-the-art and research challenges. J. Internet Serv. Appl. 1(1), 7–18 (2010)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Distributed Systems GroupTU WienViennaAustria

Personalised recommendations