BGElasor: Elastic-Scaling Framework for Distributed Streaming Processing with Deep Neural Network

  • Weimin Mu
  • Zongze JinEmail author
  • Junwei Wang
  • Weilin Zhu
  • Weiping Wang
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11783)


In face of constant fluctuations and sudden bursts of data stream, elasticity of distributed stream processing system has become increasingly important. The proactive policy offers a powerful means to realize the effective elastic scaling. The existing methods lack the latent features of data stream, it leads the poor prediction. Furthermore, the poor prediction results in the high cost of adaptation and the instability. To address these issues, we propose the framework named BGElasor, which is a proactive and low-cost elastic-scaling framework based on the accurate prediction using deep neural networks. It can capture the potentially-complicated pattern to enhance the accuracy of prediction, reduce the cost of adaptation and avoid adaptation bumps. The experimental results show that BGElasor not only improves the prediction accuracy with three kinds of typical loads, but also ensure the end-to-end latency on QoS with low cost.


Data stream processing Load prediction Deep neural network Gated recurrent units Elasticity 


  1. 1.
    Arasu, A., et al.: STREAM: the stanford stream data manager. IEEE Data Eng. Bull. 26(1), 19–26 (2003)MathSciNetGoogle Scholar
  2. 2.
    Abadi, D.J., et al.: The design of the borealis stream processing engine. In: CIDR 2005, Second Biennial Conference on Innovative Data Systems Research, pp. 277–289 (2005)Google Scholar
  3. 3.
    Neumeyer, L., Robbins, B., Nair, A., Kesari, A.: S4: distributed stream computing platform. In: ICDMW 2010, The 10th IEEE International Conference on Data Mining Workshops, pp. 170–177 (2010)Google Scholar
  4. 4.
  5. 5.
    Carbone, P., Katsifodimos, A., Ewen, S., Markl, V., Haridi, S., Tzoumas, K.: Apache flink™: stream and batch processing in a single engine. IEEE Data Eng. Bull. 38(4), 28–38 (2015)Google Scholar
  6. 6.
    Fernandez, R.C., Migliavacca, M., Kalyvianaki, E., Pietzuch, P.R.: Integrating scale out and fault tolerance in stream processing using operator state management. In: Proceedings of the ACM SIGMOD International Conference on Management of Data, SIGMOD 2013, pp. 725–736 (2013)Google Scholar
  7. 7.
    Gulisano, V., Jiménez-Peris, R., Patiño-Martínez, M., Soriente, C., Valduriez, P.: Streamcloud: an elastic and scalable data streaming system. IEEE Trans. Parallel Distrib. Syst. 23(12), 2351–2365 (2012)CrossRefGoogle Scholar
  8. 8.
    Gedik, B., Schneider, S., Hirzel, M., Wu, K.: Elastic scaling for data stream processing. IEEE Trans. Parallel Distrib. Syst. 25(6), 1447–1463 (2014)CrossRefGoogle Scholar
  9. 9.
    Herbst, N.R., Huber, N., Kounev, S., Amrehn, E.: Self-adaptive workload classification and forecasting for proactive resource provisioning. In: ACM/SPEC International Conference on Performance Engineering, ICPE 2013, pp. 187–198 (2013)Google Scholar
  10. 10.
    Balkesen, C., Tatbul, N., Özsu, M.T.: Adaptive input admission and management for parallel stream processing. In: The 7th ACM International Conference on Distributed Event-Based Systems, DEBS 2013, pp. 15–26 (2013)Google Scholar
  11. 11.
    Zacheilas, N., Kalogeraki, V., Zygouras, N., Panagiotou, N., Gunopulos, D.: Elastic complex event processing exploiting prediction. In: 2015 IEEE International Conference on Big Data, Big Data 2015, pp. 213–222 (2015)Google Scholar
  12. 12.
    Repantis, T., Kalogeraki, V.: Hot-spot prediction and alleviation in distributed stream processing applications. In: The 38th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2008, pp. 346–355 (2008)Google Scholar
  13. 13.
    Hidalgo, N., Wladdimiro, D., Rosas, E.: Self-adaptive processing graph with operator fission for elastic stream processing. J. Syst. Softw. 127, 205–216 (2017)CrossRefGoogle Scholar
  14. 14.
    Xing, Y., Hwang, J., Çetintemel, U., Zdonik, S.B.: Providing resiliency to load variations in distributed stream processing. In: Proceedings of the 32nd International Conference on Very Large Data Bases, pp. 775–786 (2006)Google Scholar
  15. 15.
    Xing, Y., Zdonik, S.B., Hwang, J.: Dynamic load distribution in the borealis stream processor. In: Proceedings of the 21st International Conference on Data Engineering, ICDE 2005, pp. 791–802 (2005)Google Scholar

Copyright information

© IFIP International Federation for Information Processing 2019

Authors and Affiliations

  • Weimin Mu
    • 1
    • 2
  • Zongze Jin
    • 1
    • 2
    Email author
  • Junwei Wang
    • 1
    • 2
  • Weilin Zhu
    • 2
  • Weiping Wang
    • 2
  1. 1.Institute of Information EngineeringChinese Academy of SciencesBeijingChina
  2. 2.School of Cyber SecurityUniversity of Chinese Academy of SciencesBeijingChina

Personalised recommendations