Skip to main content
SpringerLink
Log in

QAAS: quick accurate auto-scaling for streaming processing

  • Research Article
  • Published:
Frontiers of Computer Science Aims and scope Submit manuscript

Abstract

In recent years, the demand for real-time data processing has been increasing, and various stream processing systems have emerged. When the amount of data input to the stream processing system fluctuates, the computing resources required by the stream processing job will also change. The resources used by stream processing jobs need to be adjusted according to load changes, avoiding the waste of computing resources. At present, existing works adjust stream processing jobs based on the assumption that there is a linear relationship between the operator parallelism and operator resource consumption (e.g., throughput), which makes a significant deviation when the operator parallelism increases. This paper proposes a nonlinear model to represent operator performance. We divide the operator performance into three stages, the Non-competition stage, the Non-full competition stage, and the Full competition stage. Using our proposed performance model, given the parallelism of the operator, we can accurately predict the CPU utilization and operator throughput. Evaluated with actual experiments, the prediction error of our model is below 5%. We also propose a quick accurate auto-scaling (QAAS) method that uses the operator performance model to implement the auto-scaling of the operator parallelism of the Flink job. Compared to previous work, QAAS is able to maintain stable job performance under load changes, minimizing the number of job adjustments and reducing data backlogs by 50%.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Vastardis N, Yang K. Mobile social networks: architectures, social properties, and key research challenges. IEEE Communications Surveys & Tutorials, 2013, 15(3): 1355–1371

    Article  Google Scholar 

  2. Chen H, Chiang R H L, Storey V C. Business intelligence and analytics: from big data to big impact. MIS Quarterly, 2012, 36(4): 1165–1188

    Article  Google Scholar 

  3. Carbone P, Katsifodimos A, Ewen S, Markl V, Haridi S, Tzoumas K. Apache flinkTM: stream and batch processing in a single engine. IEEE Data Engineering Bulletin, 2015, 38(4): 28–38

    Google Scholar 

  4. Kulkarni S, Bhagat N, Fu M, Kedigehalli V, Kellogg C, Mittal S, Patel J M, Ramasamy K, Taneja S. Twitter heron: stream processing at scale. In: Proceedings of 2015 ACM SIGMOD International Conference on Management of Data. 2015, 239–250

  5. Toshniwal A, Taneja S, Shukla A, Ramasamy K, Patel J M, Kulkarni S, Jackson J, Gade K, Fu M, Donham J, Bhagat N, Mittal S, Ryaboy D. Storm@twitter. In: Proceedings of 2014 ACM SIGMOD International Conference on Management of Data. 2014, 147–156

  6. Song Y, Li Y, Yang H, Xu J, Luan Z, Li W. Adaptive watermark generation mechanism based on time series prediction for stream processing. Frontiers of Computer Science, 2021, 15(6): 156213

    Article  Google Scholar 

  7. Kahveci B, Gedik B. Joker: elastic stream processing with organic adaptation. Journal of Parallel and Distributed Computing, 2020, 137: 205–223

    Article  Google Scholar 

  8. Kalim F, Cooper T, Wu H, Li Y, Wang N, Lu N, Fu M, Qian X, Luo H, Cheng D, Wang Y, Dai F, Ghosh M, Wang B. Caladrius: a performance modelling service for distributed stream processing systems. In: Proceedings of the 35th IEEE International Conference on Data Engineering (ICDE). 2019, 1886–1897

  9. Kalavri V, Liagouris J, Hoffmann M, Dimitrova D C, Forshaw M, Roscoe T. Three steps is all you need: fast, accurate, automatic scaling decisions for distributed streaming dataflows. In: Proceedings of the 13th USENIX Symposium on Operating Systems Design and Implementation. 2018, 783–798

  10. Mei Y, Cheng L, Talwar V, Levin M Y, Jacques-Silva G, Simha N, Banerjee A, Smith B, Williamson T, Yilmaz S, Chen W, Chen G J. Turbine: Facebook’s service management platform for stream processing. In: Proceedings of the 36th IEEE International Conference on Data Engineering (ICDE). 2020, 1591–1602

  11. Kreps J, Narkhede N, Rao J. Kafka: a distributed messaging system for log processing. In: Proceedings of the NetDB. 2011, 1–7

  12. Burns B, Grant B, Oppenheimer D, Brewer E, Wilkes J. Borg, omega, and kubernetes. Communications of the ACM, 2016, 59(5): 50–57

    Article  Google Scholar 

  13. Vavilapalli V K, Murthy A C, Douglas C, Agarwal S, Konar M, Evans R, Graves T, Lowe J, Shah H, Seth S, Saha B, Curino C, O’Malley O, Radia S, Reed B, Baldeschwieler E. Apache hadoop YARN: yet another resource negotiator. In: Proceedings of the 4th Annual Symposium on Cloud Computing. 2013, 5

  14. Roger H, Mayer R. A comprehensive survey on parallelization and elasticity in stream processing. ACM Computing Surveys, 2020, 52(2): 36

    Article  Google Scholar 

  15. Floratou A, Agrawal A, Graham B, Rao S, Ramasamy K. Dhalion: self-regulating stream processing in heron. Proceedings of the VLDB Endowment, 2017, 10(12): 1825–1836

    Article  Google Scholar 

  16. Marangozova-Martin V, de Palma N, El Rheddane A. Multi-level elasticity for data stream processing. IEEE Transactions on Parallel and Distributed Systems, 2019, 30(10): 2326–2337

    Article  Google Scholar 

  17. Arkian H, Pierre G, Tordsson J, Elmroth E. An experiment-driven performance model of stream processing operators in fog computing environments. In: Proceedings of the 35th Annual ACM Symposium on Applied Computing. 2020, 1763–1771

Download references

Acknowledgements

This work was supported by the National Key Research and Development Program of China (2020YFB1506703), the National Natural Science Foundation of China (Grant No. 62072018), the State Key Laboratory of Software Development Environment (SKLSDE-2021ZX-06), and the Fundamental Research Funds for the Central Universities.

Author information

Authors and Affiliations

  1. State Key Laboratory of Software Development Environment, Beijing, 100191, China

    Shiyuan Liu, Hailong Yang & Wei Li

  2. School of Computer Science and Engineering, Beihang University, Beijing, 100191, China

    Shiyuan Liu, Yunchun Li & Hailong Yang

  3. School of Cyber Science and Technology, Beihang University, Beijing, 100191, China

    Yunchun Li, Ming Dun & Chen Chen

  4. School of Computer Science and Technology, Xi’an Jiaotong University, Xi’an, 710049, China

    Huaitao Zhang

Authors
  1. Shiyuan Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yunchun Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hailong Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ming Dun
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Chen Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Huaitao Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Wei Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hailong Yang.

Additional information

Shiyuan Liu is a master student in School of Computer Science and Engineering, Beihang University, China. He is currently working on big data processing optimization. His research interests include data streaming systems.

Yunchun Li is a professor in School of Computer Science and Engineering, Beihang University, China. He received the PhD degree in Computer Science from Beihang University, China. He went to University of Illinois at Urbana-Champaign (UIUC), USA as a visiting scholar in 2010. His research interests include big data, cloud computing, and parallel computing.

Hailong Yang is an associate professor in School of Computer Science and Engineering, Beihang University, China. He received the PhD degree in the School of Computer Science and Engineering, Beihang University, China in 2014. His research interests include parallel and distributed computing, HPC, performance optimization and energy efficiency.

Ming Dun is a master student in School of Cyber Science and Technology, Beihang University, China. She is currently working on performance optimization for large scale applications. Her research interests include parallel computing and HPC.

Chen Chen is studying for PhD degree in School of Cyber Science and Technology, Beihang University, China. His current research interests include Web security, machine learning, cloud computing, and attack detection.

Huaitao Zhang is an undergraduate student in School of Computer Science and Technology, Xi’an Jiaotong University, China. His research interest is high performance computing.

Wei Li is an associate professor with the School of Computer Science and Engineering, Beihang University, China. She received the PhD degree in computer science from Beihang University, China. Her research interests include network measurement, network virtualization, and cloud computing.

Electronic Supplementary Material

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, S., Li, Y., Yang, H. et al. QAAS: quick accurate auto-scaling for streaming processing. Front. Comput. Sci. 18, 181201 (2024). https://doi.org/10.1007/s11704-022-1706-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11704-022-1706-4

Keywords

Search

Navigation