Abstract
The increasing size of big data and the speed with which it is generated have put a tremendous burden on cloud storage and communication systems. Network traffic and server capacity are crucial to having systems that are cost aware during big data stream processing in Software-Defined Network (SDN) enabled cloud environment. The common approach to address this problem has been through various optimization techniques. In this paper, we propose SDN based cost optimization approach to address the problem. Although SDN has been shown to improve cloud system performance, there is little attention given to SDN-based cost optimization approach to address the challenges of the increasing big data. To this end, we used Spark Streaming Processing approach (SSP). The proposed cost optimization approach is based on SDN within the cloud environment and focuses on optimizing the communication and computational costs. We performed extensive experiments to valid the approach and compared it with a Spark Streaming approach. The results of the experiment show that the proposed approach has better cost optimization than the baseline approach.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Abawajy, J.: Comprehensive analysis of big data variety landscape. Int. J. Parallel Emergent Distrib. Syst. 30(1), 5–14 (2015)
Chowdhury, M., Abawajy, J., Kelarev, A., Jelinek, H.: A clustering-based multi-layer distributed ensemble for neurological diagnostics in cloud services. IEEE Trans. Cloud Comput. 8, 473–483 (2016)
Shojafar, M., Canali, C., Lancellotti, R., Abawajy, J.: Adaptive computing-plus-communication optimization framework for multimedia processing in cloud systems. IEEE Trans. Cloud Comput. 8(4), 1162–1175 (2020). https://doi.org/10.1109/TCC.2016.2617367
Zhou, Z., et al.: Minimizing SLA violation and power consumption in cloud data centers using adaptive energy-aware algorithms. Future Gener. Comput. Syst. 86, 836–850 (2018)
Wang, Y., Wang, X., Li, H., Dong, Y., Liu, Q., Shi, X.: A multi-service differentiation traffic management strategy in SDN cloud data center. Comput. Netw. 171, 107143 (2020)
Bouras, C., Ntarzanos, P., Papazois, A.: Cost modeling for SDN/NFV based mobile 5G networks. In: 2016 8th International Congress on Ultra Modern Telecommunications and Control Systems and Workshops (ICUMT), pp. 56–61. IEEE (2016)
Gu, L., Zeng, D., Guo, S., Xiang, Y., Hu, J.: A general communication cost optimization framework for big data stream processing in geo-distributed data centers. IEEE Trans. Comput. 65(1), 19–29 (2015)
Abawajy, J., Chowdhury, M., Kelarev, A.: Hybrid consensus pruning of ensemble classifiers for big data malware detection. IEEE Trans. Cloud Comput. 8(2), 398–407 (2020). https://doi.org/10.1109/TCC.2015.2481378
Sowmya, T.S.R.: Cost minimization for big data processing in geo-distributed data centers. Asia-Pac. J. Convergent Res. Interchange 2(4), 33–41 (2016)
Bhattacharya, M., Islam, R., Abawajy, J.: Evolutionary optimization: a big data perspective. J. Netw. Comput. Appl. 59, 416–426 (2016)
Cao, H., Wachowicz, M.: The design of an IoT-GIS platform for performing automated analytical tasks. Comput. Environ. Urban Syst. 74, 23–40 (2019)
Shah, S.A.R., et al.: AmoebaNet: an SDN-enabled network service for big data science. J. Netw. Comput. Appl. 119, 70–82 (2018)
Adami, D., et al.: An SDN orchestrator for cloud data center: system design and experimental evaluation. Trans. Emerg. Telecommun. Technol. 28(11), e3172 (2017)
Bagci, K.T., Tekalp, A.M.: SDN-enabled distributed open exchange: dynamic QoS-path optimization in multi-operator services. Comput. Netw. 162, 106845 (2019)
Vicentini, C., Santin, A., Viegas, E., Abreu, V.: SDN-based and multitenant-aware resource provisioning mechanism for cloud-based big data streaming. J. Netw. Comput. Appl. 126, 133–149 (2019)
Poobalan, A., Selvi, V.: Optimization of cost in cloud computing using OCRP algorithm. Int. J. Eng. Trends Technol. 4(5), 2105–2107 (2013)
Chen, W., Paik, I., Li, Z.: Cost-aware streaming workflow allocation on geo-distributed data centers. IEEE Trans. Comput. 66(2), 256–271 (2016)
Chen, W., Paik, I., Hung, P.C.: Transformation-based streaming workflow allocation on geo-distributed datacenters for streaming big data processing. IEEE Trans. Serv. Comput. 12, 654–668 (2016)
Zhao, G.: Cost-aware scheduling algorithm based on PSO in cloud computing environment. Int. J. Grid Distrib. Comput. 7(1), 33–42 (2014)
Habib ur Rehman, M., Jayaraman, P.P., Malik, S.U.R., Khan, A.U.R., Medhat Gaber, M.: Rededge: a novel architecture for big data processing in mobile edge computing environments. J. Sensor Actuator Netw. 6(3), 17 (2017)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 ICST Institute for Computer Sciences, Social Informatics and Telecommunications Engineering
About this paper
Cite this paper
Al-Mansoori, A., Abawajy, J., Chowdhury, M. (2021). Cost-Aware Big Data Stream Processing in Cloud Environment. In: Qi, L., Khosravi, M.R., Xu, X., Zhang, Y., Menon, V.G. (eds) Cloud Computing. CloudComp 2020. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, vol 363. Springer, Cham. https://doi.org/10.1007/978-3-030-69992-5_10
Download citation
DOI: https://doi.org/10.1007/978-3-030-69992-5_10
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-69991-8
Online ISBN: 978-3-030-69992-5
eBook Packages: Computer ScienceComputer Science (R0)