Skip to main content

Advertisement

SpringerLink
Log in

A Cost-Aware Management Framework for Placement of Data-Intensive Applications on Federated Cloud

  • Published:
Journal of Network and Systems Management Aims and scope Submit manuscript

Abstract

Modern big data analysis and business intelligence applications come with high resource demands, along with a requirement for large data transfer between storage and compute nodes. Since the available resources of a single data center might not be sufficient to host these applications, federated cloud systems present a promising solution. The objective of each cloud service provider in a federation is to maximize its own profit and to minimize its total operating cost. To achieve these objectives, spatial variation in the energy cost and bandwidth cost could be leveraged while allocating the workload. We propose a hierarchical approach for resource management in a federated cloud, catering to the requirements of each provider in the federation. We formulate the placement of a data-intensive applications as an optimization problem to minimize the total operating cost, including the energy and communication costs. We propose an algorithm to allocate the virtual components of a data-intensive application, in two phases; partitioning and mapping. While partitioning creates clusters of correlated nodes, mapping allocates the created clusters to data centers that minimize the total cost. Through extensive experiments in different scenarios, we demonstrate that the proposed algorithm achieves a significant reduction in the total operating cost.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Notes

  1. https://www.gartner.com/en/newsroom/pressreleases/2018-08-15-gartner-says-cloud-computing-remainstop-emerging-business-risk.

  2. https://www.nrdc.org/media/2014/140826.

  3. https://en.wikipedia.org/wiki/Electricity_pricing.

  4. https://www.energy.eu/electricity_natural-gas_prices_european_union, https://www.eia.gov/.

  5. https://www.google.com/about/datacenters/inside/locations, https://azure.microsoft.com/en-in/global-infrastructure/, https://aws.amazon.com/about-aws/global-infrastructure.

  6. https://www.vertatique.com/no-one-can-agree-typical-pue.

  7. https://www.datacenterdynamics.com/news/apac-data-center-survey-reveals-high-pue-figures-across-the-region/.

References

  1. Al Jawarneh, I.M., Bellavista, P., Corradi, A., Foschini, L., Montanari, R.: Big spatial data management for the Internet of Things: a survey. J. Netw. Syst. Manag. (2020). https://doi.org/10.1007/s10922-020-09549-6

    Article  Google Scholar 

  2. Hajeer, M., Dasgupta, D.: Handling Big Data using a data-aware HDFS and evolutionary clustering technique. IEEE Trans. Big Data 5(2), 134–147 (2019). https://doi.org/10.1109/TBDATA.2017.2782785

    Article  Google Scholar 

  3. Calcaterra, C., Carmenini, A., Marotta, A., Bucci, U., Cassioli, D.: MaxHadoop: An efficient scalable emulation tool to test SDN protocols in emulated Hadoop environments. J. Netw. Syst. Manag. (2020). https://doi.org/10.1007/s10922-020-09552-x

    Article  Google Scholar 

  4. Pretto, G.R., Dalmazo, B.L., Marques, J.A., Wu, Z., Wang, X., Korkhov, V., Navaux, P.O.A., Gaspary, L.P.: Boosting HPC applications in the cloud through JIT traffic-aware path provisioning. In: Computational Science and Its Applications—ICCSA 2019. Lecture Notes in Computer Science, pp. 702–716. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-24305-0_52

  5. Hashem, I.A.T., Yaqoob, I., Anuar, N.B., Mokhtar, S., Gani, A., Khan, S.U.: The rise of “big data” on cloud computing: review and open research issues. Inf. Syst. 47, 98–115 (2015). https://doi.org/10.1016/j.is.2014.07.006

    Article  Google Scholar 

  6. Clemm, A., Zhani, M.F., Boutaba, R.: Network management 2030: operations and control of network 2030 services. J. Netw. Syst. Manag. (2020). https://doi.org/10.1007/s10922-020-09517-0

    Article  Google Scholar 

  7. Li, P., Guo, S., Yu, S., Zhuang, W.: Cross-cloud MapReduce for big data. IEEE Trans. Cloud Comput. 8(2), 375–386 (2020). https://doi.org/10.1109/TCC.2015.2474385

    Article  Google Scholar 

  8. Mashayekhy, L., Nejad, M.M., Grosu, D.: A trust-aware mechanism for cloud federation formation. IEEE Trans. Cloud Comput. (2019). https://doi.org/10.1109/TCC.2019.2911831

    Article  Google Scholar 

  9. Ehsanfar, A., Grogan, P.T.: Auction-based algorithms for routing and task scheduling in federated networks. J. Netw. Syst. Manag. 28(2), 271–297 (2020). https://doi.org/10.1007/s10922-019-09506-y

    Article  Google Scholar 

  10. Lee, C.A., Bohn, R.B., Michel, M.: The NIST Cloud Federation Reference Architecture. Tech. Rep. NIST SP 500-332. National Institute of Standards and Technology, Gaithersburg (2020). https://doi.org/10.6028/NIST.SP.500-332

  11. Hadji, M., Zeghlache, D.: Mathematical programming approach for revenue maximization in cloud federations. IEEE Trans. Cloud Comput. 5(1), 99–111 (2017). https://doi.org/10.1109/TCC.2015.2402674

    Article  Google Scholar 

  12. Middya, A.I., Ray, B., Roy, S.: Auction based resource allocation mechanism in federated cloud environment: TARA. IEEE Trans. Serv. Comput. (2019). https://doi.org/10.1109/TSC.2019.2952772

    Article  Google Scholar 

  13. Deng, H., Huang, L., Xu, H., Liu, X., Wang, P., Fang, X.: Revenue maximization for dynamic expansion of geo-distributed cloud data centers. IEEE Trans. Cloud Comput. 8(3), 899–913 (2020). https://doi.org/10.1109/TCC.2018.2808351

    Article  Google Scholar 

  14. Ray, B.K., Saha, A., Khatua, S., Roy, S.: Toward maximization of profit and quality of cloud federation: solution to cloud federation formation problem. J. Supercomput. 75(2), 885–929 (2019). https://doi.org/10.1007/s11227-018-2620-2

    Article  Google Scholar 

  15. Darzanos, G., Koutsopoulos, I., Stamoulis, G.D.: Cloud federations: economics, games and benefits. IEEE/ACM Trans. Netw. 27(5), 2111–2124 (2019). https://doi.org/10.1109/TNET.2019.2943810

    Article  Google Scholar 

  16. Venkatesh, G., Arunesh, K.: Map reduce for big data processing based on traffic aware partition and aggregation. Clust. Comput. (2018). https://doi.org/10.1007/s10586-018-1799-6

    Article  Google Scholar 

  17. Eugster, P., Jayalath, C., Kogan, K., Stephen, J.: Big data analytics beyond the single datacenter. Computer 50(6), 60–68 (2017). https://doi.org/10.1109/MC.2017.163

    Article  Google Scholar 

  18. Cisco: Cisco Global Cloud Index, Forecast and Methodology, 2016–2021. White paper. Cisco System (2018). https://www.cisco.com/c/en/us/solutions/collateral/service-provider/global-cloud-index-gci/white-paper-c11-738085.html

  19. Andrae, A.: Total consumer power consumption forecast. Nordic Digital Business Summit 10 (2017)

  20. Forestiero, A., Mastroianni, C., Meo, M., Papuzzo, G., Sheikhalishahi, M.: Hierarchical approach for efficient workload management in geo-distributed data centers. IEEE Trans. Green Commun. Netw. 1(1), 97–111 (2017)

    Article  Google Scholar 

  21. Ehsanfar, A., Grogan, P.T.: Mechanism design for exchanging resources in federated networks. J. Netw. Syst. Manag. 28(1), 108–132 (2020). https://doi.org/10.1007/s10922-019-09498-9

    Article  Google Scholar 

  22. Najm, M., Tamarapalli, V.: Cost-efficient deployment of big data applications in federated cloud systems. In: 2019 11th International Conference on Communication Systems Networks (COMSNETS), pp 428–431 (2019). https://doi.org/10.1109/COMSNETS.2019.8711284

  23. Ahmad, B., Maroof, Z., McClean, S., Charles, D., Parr, G.: Economic impact of energy saving techniques in cloud server. Clust. Comput. 23(2), 611–621 (2020). https://doi.org/10.1007/s10586-019-02946-w

    Article  Google Scholar 

  24. Ismaeel, S., Karim, R., Miri, A.: Proactive dynamic virtual-machine consolidation for energy conservation in cloud data centres. J. Cloud Comput. 7(1), 10 (2018). https://doi.org/10.1186/s13677-018-0111-x

    Article  Google Scholar 

  25. Tripathi, A., Pathak, I., Vidyarthi, D.P.: Modified dragonfly algorithm for optimal virtual machine placement in cloud computing. J. Netw. Syst. Manag. (2020). https://doi.org/10.1007/s10922-020-09538-9

    Article  Google Scholar 

  26. Marcon, D.S., Neves, M.C., Oliveira, R.R., Bays, L.R., Boutaba, R., Gaspary, L.P., Barcellos, M.P.: IoNCloud: Exploring application affinity to improve utilization and predictability in datacenters. In: 2015 IEEE International Conference on Communications (ICC), pp. 5497–5503 (2015). https://doi.org/10.1109/ICC.2015.7249198

  27. Qie, X., Jin, S., Yue, W.: An energy-efficient strategy for virtual machine allocation over cloud data centers. J. Netw. Syst. Manag. 27(4), 860–882 (2019). https://doi.org/10.1007/s10922-019-09489-w

    Article  Google Scholar 

  28. Kumar, M., Sharma, S.C., Goel, S., Mishra, S.K., Husain, A.: Autonomic cloud resource provisioning and scheduling using meta-heuristic algorithm. Neural Comput. Appl. (2020). https://doi.org/10.1007/s00521-020-04955-y

    Article  Google Scholar 

  29. Khosravi, A., Andrew, L.L.H., Buyya, R.: Dynamic VM placement method for minimizing energy and carbon cost in geographically distributed cloud data centers. IEEE Trans. Sustain. Comput. 2(2), 183–196 (2017). https://doi.org/10.1109/TSUSC.2017.2709980

    Article  Google Scholar 

  30. Najm, M., Tamarapalli, V.: VM migration for profit maximization in federated cloud data centers. In: 2020 International Conference on COMmunication Systems NETworkS (COMSNETS), pp. 882–884 (2020). https://doi.org/10.1109/COMSNETS48256.2020.9027429

  31. Najm, M., Tamarapalli, V.: Inter-data center virtual machine migration in federated cloud. In: Proceedings of the 21st International Conference on Distributed Computing and Networking, p. 1 (2020)

  32. Maenhaut, P.J., Volckaert, B., Ongenae, V., De Turck, F.: Resource management in a containerized cloud: status and challenges. J. Netw. Syst. Manag. 28(2), 197–246 (2020). https://doi.org/10.1007/s10922-019-09504-0

    Article  Google Scholar 

  33. Kumar, M., Sharma, S.C., Goel, A., Singh, S.P.: A comprehensive survey for scheduling techniques in cloud computing. J. Netw. Comput. Appl. 143, 1–33 (2019). https://doi.org/10.1016/j.jnca.2019.06.006

    Article  Google Scholar 

  34. Zhang, X., Li, K., Zhang, Y.: Optimising data access latencies of virtual machine placement based on greedy algorithm in datacentre. Int. J. Comput. Sci. Eng. 18(2), 186–194 (2019). https://doi.org/10.1504/IJCSE.2019.097945

    Article  Google Scholar 

  35. Ferdaus, M.H., Murshed, M.M., Calheiros, R.N., Buyya, R.: An algorithm for network and data-aware placement of multi-tier applications in cloud data centers. CoRR (2017). abs/1706.06035

  36. Shabeera, T., Kumar, S.M., Salam, S.M., Krishnan, K.M.: Optimizing VM allocation and data placement for data-intensive applications in cloud using ACO metaheuristic algorithm. Int. J. Eng. Sci. Technol. 20(2), 616–628 (2017). https://doi.org/10.1016/j.jestch.2016.11.006

    Article  Google Scholar 

  37. Gu, L., Zeng, D., Li, P., Guo, S.: Cost minimization for big data processing in geo-distributed data centers. IEEE Trans. Emerg. Top. Comput. 2(3), 314–323 (2014)

    Article  Google Scholar 

  38. Xiao, W., Bao, W., Zhu, X., Liu, L.: Cost-aware big data processing across geo-distributed datacenters. IEEE Trans. Parallel Distrib. Syst. 28(11), 3114–3127 (2017)

    Article  Google Scholar 

  39. Deng, K., Ren, K., Zhu, M., Song, J.: A data and task co-scheduling algorithm for scientific cloud workflows. IEEE Trans. Cloud Comput. 8(2), 349–362 (2020). https://doi.org/10.1109/TCC.2015.2511745

    Article  Google Scholar 

  40. Latif, S., Gilani, S.M.M., Ali, L., Iqbal, S., Liaqat, M.: Characterizing the architectures and brokering protocols for enabling clouds interconnection. Concurr. Comput. Pract. Experience 32(21), e5676 (2020)

    Article  Google Scholar 

  41. Khorasani, N., Abrishami, S., Feizi, M., Esfahani, M.A., Ramezani, F.: Resource management in the federated cloud environment using Cournot and Bertrand competitions. Fut. Gener. Comput. Syst. 113, 391–406 (2020). https://doi.org/10.1016/j.future.2020.07.010

    Article  Google Scholar 

  42. (2020) Virtual network pricing. https://azure.microsoft.com/en-in/pricing/details/virtual-network/. Accessed 25 July 2020

  43. Dayarathna, M., Wen, Y., Fan, R.: Data center energy consumption modeling: a survey. IEEE Commun. Surv. Tutorials 18(1), 732–794 (2016). https://doi.org/10.1109/COMST.2015.2481183

    Article  Google Scholar 

  44. Singh, N., Rao, S.: Modeling and reducing power consumption in large it systems. In: Proceedings of IEEE International Systems Conference, pp. 178–183 (2010). https://doi.org/10.1109/SYSTEMS.2010.5482354

  45. Jennings, B., Stadler, R.: Resource management in clouds: survey and research challenges. J. Netw. Syst. Manag. 23(3), 567–619 (2015). https://doi.org/10.1007/s10922-014-9307-7

    Article  Google Scholar 

  46. Masdari, M., Salehi, F., Jalali, M., Bidaki, M.: A survey of PSO-based scheduling algorithms in cloud computing. J. Netw. Syst. Manag. 25(1), 122–158 (2017). https://doi.org/10.1007/s10922-016-9385-9

    Article  Google Scholar 

  47. Yassine, A., Shirehjini, A.A.N., Shirmohammadi, S.: Bandwidth on-demand for multimedia big data transfer across geo-distributed cloud data centers. IEEE Trans. Cloud Comput. 8(4), 1189–1198 (2020). https://doi.org/10.1109/TCC.2016.2617369

    Article  Google Scholar 

  48. (2020) Google cloud network pricing. https://cloud.google.com/compute/pricing#network. Accessed 25 July 2020

  49. Juve, G., Chervenak, A., Deelman, E., Bharathi, S., Mehta, G., Vahi, K.: Characterizing and profiling scientific workflows. Fut. Gener. Comput. Syst. 29(3), 682–692 (2013). https://doi.org/10.1016/j.future.2012.08.015

    Article  Google Scholar 

  50. Baker, T., Aldawsari, B., Asim, M., Tawfik, H., Maamar, Z., Buyya, R.: Cloud-SEnergy: a bin-packing based multi-cloud service broker for energy efficient composition and execution of data-intensive applications. Sustain. Comput. Inf. Syst. 19, 242–252 (2018). https://doi.org/10.1016/j.suscom.2018.05.011

    Article  Google Scholar 

  51. Wolke, A., Tsend-Ayush, B., Pfeiffer, C., Bichler, M.: More than bin packing: Dynamic resource allocation strategies in cloud data centers. Inf. Syst. 52, 83–95 (2015). https://doi.org/10.1016/j.is.2015.03.003

    Article  Google Scholar 

  52. Assis, M.R.M., Bittencourt, L.F.: MultiCloud tournament: a cloud federation approach to prevent free-riders by encouraging resource sharing. J. Netw. Comput. Appl. 166, 102694 (2020). https://doi.org/10.1016/j.jnca.2020.102694

    Article  Google Scholar 

Download references

Acknowledgements

The authors would like to thank Dr. Bala Prakasa Rao Killi for his help in the simulation part.

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, Indian Institute of Technology, Guwahati, Assam, 781039, India

    Moustafa Najm, Rakesh Tripathi & Venkatesh Tamarapalli

  2. Naif Arab University for Security Sciences, Riyadh, 11452, Kingdom of Saudi Arabia

    Mohammad Shadi Alhakeem

Authors
  1. Moustafa Najm
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Rakesh Tripathi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mohammad Shadi Alhakeem
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Venkatesh Tamarapalli
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Moustafa Najm.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Najm, M., Tripathi, R., Alhakeem, M.S. et al. A Cost-Aware Management Framework for Placement of Data-Intensive Applications on Federated Cloud. J Netw Syst Manage 29, 25 (2021). https://doi.org/10.1007/s10922-021-09594-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10922-021-09594-9

Keywords

Search

Navigation