Abstract
Due to unreasonable virtual machine (VM) resource planning and complex load variation, the waste of VM resource has become a significant issue for many enterprises. Although existing technical solutions have proven to have certain ability to identify idle VMs, most of them are researched in private cloud or public cloud scenarios. And it lacks an effective method customized for managed clouds, where the previous work still suffers from the challenges of fewer labels, poor data quality and large scale of VMs in production environments. For this reason, we first investigate the resource usage data of thousands of VMs from a real managed cloud. Based on the analysis results, we propose an innovative and practical method to identify idle VMs. Through elaborate data processing, feature engineering, and model training, the proposed method enables to achieve excellent performance. Sufficient experiments based on real data from the managed cloud of Sangfor company also prove its practicality and effectiveness in the production environment. Up to now, this service has been deployed in Sangfor cloud for more than 5 months, continuously detecting over 10K VMs, and helping to save at least 1K vCPU cores, 2.5 TB memory and 100 TB disk space.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Insight, Managed cloud services. https://www.insight.com/en_US/glossary/m/managed-cloud-services.html (2022)
Wood, T., Shenoy, P., Venkataramani, A., Yousif, M.: Black-box and gray-box strategies for virtual machine migration. In: NSDI, vol. 7, pp. 17–17 (2007)
Calheiros, R.N., Ranjan, R., Buyya, R.: Virtual machine provisioning based on analytical performance and QOS in cloud computing environments. In: 2011 International Conference on Parallel Processing. IEEE, 2011, pp. 295–304 (2011)
Breitgand, D., et al.: An adaptive utilization accelerator for virtualized environments. In: 2014 IEEE International Conference on Cloud Engineering. IEEE, 2014, pp. 165–174 (2014)
Chunlin, L., Hammad-Ur-Rehman, Q.: Adaptive threshold detection based on current demand for efficient utilization of cloud resources. In: 2019 IEEE 4th International Conference on Computer and Communication Systems (ICCCS). IEEE, 2019, pp. 341–346 (2019)
IKim, I.K., Zeng, S., Young, C., Hwang, J., Humphrey, M.: iCSI: A cloud garbage VM collector for addressing inactive VMs with machine learning. In: 2017 IEEE International Conference on Cloud Engineering (IC2E) (2017)
Mazidi, A., Mahdavi, M., Roshanfar, F.: An autonomic decision tree-based and deadline-constraint resource provisioning in cloud applications. Concurr. Comput.: Pract. Exp. 33(10), e6196 (2021)
Khandros, M., et al.: Machine learning computing model for virtual machine underutilization detection. Jul. 6 2021, uS Patent 11,055,126 (2021)
Gopisetti, A., Jha, C., George, J.R., Gaurav, K., Singh, J.: Usage pattern virtual machine idle detection. Feb. 10 2022, uS Patent App. 17/510,546
Breiman, L.: Random forests. Mach. Learn. 45(1), 5–32 (2001)
Lindner, M., McDonald, F., McLarnon, B., Robinson, P.: Towards automated business-driven indication and mitigation of vm sprawl in cloud supply chains. In: 12th IFIP/IEEE International Symposium on Integrated Network Management (IM: and Workshops. IEEE 2011, pp. 1062–1065 (2011)
Microsoft Azure, What is virtual desktop infrastructure (vdi)? https://azure.microsoft.com/en-us/resources/cloud-computing-dictionary/what-is-virtual-desktop-infrastructure-vdi/ (2022)
Fesl, J., Gokhale, V., Feslová, M.: Efficient virtual machine consolidation approach based on user inactivity detection. Cloud Comput. 2019, 115 (2019)
Kim, I.K., Zeng, S., Young, C., Hwang, J., Humphrey, M.: A supervised learning model for identifying inactive vms in private cloud data centers. In: Proceedings of the Industrial Track of the 17th International Middleware Conference, 2016, pp. 1–7 (2016)
Zhang, B., Al Dhuraibi, Y., Rouvoy, R., Paraiso, F., Seinturier, L.: Cloudgc: Recycling idle virtual machines in the cloud. In: 2017 IEEE International Conference on Cloud Engineering (IC2E). IEEE, 2017, pp. 105–115 (2017)
Christ, M., Braun, N., Neuffer, J., Kempa-Liehr, A.W., et al.: Tsfresh. https://tsfresh.readthedocs.io/en/latest/text/introduction.html (2022)
Li, Y.: Research and application of deep learning in image recognition. In: 2022 IEEE 2nd International Conference on Power, Electronics and Computer Applications (ICPECA). IEEE, 2022, pp. 994–999 (2022)
Li, Q., et al.: A survey on text classification: from traditional to deep learning. ACM Trans. Intell. Syst. Technol. (TIST) 13(2), 1–41 (2022)
Scikit-learn.org: Scikit-learn. https://scikit-learn.org/stable/ (2022)
Kubernetes Enterprise: Kubernetes (k8s). https://kubernetes.io/ (2022)
Redis Enterprise: Reids. https://redis.io/ (2022)
Shen, Z., Young, C.C., Zeng, S., Murthy, K., Bai, K.: Identifying resources for cloud garbage collection. In: 2016 12th International Conference on Network and Service Management (CNSM) (2016)
Acknowledgment
This work was supported by Cloud Security Key Technology Research Key Laboratory of Shenzhen under grant No.ZDSY20200811143600002, the National Key R&D Program of China (No. 2021YFB3300200), National Natural Science Foundation of China (No. 62072451, 92267105), Guangdong Special Support Plan (No. 2021TQ06X990), and Shenzhen Basic Research Program (No. JCYJ20200109115418592, JCYJ20220818101610023). We would like to express thanks to the reviewers and editors for their constructive comments and suggestions. We also thank anyone who helped us improve this work.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Yu, X. et al. (2023). A Semi-supervised Learning Based Method for Identifying Idle Virtual Machines in Managed Cloud: Application and Practice. In: Zhang, Y., Zhang, LJ. (eds) Web Services – ICWS 2023. ICWS 2023. Lecture Notes in Computer Science, vol 14209. Springer, Cham. https://doi.org/10.1007/978-3-031-44836-2_5
Download citation
DOI: https://doi.org/10.1007/978-3-031-44836-2_5
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-44835-5
Online ISBN: 978-3-031-44836-2
eBook Packages: Computer ScienceComputer Science (R0)