Abstract
With the proliferation of cloud computing, cloud service providers offer users a variety of choices in terms of pricing and computing performance. A critical factor impacting computing performance is main memory, often evaluated using bandwidth and access latency metrics. For two evaluations with the same workload while under different system configurations, it is hard to determine which system delivers better memory performance for the particular workload if neither evaluation data achieves higher bandwidth and lower latency simultaneously. This dilemma is further exacerbated under different memory access patterns. We recognize that state-of-the-art memory performance metrics cannot well address the dilemma. To address this challenge, we define a holistic memory performance metric, named Hmem, which is calculated from a fusion of bandwidth and latency metrics across different access patterns. To reflect the overall performance of a given workload, we calculate the correlation between our proposed metric and the workload’s throughput. Experimental results show that Hmem exhibits an average improvement of 70% on correlation coefficients compared to state-of-the-art memory performance metrics. A large cloud service provider has adopted Hmem to improve the efficiency of their memory performance evaluation and cloud server selection.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
References
Garg, S.K., Versteeg, S., Buyya, R.: Smicloud: a framework for comparing and ranking cloud services. In: IEEE International Conference on Utility and Cloud Computing (UCC), pp. 210–218. IEEE (2011)
Gottscho, M., Govindan, S., Sharma, B., Shoaib, M., Gupta, P.: X-Mem: a cross-platform and extensible memory characterization tool for the cloud. In: IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), pp. 263–273. IEEE (2016)
Vish, V., Karthik, K., Thomas, W., Sri, S.: Intel Memory Latency Checker. https://www.intel.com/content/www/us/en/developer/articles/tool/intelr-memory-latency-checker.html
John, L.K.: More on finding a single number to indicate overall performance of a benchmark suite. ACM SIGARCH Comput. Arch. News (SIGARCH) 32(1), 3–8 (2004)
Iqbal, M.F., John, L.K.: Confusion by all means. In: Proceedings of the 6th International Workshop on Unique Chips and Systems (UCAS) (2010)
Sun, X.H., Wang, D.: APC: a performance metric of memory systems. ACM SIGMETRICS Perf. Eval. Rev. (SIGMETRICS) 40(2), 125–130 (2012)
Chou, Y., Fahs, B., Abraham, S.: Microarchitecture optimizations for exploiting memory-level parallelism. ACM SIGARCH Comput. Arch. News (SIGARCH) 32(2), 76–87 (2004)
Jain, R.: The Art of Computer Systems Performance Analysis: Techniques for Experimental Design, Measurement, Simulation, and Modeling. Wiley, New York (1991)
Liu, J., Jaiyen, B., Veras, R., Mutlu, O.: RAIDR: retention-aware intelligent DRAM refresh. ACM SIGARCH Comput. Arch. News (SIGARCH) 40, 1–12 (2012)
Hassan, H., Pekhimenko, G., Vijaykumar, N., Seshadri, V., Mutlu, O.: ChargeCache: reducing DRAM latency by exploiting row access locality. In: IEEE International Symposium on High Performance Computer Architecture (HPCA), pp. 581–593 (2016)
Hennessy, J.L., Patterson, D.A.: Computer Architecture: A Quantitative Approach. Elsevier, Amsterdam (2011)
Yi, L., Li, C., Guo, J.: CPI for runtime performance measurement: the good, the bad, and the ugly. In: IEEE International Symposium on Workload Characterization (IISWC), pp. 106–113 (2020)
Wang, L., Gao, W., Yang, K., Jiang, Z.: BOPS, a new computation-centric metric for datacenter computing. In: Benchmarking, Measuring, and Optimizing: Second BenchCouncil International Symposium (Bench), pp. 262–277 (2020)
Williams, S., Waterman, A., Patterson, D.: Roofline: an insightful visual performance model for multicore architectures. Commun. ACM 52(4), 65–76 (2009)
Lee, J., Kim, C., Lin, K., Cheng, L., Govindaraju, R., Kim, J.: WSMeter: a performance evaluation methodology for google’s production warehouse-scale computers. In: Proceedings of the Twenty-Third International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), pp. 549–563 (2018)
Sun, L., Dong, H., Hussain, F.K., Hussain, O.K., Chang, E.: Cloud service selection: state-of-the-art and future research directions. J. Netw. Comput. Appl. 45, 134–150 (2014)
Behzadian, M., Otaghsara, S.K., Yazdani, M., Ignatius, J.: A state-of-the-art survey of TOPSIS applications. Expert Syst. Appl. 39(17), 13051–13069 (2012)
Sotoudeh-Anvari, A., Sadjadi, S., Molana, S., Sadi-Nezhad, S.: A new MCDM-based approach using BWM and SAW for optimal search model. Decis. Sci. Lett. 7(4), 395–404 (2018)
de FSM Russo, R., Camanho, R.: Criteria in AHP: a systematic review of literature. Procedia Comput. Sci. 55, 1123–1132 (2015)
Mardani, A., Zavadskas, E.K., Govindan, K., Amat Senin, A., Jusoh, A.: VIKOR technique: a systematic review of the state of the art literature on methodologies and applications. Sustainability 8(1), 37–75 (2016)
McCalpin, J.D.: Stream benchmark (1995). https://www.cs.virginia.edu/stream/ref.html
McCalpin, J.D.: Stream2 benchmark (1999). https://cs.virginia.edu/stream/stream2
Staelin, C.: Lmbench: an extensible micro-benchmark suite. Softw. Pract. Exp. 35(11), 1079–1105 (2005)
Pase, D.: pChase benchmark (2013). https://github.com/maleadt/pChase
Radulovi, M.: Memory bandwidth and latency in HPC: system requirements and performance impact. Universitat Politècnica de Catalunya, pp. 14–20 (2019)
Barenblatt, G.I.: Dimensional Analysis. CRC Press, Boca Raton (1987)
Little, J.D., Graves, S.C.: Little’s law, Building intuition: insights from basic operations management models and principles, pp. 81–100 (2008)
Dh, B.: Little’s law and high performance computing, NAS applications and tools group, NASA Ames Research Center (1977). https://crd.lbl.gov/~dhbailey/dhbpapers/little.pdf
Mehta, S.: Performance analysis and optimization with little’s law. In: IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), pp. 12–23. IEEE (2022)
Islam, M.A., Arafath, M.Y., Hasan, M.J.: Design of DDR4 SDRAM controller. In: International Conference on Electrical and Computer Engineering (ICECE), pp. 148–151. IEEE (2014)
Citron, D., Hurani, A., Gnadrey, A.: The harmonic or geometric mean: does it really matter? ACM SIGARCH Comput. Arch. News (SIGARCH) 34(4), 18–25 (2006)
Mashey, J.R.: War of the benchmark means: time for a truce. ACM SIGARCH Comput. Arch. News (SIGARCH) 32(4), 1–14 (2004)
Marino, M.D.: Walter: wide I/O scaling of number of memory controllers versus frequency and voltage. IEEE Access 8, 193874–193889 (2020)
Standard Performance Evaluation Corporation: SPEC CPU 2017. https://www.spec.org/cpu2017/
Singh, S., Awasthi, M.: Memory centric characterization and analysis of SPEC CPU2017 suite. In: Proceedings of the 2019 ACM/SPEC International Conference on Performance Engineering (ICPE), pp. 285–292 (2019)
Acknowledgments
This work is supported by the National Natural Science Foundation of China (No. 62272167). We want to thank our lab colleagues and anonymous reviewers for their valuable comments and suggestions. We would also like to thank Chengdong Li from Code Title Poetry (Hangzhou) Technology for his valuable insights during the research process.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Li, Y. et al. (2024). Hmem: A Holistic Memory Performance Metric for Cloud Computing. In: Hunold, S., Xie, B., Shu, K. (eds) Benchmarking, Measuring, and Optimizing. Bench 2023. Lecture Notes in Computer Science, vol 14521. Springer, Singapore. https://doi.org/10.1007/978-981-97-0316-6_11
Download citation
DOI: https://doi.org/10.1007/978-981-97-0316-6_11
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-97-0315-9
Online ISBN: 978-981-97-0316-6
eBook Packages: Computer ScienceComputer Science (R0)