Advertisement

Log4Perf: suggesting and updating logging locations for web-based systems’ performance monitoring

  • Kundi YaoEmail author
  • Guilherme B. de Pádua
  • Weiyi Shang
  • Catalin Sporea
  • Andrei Toma
  • Sarah Sajedi
Article
  • 69 Downloads

Abstract

Performance assurance activities are an essential step in the release cycle of software systems. Logs have become one of the most important sources of information that is used to monitor, understand and improve software performance. However, developers often face the challenge of making logging decisions, i.e., neither logging too little and logging too much is desirable. Although prior research has proposed techniques to assist in logging decisions, those automated logging guidance techniques are rather general, without considering a particular goal, such as monitoring software performance. In this paper, we present Log4Perf, an automated approach that provides suggestions of where to insert logging statement with the goal of monitoring web-based systems’ CPU usage. In the first part of our approach, we leverage the performance model’s prediction errors to suggest the need for updating logging locations when software evolves. In the second part of our approach, we build and manipulate a statistical performance model to identify the locations in the source code that statistically significantly influence CPU usage. To evaluate Log4Perf, we conduct case studies on two open source systems, i.e., CloudStore and OpenMRS, and one large-scale commercial system. Our evaluation results show that our approach can suggest the need for updating logging locations and identify the logging locations that can be kept unchanged. We manually examine the logging locations that are newly suggested or deprecated. We find that our approach can build well-fit statistical performance models, indicating that such models can be leveraged to investigate the influence of locations in the source code on performance. The suggested logging locations are often in small and simple methods that do not have logging statements, and are not performance hotspots. Our approach can be used to complement traditional approaches that are based on software metrics or performance hotspots. In addition, we identify seven root-causes of these suggested or deprecated logging locations. Log4Perf is integrated into the release engineering process of the commercial software to provide logging suggestions on a regular basis.

Keywords

Software logs Logging suggestion Performance monitoring Performance modeling Performance engineering 

Notes

Acknowledgments

We would like to thank ERA Environmental Management Solutions for providing access to the enterprise system used in our case study. The findings and opinions expressed in this paper are those of the authors and do not necessarily represent or reflect those of ERA Environmental Management Solutions and/or its subsidiaries and affiliates. Moreover, our results do not reflect the quality of ERA Environmental Management Solutions’ products.

References

  1. The Java HotSpot Performance Engine Architecture (2010) Slowly Changing Dimensions. https://www.oracle.com/technetwork/java/whitepaper-135217.html. Oracle
  2. .NET Compiler Platform (”Roslyn”) (2017) .NET Compiler Platform (”Roslyn”). https://github.com/dotnet/roslyn
  3. Ahmed TM, Bezemer C, Chen TH, Hassan AE, Shang W (2016) Studying the effectiveness of application performance management (apm) tools for detecting performance regressions for web applications: an experience report. In: Proceedings of the 13th International Conference on Mining Software Repositories, MSR ’16. ACM, New York, pp 1–12Google Scholar
  4. Alghmadi HM, Syer MD, Shang W, Hassan AE (2016) An automated approach for recommending when to stop performance tests. In: 2016 IEEE International conference on software maintenance and evolution (ICSME), pp 279–289Google Scholar
  5. Apache JMeter (2015) Apache: Jmeter. http://jmeter.apache.org/. Accessed: 2015-06-01
  6. Bezemer C, Milon E, Zaidman A, Pouwelse J (2014) Detecting and analyzing i/o performance regressions. Journal of Software: Evolution and Process 26 (12):1193–1212.  https://doi.org/10.1002/smr.1657. https://onlinelibrary.wiley.com/doi/abs/10.1002/smr.1657 CrossRefGoogle Scholar
  7. Bezemer CP, Zaidman A (2014) Performance optimization of deployed software-as-a-service applications. J Syst Softw 87:87–103.  https://doi.org/10.1016/j.jss.2013.09.013 CrossRefGoogle Scholar
  8. Bootstrap (2017) Bootstrap. https://cran.r-project.org/web/packages/bootstrap/bootstrap.pdf. Accessed: 2017-02-27
  9. Brebner PC (2016) Automatic performance modelling from application performance management (apm) data: an experience report. In: Proceedings of the 7th ACM/SPEC on International Conference on Performance Engineering, ICPE ’16. ACM, New York, pp 55–61Google Scholar
  10. Chen J, Shang W (2017) An exploratory study of performance regression introducing code changes. In: 2017 IEEE international conference on software maintenance and evolution (ICSME). IEEE, pp 341–352Google Scholar
  11. Chen TH, Shang W, Hassan AE, Nasser M, Flora P (2016) Cacheoptimizer: Helping developers configure caching frameworks for hibernate-based database-centric web applications. In: Proceedings of the 24th ACM SIGSOFT international symposium on the foundations of software engineering, FSE ’16Google Scholar
  12. Chen TH, Shang W, Hassan AE, Nasser M, Flora P (2016) Cacheoptimizer: Helping developers configure caching frameworks for hibernate-based database-centric web applications. In: Proceedings of the 2016 24th ACM SIGSOFT international symposium on foundations of software engineering, FSE 2016. ACM, New York, pp 666–677Google Scholar
  13. Chen TH, Shang W, Jiang ZM, Hassan AE, Nasser M, Flora P (2014) Detecting performance anti-patterns for applications developed using object-relational mapping. In: Proceedings of the 36th international conference on software engineering, ICSE 2014. ACM, New York, pp 1001–1012Google Scholar
  14. Chen TH, Shang W, Jiang ZM, Hassan AE, Nasser M, Flora P (2016) Finding and evaluating the performance impact of redundant data access for applications that are developed using object-relational mapping frameworks. IEEE Trans Softw Eng PP(99):1–1CrossRefGoogle Scholar
  15. Cliff N (1993) Dominance statistics: ordinal analyses to answer ordinal questions. Psychol Bull 114(3):494CrossRefGoogle Scholar
  16. CloudScale (2017) CloudScale Project. http://www.cloudscale-project.eu/
  17. Cohen I, Chase JS, Goldszmidt M, Kelly T, Symons J (2004) Correlating instrumentation data to system states: a building block for automated diagnosis and control. In: OSDI, vol 4, pp 16–16Google Scholar
  18. Cohen I, Zhang S, Goldszmidt M, Symons J, Kelly T, Fox A (2005) Capturing, indexing, clustering, and retrieving system history. In: Proceedings of the twentieth ACM symposium on operating systems principles, SOSP ’05. ACM, New York, pp 105–118Google Scholar
  19. Eclipse JDT (2017) Eclipse Java development tools (JDT). http://www.eclipse.org/jdt/
  20. Farshchi M, Schneider JG, Weber I, Grundy J (2015) Experience report: Anomaly detection of cloud application operations using log and cloud metric correlation analysis. In: 2015 IEEE 26th international symposium on software reliability engineering (ISSRE), pp 24–34Google Scholar
  21. Foo KC, Jiang ZM, Adams B, Hassan AE, Zou Y, Flora P (2010) Mining performance regression testing repositories for automated performance analysis. In: 2010 10th international conference on quality software (QSIC). IEEE, pp 32–41Google Scholar
  22. Freedman DA (2009) Statistical models: theory and practice. Cambridge University Press, CambridgeCrossRefGoogle Scholar
  23. Fu Q, Zhu J, Hu W, Lou JG, Ding R, Lin Q, Zhang D, Xie T (2014) Where do developers log? an empirical study on logging practices in industry. In: Companion proceedings of the 36th international conference on software engineering, ICSE companion 2014. ACM, New York, pp 24–33Google Scholar
  24. Gao R, Jiang ZM, Barna C, Litoiu M (2016) A framework to evaluate the effectiveness of different load testing analysis techniques. In: 2016 IEEE International conference on software testing, verification and validation (ICST), pp 22–32Google Scholar
  25. Harrell F (2001) Regression modeling strategies 2001. Springer CrossRef Google Scholar, NashvilleCrossRefGoogle Scholar
  26. Van Hoorn A, Rohr M, Hasselbring W, Waller J, Ehlers J, Frey S, Kieselhorst D. (2009) Continuous monitoring of software services: Design and application of the Kieker frameworkGoogle Scholar
  27. Van Hoorn A, Waller J, Hasselbring W (2012) Kieker: a framework for application performance monitoring and dynamic software analysis. In: Proceedings of the 3rd ACM/SPEC international conference on performance engineering, ICPE ’12. ACM, New York, pp 247–248Google Scholar
  28. Jiang ZM, Hassan AE, Hamann G, Flora P (2009) Automated performance analysis of load tests. In: ICSM ’09: 25th IEEE international conference on software maintenanceGoogle Scholar
  29. JIT (2018) Tracing just-in-time compilation. https://en.wikipedia.org/wiki/Tracing_just-in-time_compilation
  30. JVM (2018) Jvm jit compilation as a way of performance optimisation. https://jakubstransky.com/2018/01/15/java-jvm-jit-compilation-performance-optimisation/
  31. Kabinna S, Bezemer C, Shang W, Hassan AE (2016) Logging library migrations: a case study for the apache software foundation projects. In: Proceedings of the 13th international conference on mining software repositories. ACM, pp 154–164Google Scholar
  32. Kabinna S, Bezemer C, Shang W, Syer MD, Hassan AE (2018) Examining the stability of logging statements. Empir Softw Eng 23(1):290–333CrossRefGoogle Scholar
  33. Kabinna S, Shang W, Bezemer C, Hassan AE (2016) Examining the stability of logging statements. In: 2016 IEEE 23rd international conference on software analysis, evolution, and reengineering (SANER), vol 1, pp 326–337, DOI  https://doi.org/10.1109/SANER.2016.29
  34. Kernighan BW, Pike R (1999) The practice of programming. Addison-Wesley Longman Publishing Co., Inc., BostonGoogle Scholar
  35. Kuhn M (2008) Building predictive models in r using the caret package. Journal of Statistical Software Articles 28(5):1–26Google Scholar
  36. Li H, Chen THP, Hassan AE, Nasser M, Flora P (2018) Adopting autonomic computing capabilities in existing large-scale systems: an industrial experience report. In: Proceedings of the 40th international conference on software engineering: software engineering in practice. ACM, pp 1–10Google Scholar
  37. Li H, Shang W, Hassan AE (2017) Which log level should developers choose for a new logging statement? Empir Softw Eng 22(4):1684–1716CrossRefGoogle Scholar
  38. Li H, Shang W, Zou Y, Hassan AE (2017) Towards just-in-time suggestions for log changes. Empir Softw Eng 22(4):1831–1865CrossRefGoogle Scholar
  39. Maplesden D, von Randow K, Tempero E, Hosking J, Grundy J (2015) Performance analysis using subsuming methods: an industrial case study. In: Proceedings of the 37th international conference on software engineering - Volume 2, ICSE ’15. IEEE Press, Piscataway, pp 149–158Google Scholar
  40. Maplesden D, Tempero E, Hosking J, Grundy JC (2015) Subsuming methods: Finding new optimisation opportunities in object-oriented software. In: Proceedings of the 6th ACM/SPEC international conference on performance engineering, ICPE ’15. ACM, New York, pp 175–186Google Scholar
  41. Mockus A (2010) Organizational volatility and its effects on software defects. In: Proceedings of the Eighteenth ACM SIGSOFT international symposium on foundations of software engineering, FSE ’10. ACM, New York, pp 117–126Google Scholar
  42. Moore DS, Craig BA, McCabe GP (2012) Introduction to the practice of statistics. WH Freeman, New YorkGoogle Scholar
  43. Nguyen TH, Adams B, Jiang ZM, Hassan AE, Nasser M, Flora P (2012) Automated detection of performance regressions using statistical process control techniques. In: Proceedings of the 3rd ACM/SPEC international conference on performance engineering, ICPE ’12. ACM, New York, pp 299–310Google Scholar
  44. Sandoval Alcocer JP, Bergel A, Valente MT (2016) Learning from source code history to identify performance failures. In: Proceedings of the 7th ACM/SPEC on international conference on performance engineering, ICPE ’16. ACM, New York, pp 37–48, DOI  https://doi.org/10.1145/2851553.2851571
  45. Shang W, Hassan AE, Nasser M, Flora P (2015) Automated detection of performance regressions using regression models on clustered performance counters. In: Proceedings of the 6th ACM/SPEC international conference on performance engineering, ICPE ’15. ACM, New York, pp 15–26Google Scholar
  46. Shang W, Jiang ZM, Adams B, Hassan AE, Godfrey MW, Nasser M, Flora P (2011) An exploratory study of the evolution of communicated information about the execution of large software systems. In: 2011 18Th working conference on reverse engineering, pp 335–344.  https://doi.org/10.1109/WCRE.2011.48
  47. Shang W, Jiang ZM, Adams B, Hassan AE, Godfrey MW, Nasser M, Flora P (2014) An exploratory study of the evolution of communicated information about the execution of large software systems. Journal of Software: Evolution and Process 26 (1):3–26Google Scholar
  48. Shihab E, Jiang ZM, Ibrahim WM, Adams B, Hassan AE (2010) Understanding the impact of code and process metrics on post-release defects: a case study on the eclipse project. In: Proceedings of the 2010 ACM-IEEE international symposium on empirical software engineering and measurement, ESEM ’10. ACM, New York, pp 4:1–4:10Google Scholar
  49. Shihab E, Mockus A, Kamei Y, Adams B, Hassan AE (2011) High-impact defects: a study of breakage and surprise defects. In: Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on foundations of software engineering, ESEC/FSE ’11. ACM, New York, pp 300–310Google Scholar
  50. Sim SE, Easterbrook S, Holt RC (2003) Using benchmarking to advance research: a challenge to software engineering. In: proceedings of the 25th international conference on software engineering. IEEE Computer Society, pp 74–83Google Scholar
  51. Stewart C, Kelly T, Zhang A (2007) Exploiting nonstationarity for performance prediction. In: ACM SIGOPS operating systems review, vol 41. ACM, pp 31–44Google Scholar
  52. Syer MD, Shang W, Jiang ZM, Hassan AE (2017) Continuous validation of performance test workloads. Autom Softw Eng 24(1):189–231.  https://doi.org/10.1007/s10515-016-0196-8 CrossRefGoogle Scholar
  53. TPC-W (2017) TPC Benchmark W (TPC-W). http://www.tpc.org/tpcw/
  54. Visual Studio Profiling (2017) Visual Studio Profiling. https://docs.microsoft.com/en-us/visualstudio/profiling
  55. Waller J, Ehmke NC, Hasselbring W (2015) Including performance benchmarks into continuous integration to enable devops. ACM SIGSOFT Software Engineering Notes 40(2):1–4CrossRefGoogle Scholar
  56. Weyuker E, Vokolos F (2000) Experience with performance testing of software systems: issues, an approach, and case study. Transactions on Software Engineering 26 (12):1147–1156CrossRefGoogle Scholar
  57. Xiong P, Pu C, Zhu X, Griffith R (2013) vperfguard: an automated model-driven framework for application performance diagnosis in consolidated cloud environments. In: Proceedings of the 4th ACM/SPEC international conference on performance engineering, ICPE ’13. ACM, New York, pp 271–282Google Scholar
  58. Yao K, B de Pádua G, Shang W, Sporea S, Toma A, Sajedi S (2018) Log4perf: Suggesting logging locations for web-based systems’ performance monitoring. In: Proceedings of the 2018 ACM/SPEC international conference on performance engineering. ACM, pp 127–138Google Scholar
  59. Yin F, Dong D, Li S, Guo J, Chow K (2018) Java performance troubleshooting and optimization at Alibaba. In: Proceedings of the 40th international conference on software engineering: software engineering in practice, ICSE-SEIP ’18. ACM, New York, pp 11–12, DOI  https://doi.org/10.1145/3183519.3183536
  60. Yuan D, Luo Y, Zhuang X, Rodrigues GR, Zhao X, Zhang Y, Jain PU, Stumm M (2014) Simple testing can prevent most critical failures: an analysis of production failures in distributed data-intensive systems. In: Proceedings of the 11th USENIX conference on operating systems design and implementation, OSDI’14. USENIX Association, Berkeley, pp 249–265Google Scholar
  61. Yuan D, Park S, Huang P, Liu Y, Lee MM, Tang X, Zhou Y, Savage S (2012) Be conservative: Enhancing failure diagnosis with proactive logging. In: OSDI ’12: Proceedings of the 10th USENIX conference on operating systems design and implementation, vol 12, pp 293–306Google Scholar
  62. Yuan D, Zheng J, Park S, Zhou Y, Savage S (2011) Improving software diagnosability via log enhancement. In: ASPLOS ’11: Proceedings of the 16th international conference on architectural support for programming languages and operating systemsGoogle Scholar
  63. Zhang S, Ernst MD (2014) Which configuration option should i change?. In: Proceedings of the 36th international conference on software engineering, ICSE 2014. ACM, New York, pp 152–163Google Scholar
  64. Zhang S, Ernst MD (2015) Proactive detection of inadequate diagnostic messages for software configuration errors. In: Proceedings of the 2015 international symposium on software testing and analysis, ISSTA 2015. ACM, New York, pp 12–23Google Scholar
  65. Zhao X, Rodrigues K, Luo Y, Stumm M, Yuan D, Zhou Y (2017) The game of twenty questions: Do you know where to log?. In: Proceedings of the 16th workshop on hot topics in operating systems, HotOS ’17. ACM, New York, pp 125–131Google Scholar
  66. Zhu J, He P, Fu Q, Zhang H, Lyu MR, Zhang D (2015) Learning to log: Helping developers make informed logging decisions. In: Proceedings of the 37th international conference on software engineering - volume 1, ICSE ’15. IEEE Press, Piscataway, pp 415–425Google Scholar
  67. Zhuang Z, Ramachandra H, Tran C, Subramaniam S, Botev C, Xiong C, Sridharan B (2015) Capacity planning and headroom analysis for taming database replication latency: Experiences with linkedin internet traffic. In: Proceedings of the 6th ACM/SPEC international conference on performance engineering, ICPE ’15. ACM, New York, pp 39–50Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Department of Computer Science and Software EngineeringConcordia UniversityMontrealCanada
  2. 2.ERA Environmental Management SolutionsMontrealCanada

Personalised recommendations