Advertisement

Simulation

A Methodology to Evaluate Recommendation Systems in Software Engineering
  • Robert J. Walker
  • Reid Holmes
Chapter

Abstract

Scientists and engineers have long used simulation as a technique for exploring and evaluating complex systems. Direct interaction with a real, complex system requires that the system be already constructed and operational, that people be trained in its use, and that its dangers already be known and mitigated. Simulation can avoid these issues, reducing costs, reducing risks, and allowing an imagined system to be studied before it is created. The explorations supported by simulation serve two purposes in the realm of evaluation: to determine whether and where undesired behavior will arise and to predict the outcomes of interactions with the real system. This chapter examines the use of simulation to evaluate recommendation systems in software engineering (RSSEs). We provide a general model of simulation for evaluation and review a small set of examples to examine how the model has been applied in practice. From these examples, we extract some general strengths and weaknesses of the use of simulation to evaluate RSSEs. We also explore prospects for making more extensive use of simulation in the future.

Keywords

Recommendation System Application Programming Interface Confusion Matrix Evaluation Evaluation Usage Scenario 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

References

  1. 1.
    Avazpour, I., Pitakrat, T., Grunske, L., Grundy, J.: Dimensions and metrics for evaluating recommendation systems. In: Robillard, M., Maalej, W., Walker, R.J., Zimmermann, T. (eds.) Recommendation Systems in Software Engineering, Chap. 10. Springer, New York (2014)Google Scholar
  2. 2.
    Babbage, C.: Passages from the Life of a Philosopher. Longman, Green, Longman, Roberts, & Green, London (1864)Google Scholar
  3. 3.
    Bachmann, A., Bird, C., Rahman, F., Devanbu, P., Bernstein, A.: The missing links: bugs and bug-fix commits. In: Proceedings of the ACM SIGSOFT International Symposium on Foundations of Software Engineering, pp. 97–106 (2010). doi:10.1145/1882291.1882308Google Scholar
  4. 4.
    Bavota, G., De Lucia, A., Marcus, A., Oliveto, R.: Automating Extract Class refactoring: an improved method and its evaluation. Empir. Software Eng. (2013). doi:10.1007/s10664-013-9256-x (in press)Google Scholar
  5. 5.
    Bird, C., Zimmermann, T.: Assessing the value of branches with what-if analysis. In: Proceedings of the ACM SIGSOFT International Symposium on Foundations of Software Engineering, pp. 45:1–45:11 (2012). doi:10.1145/2393596.2393648Google Scholar
  6. 6.
    Collinson, M., Monahan, B., Pym, D.: Semantics for structured systems modelling and simulation. In: Proceedings of the ICST International Conference on Simulation Tools and Techniques, pp. 34:1–34:8 (2010). doi:10.4108/ICST.SIMUTOOLS2010.8631Google Scholar
  7. 7.
    Cossette, B.E., Walker, R.J.: Seeking the ground truth: a retroactive study on the evolution and migration of software libraries. In: Proceedings of the ACM SIGSOFT International Symposium on Foundations of Software Engineering, pp. 55:1–55:11 (2012). doi:10.1145/2393596.2393661Google Scholar
  8. 8.
    Erzberger, C., Prein, G.: Triangulation: validity and empirically-based hypothesis construction. Qual. Quan. 31(2), 141–154 (1997). doi:10.1023/A:1004249313062CrossRefGoogle Scholar
  9. 9.
    Feinstein, A.H., Cannon, H.M.: Constructs of simulation evaluation. Simulat. Gaming 33(4), 425–440 (2002). doi:10.1177/1046878102238606CrossRefGoogle Scholar
  10. 10.
    Foss, T., Stensrud, E., Kitchenham, B., Myrtveit, I.: A simulation study of the model evaluation criterion MMRE. IEEE Trans. Software Eng. 29(11), 985–995 (2003). doi:10.1109/TSE.2003.1245300CrossRefGoogle Scholar
  11. 11.
    Hlupic, V., Irani, Z., Paul, R.J.: Evaluation framework for simulation software. Int. J. Adv. Manuf. Technol. 15(5), 366–382 (1999). doi:10.1007/s001700050079CrossRefGoogle Scholar
  12. 12.
    Holmes, R., Ratchford, T., Robillard, M., Walker, R.J.: Automatically recommending triage decisions for pragmatic reuse tasks. In: Proceedings of the IEEE/ACM International Conference on Automated Software Engineering, pp. 397–408 (2009). doi:10.1109/ASE.2009.65Google Scholar
  13. 13.
    Holmes, R., Walker, R.J.: Customized awareness: recommending relevant external change events. In: Proceedings of the ACM/IEEE International Conference on Software Engineering, pp. 465–474 (2010). doi:10.1145/1806799.1806867Google Scholar
  14. 14.
    Holmes, R., Walker, R.J.: Systematizing pragmatic software reuse. ACM Trans. Software Eng. Methodol. 21(4), 20:1–20:44 (2012). doi:10.1145/2377656.2377657Google Scholar
  15. 15.
    Holmes, R., Walker, R.J., Murphy, G.C.: Approximate structural context matching: an approach to recommend relevant examples. IEEE Trans. Software Eng. 32(12), 952–970 (2006). doi:10.1109/TSE.2006.117CrossRefGoogle Scholar
  16. 16.
    Hughes, R.I.G.: The Ising model, computer simulation, and universal physics. In: Morgan, M.S., Morrison, M. (eds.) Models as Mediators: Perspectives on Natural and Social Science, No. 52 in Ideas in Context, Chap. 5. Cambridge University Press, Cambridge (1999). doi:10.1017/CBO9780511660108.006Google Scholar
  17. 17.
    Kelley, J.F.: An iterative design methodology for user-friendly natural language office information applications. ACM Trans. Inform. Syst. 2(1), 26–41 (1984). doi:10.1145/357417.357420CrossRefMathSciNetGoogle Scholar
  18. 18.
    Kim, S., Zimmermann, T., Whitehead Jr., E.J., Zeller, A.: Predicting faults from cached history. In: Proceedings of the ACM/IEEE International Conference on Software Engineering, pp. 489–498 (2007). doi:10.1109/ICSE.2007.66Google Scholar
  19. 19.
    Kononenko, O., Dietrich, D., Sharma, R., Holmes, R.: Automatically locating relevant programming help online. In: Proceedings of the IEEE Symposium on Visual Languages and Human-Centric Computing, pp. 127–134 (2012). doi:10.1109/VLHCC.2012.6344497Google Scholar
  20. 20.
    Marcus, A., Poshyvanyk, D., Ferenc, R.: Using the conceptual cohesion of classes for fault prediction in object-oriented systems. IEEE Trans. Software Eng. 34(2), 287–300 (2008). doi:10.1109/TSE.2007.70768CrossRefGoogle Scholar
  21. 21.
    Matejka, J., Li, W., Grossman, T., Fitzmaurice, G.: CommunityCommands: command recommendations for software applications. In: Proceedings of the ACM Symposium on User Interface Software and Technology, pp. 193–202 (2009). doi:10.1145/1622176.1622214Google Scholar
  22. 22.
    Mosteller, F.: A k-sample slippage test for an extreme population. Ann. Math. Stat. 19(1), 58–65 (1948). doi:10.1214/aoms/1177730290CrossRefMATHMathSciNetGoogle Scholar
  23. 23.
    Murphy-Hill, E., Jiresal, R., Murphy, G.C.: Improving software developers’ fluency by recommending development environment commands. In: Proceedings of the ACM SIGSOFT International Symposium on Foundations of Software Engineering, pp. 42:1–42:11 (2012). doi:10.1145/2393596.2393645Google Scholar
  24. 24.
    Murphy-Hill, E., Murphy, G.C.: Recommendation delivery: getting the user interface just right. In: Robillard, M., Maalej, W., Walker, R.J., Zimmermann, T. (eds.) Recommendation Systems in Software Engineering, Chap. 9. Springer, New York (2014)Google Scholar
  25. 25.
    Robillard, M.P.: Topology analysis of software dependencies. ACM Trans. Software Eng. Methodol. 17(4), 18:1–18:36 (2008). doi:10.1145/13487689.13487691Google Scholar
  26. 26.
    Robillard, M., Walker, R.J.: An introduction to recommendation systems in software engineering. In: Robillard, M., Maalej, W., Walker, R.J., Zimmermann, T. (eds.) Recommendation Systems in Software Engineering, Chap. 1. Springer, New York (2014)Google Scholar
  27. 27.
    Robillard, M.P., Walker, R.J., Zimmermann, T.: Recommendation systems for software engineering. IEEE Software 27(4), 80–86 (2010). doi:10.1109/MS.2009.161CrossRefGoogle Scholar
  28. 28.
    Said, A., Tikk, D., Cremonesi, P.: Benchmarking: a methodology for ensuring the relative quality of recommendation systems in software engineering. In: Robillard, M., Maalej, W., Walker, R.J., Zimmermann, T. (eds.) Recommendation Systems in Software Engineering, Chap. 11. Springer, New York (2014)Google Scholar
  29. 29.
    Sánchez, P.J.: Fundamentals of simulation modeling. In: Proceedings of the Winter Simulation Conference, pp. 54–62 (2007). doi:10.1109/WSC.2007.4419588Google Scholar
  30. 30.
    Sebastiani, F.: Machine learning in automated text categorization. ACM Comput. Surv. 34(1), 1–47 (2002). doi:10.1145/505282.505283CrossRefGoogle Scholar
  31. 31.
    Shepperd, M., Kadoda, G.: Using simulation to evaluate prediction techniques. In: Proceedings of the IEEE International Symposium on Software Metrics, pp. 349–359 (2001). doi:10.1109/METRIC.2001.915542Google Scholar
  32. 32.
    Teorey, T.J., Merten, A.G.: Considerations on the level of detail in simulation. In: Proceedings of the Symposium on Simulation of Computer Systems, pp. 137–143 (1973)Google Scholar
  33. 33.
    Tosun Mısırlı, A., Bener, A., Çağlayan, B., Çalıklı, G., Turhan, B.: Field studies: a methodology for construction and evaluation of recommendation systems in software engineering. In: Robillard, M., Maalej, W., Walker, R.J., Zimmermann, T. (eds.) Recommendation Systems in Software Engineering, Chap. 13. Springer, New York (2014)Google Scholar
  34. 34.
    van Rijsbergen, C.J.: Information Retrieval. 2nd edn. Butterworth–Heinemann, London (1979)Google Scholar
  35. 35.
    Winsberg, E.: Simulated experiments: methodology for a virtual world. Philos. Sci. 70(1), 105–125 (2003). doi:10.1086/367872CrossRefGoogle Scholar
  36. 36.
    Ye, Y., Yamamoto, Y., Nakakoji, K., Nishinaka, Y., Asada, M.: Searching the library and asking the peers: learning to use Java APIs on demand. In: Proceedings of the International Symposium on Principles and Practice of Programming in Java, pp. 41–50 (2007). doi:10.1145/1294325.1294332Google Scholar
  37. 37.
    Zimmermann, T., Weißgerber, P., Diehl, S., Zeller, A.: Mining version histories to guide software changes. IEEE Trans. Software Eng. 31(6), 429–445 (2005). doi:10.1109/TSE.2005.72CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2014

Authors and Affiliations

  1. 1.Department of Computer ScienceUniversity of CalgaryCalgaryCanada
  2. 2.David R. Cheriton School of Computer ScienceUniversity of WaterlooWaterlooCanada

Personalised recommendations