Skip to main content
SpringerLink
Log in

Simulation

A Methodology to Evaluate Recommendation Systems in Software Engineering

  • Chapter
  • First Online:
Recommendation Systems in Software Engineering

Abstract

Scientists and engineers have long used simulation as a technique for exploring and evaluating complex systems. Direct interaction with a real, complex system requires that the system be already constructed and operational, that people be trained in its use, and that its dangers already be known and mitigated. Simulation can avoid these issues, reducing costs, reducing risks, and allowing an imagined system to be studied before it is created. The explorations supported by simulation serve two purposes in the realm of evaluation: to determine whether and where undesired behavior will arise and to predict the outcomes of interactions with the real system. This chapter examines the use of simulation to evaluate recommendation systems in software engineering (RSSEs). We provide a general model of simulation for evaluation and review a small set of examples to examine how the model has been applied in practice. From these examples, we extract some general strengths and weaknesses of the use of simulation to evaluate RSSEs. We also explore prospects for making more extensive use of simulation in the future.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 169.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 169.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Avazpour, I., Pitakrat, T., Grunske, L., Grundy, J.: Dimensions and metrics for evaluating recommendation systems. In: Robillard, M., Maalej, W., Walker, R.J., Zimmermann, T. (eds.) Recommendation Systems in Software Engineering, Chap. 10. Springer, New York (2014)

    Google Scholar 

  2. Babbage, C.: Passages from the Life of a Philosopher. Longman, Green, Longman, Roberts, & Green, London (1864)

    Google Scholar 

  3. Bachmann, A., Bird, C., Rahman, F., Devanbu, P., Bernstein, A.: The missing links: bugs and bug-fix commits. In: Proceedings of the ACM SIGSOFT International Symposium on Foundations of Software Engineering, pp. 97–106 (2010). doi:10.1145/1882291.1882308

    Google Scholar 

  4. Bavota, G., De Lucia, A., Marcus, A., Oliveto, R.: Automating Extract Class refactoring: an improved method and its evaluation. Empir. Software Eng. (2013). doi:10.1007/s10664-013-9256-x (in press)

    Google Scholar 

  5. Bird, C., Zimmermann, T.: Assessing the value of branches with what-if analysis. In: Proceedings of the ACM SIGSOFT International Symposium on Foundations of Software Engineering, pp. 45:1–45:11 (2012). doi:10.1145/2393596.2393648

    Google Scholar 

  6. Collinson, M., Monahan, B., Pym, D.: Semantics for structured systems modelling and simulation. In: Proceedings of the ICST International Conference on Simulation Tools and Techniques, pp. 34:1–34:8 (2010). doi:10.4108/ICST.SIMUTOOLS2010.8631

    Google Scholar 

  7. Cossette, B.E., Walker, R.J.: Seeking the ground truth: a retroactive study on the evolution and migration of software libraries. In: Proceedings of the ACM SIGSOFT International Symposium on Foundations of Software Engineering, pp. 55:1–55:11 (2012). doi:10.1145/2393596.2393661

    Google Scholar 

  8. Erzberger, C., Prein, G.: Triangulation: validity and empirically-based hypothesis construction. Qual. Quan. 31(2), 141–154 (1997). doi:10.1023/A:1004249313062

    Article  Google Scholar 

  9. Feinstein, A.H., Cannon, H.M.: Constructs of simulation evaluation. Simulat. Gaming 33(4), 425–440 (2002). doi:10.1177/1046878102238606

    Article  Google Scholar 

  10. Foss, T., Stensrud, E., Kitchenham, B., Myrtveit, I.: A simulation study of the model evaluation criterion MMRE. IEEE Trans. Software Eng. 29(11), 985–995 (2003). doi:10.1109/TSE.2003.1245300

    Article  Google Scholar 

  11. Hlupic, V., Irani, Z., Paul, R.J.: Evaluation framework for simulation software. Int. J. Adv. Manuf. Technol. 15(5), 366–382 (1999). doi:10.1007/s001700050079

    Article  Google Scholar 

  12. Holmes, R., Ratchford, T., Robillard, M., Walker, R.J.: Automatically recommending triage decisions for pragmatic reuse tasks. In: Proceedings of the IEEE/ACM International Conference on Automated Software Engineering, pp. 397–408 (2009). doi:10.1109/ASE.2009.65

    Google Scholar 

  13. Holmes, R., Walker, R.J.: Customized awareness: recommending relevant external change events. In: Proceedings of the ACM/IEEE International Conference on Software Engineering, pp. 465–474 (2010). doi:10.1145/1806799.1806867

    Google Scholar 

  14. Holmes, R., Walker, R.J.: Systematizing pragmatic software reuse. ACM Trans. Software Eng. Methodol. 21(4), 20:1–20:44 (2012). doi:10.1145/2377656.2377657

    Google Scholar 

  15. Holmes, R., Walker, R.J., Murphy, G.C.: Approximate structural context matching: an approach to recommend relevant examples. IEEE Trans. Software Eng. 32(12), 952–970 (2006). doi:10.1109/TSE.2006.117

    Article  Google Scholar 

  16. Hughes, R.I.G.: The Ising model, computer simulation, and universal physics. In: Morgan, M.S., Morrison, M. (eds.) Models as Mediators: Perspectives on Natural and Social Science, No. 52 in Ideas in Context, Chap. 5. Cambridge University Press, Cambridge (1999). doi:10.1017/CBO9780511660108.006

    Google Scholar 

  17. Kelley, J.F.: An iterative design methodology for user-friendly natural language office information applications. ACM Trans. Inform. Syst. 2(1), 26–41 (1984). doi:10.1145/357417.357420

    Article  MathSciNet  Google Scholar 

  18. Kim, S., Zimmermann, T., Whitehead Jr., E.J., Zeller, A.: Predicting faults from cached history. In: Proceedings of the ACM/IEEE International Conference on Software Engineering, pp. 489–498 (2007). doi:10.1109/ICSE.2007.66

    Google Scholar 

  19. Kononenko, O., Dietrich, D., Sharma, R., Holmes, R.: Automatically locating relevant programming help online. In: Proceedings of the IEEE Symposium on Visual Languages and Human-Centric Computing, pp. 127–134 (2012). doi:10.1109/VLHCC.2012.6344497

    Google Scholar 

  20. Marcus, A., Poshyvanyk, D., Ferenc, R.: Using the conceptual cohesion of classes for fault prediction in object-oriented systems. IEEE Trans. Software Eng. 34(2), 287–300 (2008). doi:10.1109/TSE.2007.70768

    Article  Google Scholar 

  21. Matejka, J., Li, W., Grossman, T., Fitzmaurice, G.: CommunityCommands: command recommendations for software applications. In: Proceedings of the ACM Symposium on User Interface Software and Technology, pp. 193–202 (2009). doi:10.1145/1622176.1622214

    Google Scholar 

  22. Mosteller, F.: A k-sample slippage test for an extreme population. Ann. Math. Stat. 19(1), 58–65 (1948). doi:10.1214/aoms/1177730290

    Article  MATH  MathSciNet  Google Scholar 

  23. Murphy-Hill, E., Jiresal, R., Murphy, G.C.: Improving software developers’ fluency by recommending development environment commands. In: Proceedings of the ACM SIGSOFT International Symposium on Foundations of Software Engineering, pp. 42:1–42:11 (2012). doi:10.1145/2393596.2393645

    Google Scholar 

  24. Murphy-Hill, E., Murphy, G.C.: Recommendation delivery: getting the user interface just right. In: Robillard, M., Maalej, W., Walker, R.J., Zimmermann, T. (eds.) Recommendation Systems in Software Engineering, Chap. 9. Springer, New York (2014)

    Google Scholar 

  25. Robillard, M.P.: Topology analysis of software dependencies. ACM Trans. Software Eng. Methodol. 17(4), 18:1–18:36 (2008). doi:10.1145/13487689.13487691

    Google Scholar 

  26. Robillard, M., Walker, R.J.: An introduction to recommendation systems in software engineering. In: Robillard, M., Maalej, W., Walker, R.J., Zimmermann, T. (eds.) Recommendation Systems in Software Engineering, Chap. 1. Springer, New York (2014)

    Google Scholar 

  27. Robillard, M.P., Walker, R.J., Zimmermann, T.: Recommendation systems for software engineering. IEEE Software 27(4), 80–86 (2010). doi:10.1109/MS.2009.161

    Article  Google Scholar 

  28. Said, A., Tikk, D., Cremonesi, P.: Benchmarking: a methodology for ensuring the relative quality of recommendation systems in software engineering. In: Robillard, M., Maalej, W., Walker, R.J., Zimmermann, T. (eds.) Recommendation Systems in Software Engineering, Chap. 11. Springer, New York (2014)

    Google Scholar 

  29. Sánchez, P.J.: Fundamentals of simulation modeling. In: Proceedings of the Winter Simulation Conference, pp. 54–62 (2007). doi:10.1109/WSC.2007.4419588

    Google Scholar 

  30. Sebastiani, F.: Machine learning in automated text categorization. ACM Comput. Surv. 34(1), 1–47 (2002). doi:10.1145/505282.505283

    Article  Google Scholar 

  31. Shepperd, M., Kadoda, G.: Using simulation to evaluate prediction techniques. In: Proceedings of the IEEE International Symposium on Software Metrics, pp. 349–359 (2001). doi:10.1109/METRIC.2001.915542

    Google Scholar 

  32. Teorey, T.J., Merten, A.G.: Considerations on the level of detail in simulation. In: Proceedings of the Symposium on Simulation of Computer Systems, pp. 137–143 (1973)

    Google Scholar 

  33. Tosun Mısırlı, A., Bener, A., Çağlayan, B., Çalıklı, G., Turhan, B.: Field studies: a methodology for construction and evaluation of recommendation systems in software engineering. In: Robillard, M., Maalej, W., Walker, R.J., Zimmermann, T. (eds.) Recommendation Systems in Software Engineering, Chap. 13. Springer, New York (2014)

    Google Scholar 

  34. van Rijsbergen, C.J.: Information Retrieval. 2nd edn. Butterworth–Heinemann, London (1979)

    Google Scholar 

  35. Winsberg, E.: Simulated experiments: methodology for a virtual world. Philos. Sci. 70(1), 105–125 (2003). doi:10.1086/367872

    Article  Google Scholar 

  36. Ye, Y., Yamamoto, Y., Nakakoji, K., Nishinaka, Y., Asada, M.: Searching the library and asking the peers: learning to use Java APIs on demand. In: Proceedings of the International Symposium on Principles and Practice of Programming in Java, pp. 41–50 (2007). doi:10.1145/1294325.1294332

    Google Scholar 

  37. Zimmermann, T., Weißgerber, P., Diehl, S., Zeller, A.: Mining version histories to guide software changes. IEEE Trans. Software Eng. 31(6), 429–445 (2005). doi:10.1109/TSE.2005.72

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science, University of Calgary, Calgary, AB, Canada

    Robert J. Walker

  2. David R. Cheriton School of Computer Science, University of Waterloo, Waterloo, ON, Canada

    Reid Holmes

Authors
  1. Robert J. Walker
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Reid Holmes
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Robert J. Walker .

Editor information

Editors and Affiliations

  1. McGill University, Montréal, Québec, Canada

    Martin P. Robillard

  2. University of Hamburg, Hamburg, Germany

    Walid Maalej

  3. University of Calgary, Calgary, Alberta, Canada

    Robert J. Walker

  4. Microsoft Research, Redmond, Washington, USA

    Thomas Zimmermann

Rights and permissions

Reprints and permissions

Copyright information

© 2014 Springer-Verlag Berlin Heidelberg

About this chapter

Cite this chapter

Walker, R.J., Holmes, R. (2014). Simulation. In: Robillard, M., Maalej, W., Walker, R., Zimmermann, T. (eds) Recommendation Systems in Software Engineering. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-45135-5_12

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-45135-5_12

  • Published:

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-45134-8

  • Online ISBN: 978-3-642-45135-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation