Software Quality Journal

, Volume 25, Issue 4, pp 1239–1268 | Cite as

Onboarding inexperienced developers: struggles and perceptions regarding automated testing

  • Raphael PhamEmail author
  • Stephan Kiesling
  • Leif Singer
  • Kurt Schneider


Previous research found that inexperienced software engineers may tend to view automatic testing as a waste of time and as an activity completely separate from programming. This could have a negative impact on their later careers and could be a sign that improvements in software engineering education are needed when it comes to testing. At the same time, this stance could negatively influence the perception that practitioners have of recent university graduates. To explore this issue, we conducted a qualitative study and surveyed 170 and interviewed 22 practitioners about their experiences with recent graduates, focusing on software testing skills. We find that practitioners do recognize a skill gap between university graduates and industry expectations and that this perception could be engrained deeply enough already to influence hiring practices. Practitioners use different and at times costly strategies to alleviate this skill gap, such as training and mentoring efforts. We validated core findings in a survey with 698 professional software developers. Our qualitative insights can help industry, research, and educational institutions guide in-depth studies that explore the severity of the effects we have found. The coping strategies we have found can provide valuable starting points that can inform changes in how we educate the software engineers of the future.


Automated testing Onboarding Newcomers Software developer Required skills 


  1. Begel, A., & Simon, B. (2008a). Novice software developers, all over again. In Proceedings of the fourth international workshop on computing education research (pp. 3–14). ACM.Google Scholar
  2. Begel, A., & Simon, B. (2008b). Struggles of new college graduates in their first software development job. In ACM SIGCSE Bulletin (Vol. 40, pp. 226–230). ACM.Google Scholar
  3. Boud, D., & Feletti, G. (1998). The challenge of problem-based learning. Hove: Psychology Press.Google Scholar
  4. Brechner, E. (2003). Things they would not teach me of in college: What microsoft developers learn later. In Companion of the 18th annual ACM SIGPLAN conference on object-oriented programming, systems, languages, and applications (pp. 134–136). ACM.Google Scholar
  5. Bruegge, B. (1992). Teaching an industry-oriented software engineering course. In C. Sledge (Ed.), Software engineering education, lecture notes in computer science (Vol. 640, pp. 63–87). Berlin: Springer. doi: 10.1007/3-540-55963-9_40.
  6. Carrington, D., & Kim, S. K. (2003). Teaching software design with open source software. In FIE 2003 33rd annual frontiers in education, 2003 (Vol. 3, p. S1C-9). IEEE.Google Scholar
  7. Dagenais, B., Ossher, H., Bellamy, R. K., Robillard, M. P., & De Vries, J. P. (2010). Moving into a new software project landscape. In Proceedings of the 32nd ACM/IEEE international conference on software engineering (Vol. 1, pp. 275–284). ACM.Google Scholar
  8. Deak, A., & Sindre, G. (2014). Analyzing the importance of teaching about testing from alumni survey data. Norsk informatikkonferanse (NIK) 2013.Google Scholar
  9. DeMarco, T., & Lister, T. (1987). Peopleware: Productive people and teams. Oxford: Dorset House.Google Scholar
  10. Edwards, S. H. (2003a). Rethinking computer science education from a test-first perspective. In Companion of the 18th annual ACM SIGPLAN conference on object-oriented programming, systems, languages, and applications (pp. 148–155). ACM.Google Scholar
  11. Edwards, S. H. (2003b). Using test-driven development in the classroom: Providing students with automatic, concrete feedback on performance. In Proceedings of the international conference on education and information systems: technologies and applications EISTA (Vol. 3).Google Scholar
  12. Elbaum, S., Person, S., Dokulil, J., & Jorde, M., (2007). Bug hunt: Making early software testing lessons engaging and affordable. In 29th international conference on software engineering, 2007. ICSE 2007 (pp. 688–697). IEEE.Google Scholar
  13. Fagerholm, F., Guinea, A.S., Münch, J., & Borenstein, J. (2014). The role of mentoring and project characteristics for onboarding in open source software projects. In Proceedings of the 8th ACM/IEEE international symposium on empirical software engineering and measurement (p. 55). ACM.Google Scholar
  14. Fagerholm, F., Johnson, P., Guinea, A. S., Borenstein, J., & Munch, J. (2013). Onboarding in open source software projects: A preliminary analysis. In 2013 IEEE 8th international conference on global software engineering workshops (ICGSEW) (pp. 5–10). IEEE.Google Scholar
  15. Garousi, V., & Felderer, M. (2016). Developing, verifying, and maintaining high-quality automated test scripts. IEEE Software, 33(3), 68–75.CrossRefGoogle Scholar
  16. Glaser, B., & Strauss, A. (1967). The discovery of grounded theory: Strategies for qualitative research (Vol. 3). Chicago: Aldine de Gruyter.Google Scholar
  17. Glass, R. L., Collard, R., Bertolino, A., Bach, J., & Kaner, C. (2006). Software testing and industry needs. IEEE Software, 23(4), 55–57.CrossRefGoogle Scholar
  18. Janzen, D. S., & Saiedian, H. (2006). Test-driven learning: Intrinsic integration of testing into the CS/SE curriculum. In In ACM SIGCSE bulletin (Vol. 38, pp. 254–258). ACM.Google Scholar
  19. Johnson, M., & Senges, M. (2010). Learning to be a programmer in a complex organization: A case study on practice-based learning during the onboarding process at google. Journal of Workplace Learning, 22(3), 180–194.CrossRefGoogle Scholar
  20. Jones, E. L. (2001). An experiential approach to incorporating software testing into the computer science curriculum. In 31st annual frontiers in education conference, 2001 (Vol. 2, p. F3D-7). IEEE.Google Scholar
  21. Jones, E. L., & Chatmon, C. L. (2001). A perspective on teaching software testing. In Journal of computing sciences in colleges, consortium for computing sciences in colleges (Vol. 16, pp. 92–100).Google Scholar
  22. Kaner, C., & Padmanabhan, S. (2007). Practice and transfer of learning in the teaching of software testing. In 20th conference on software engineering education and training, 2007. CSEET’07 (pp. 157–166). IEEE.Google Scholar
  23. Lübke, D., Flohr, T., & Schneider, K. (2004). Serious insights through fun software-projects. In T. Dingsøyr (Ed.), Software process improvement, lecture notes in computer science (Vol. 3281, pp. 57–68). Berlin: Springer. doi: 10.1007/978-3-540-30181-3_6.
  24. Lustosa Neto, V., Coelho, R., Leite, L., Guerrero, D. S., & Mendonça, A. P. (2013). Popt: A problem-oriented programming and testing approach for novice students. In 2013 35th international conference on software engineering (ICSE) (pp. 1099–1108). IEEE.Google Scholar
  25. Marrero, W., & Settle, A. (2005). Testing first: Emphasizing testing in early programming courses. In ACM SIGCSE bulletin (Vol. 37, pp. 4–8). ACM.Google Scholar
  26. Ng, S., Murnane, T., Reed, K., Grant, D., & Chen, T. (2004). A preliminary survey on software testing practices in Australia. In Software engineering conference, 2004. Proceedings. 2004 Australian (pp. 116–125). IEEE.Google Scholar
  27. Pham, R., Kiesling, S., Liskin, O., Singer, L., & Schneider, K. (2014). Enablers, inhibitors, and perceptions of testing in novice software teams. In 22th international symposium on the foundations of software engineering (FSE 2014), Hong Kong, China. (to appear).Google Scholar
  28. Pham, R., Mörschbach, J., & Schneider, K. (2015a). Communicating software testing culture through visualizing testing activity. In 7th international workshop on social software engineering (SSE 2015). ACM, SSE 2015. doi: 10.1145/2804381.2804382.
  29. Pham, R., Singer, L., Liskin, O., Figueira Filho, F., & Schneider, K. (2013). Creating a shared understanding of testing culture on a social coding site. In International conference on software engineering (ICSE) (pp. 112–121). IEEE Press.Google Scholar
  30. Pham, R., Stoliar, Y., & Schneider, K. (2015b). Automatically recommending test code examples to inexperienced developers. In Proceedings of the 10th joint meeting of the european software engineering conference and the ACM SIGSOFT symposium on the foundations of software engineering (ESEC/FSE 2015). ACM, ESEC/FSE 2015. doi: 10.1145/2786805.2803202.
  31. Radermacher, A., Walia, G., & Knudson, D. (2014). Investigating the skill gap between graduating students and industry expectations. In Companion Proceedings of the 36th international conference on software engineering (pp. 291–300) ACM.Google Scholar
  32. Rajlich, V. (2013). Teaching developer skills in the first software engineering course. In 2013 35th international conference on software engineering (ICSE) (pp. 1109–1116). IEEE.Google Scholar
  33. Rogers, E. M. (2003). Diffusion of innovations (5th ed.). New York: Free Press.Google Scholar
  34. Shepard, T., Lamb, M., & Kelly, D. (2001). More testing should be taught. Communications of the ACM, 44(6), 103–108.CrossRefGoogle Scholar
  35. Sim, S. E., & Holt, R. C. (1998). The ramp-up problem in software projects: A case study of how software immigrants naturalize. In Proceedings of the international conference on software engineering, 1998 (pp. 361–370). IEEE.Google Scholar
  36. Singer, L., Figueira Filho, F., Cleary, B., Treude, C., Storey, M. A., & Schneider, K. (2013). Mutual assessment in the social programmer ecosystem: An empirical investigation of developer profile aggregators. In Proceedings of the 2013 conference on computer supported cooperative work. CSCW ’13 (pp. 103–116). NY: ACM. doi: 10.1145/2441776.2441791.
  37. Sowe, S. K., Stamelos, I., & Deligiannis, I. (2006). A framework for teaching software testing using f/oss methodology. Berlin: Springer. (In Open Source Systems).CrossRefGoogle Scholar
  38. Spacco, J., & Pugh, W. (2006). Helping students appreciate test-driven development (TDD). In Companion to the 21st ACM SIGPLAN symposium on object-oriented programming systems, languages, and applications (pp. 907–913). ACM.Google Scholar
  39. Steinmacher, I., Wiese, I. S., Conte, T., Gerosa, M. A., & Redmiles, D. (2014). The hard life of open source software project newcomers. In Proceedings of the 7th international workshop on cooperative and human aspects of software engineering (pp. 72–78). ACM.Google Scholar
  40. Stuart, H. C., Dabbish, L., Kiesler, S., Kinnaird, P., & Kang, R. (2012a). Social transparency in networked information exchange: A theoretical framework. In Proceedings of the ACM 2012 conference on computer supported cooperative work. CSCW ’12 (pp. 451–460). New York, NY: ACM. doi: 10.1145/2145204.2145275.
  41. Stuart, H. C., Dabbish, L., Kiesler, S., Kinnaird, P., & Kang, R. (2012b). Social transparency in networked information exchange: A theoretical framework. doi: 10.1145/2145204.2145275.Google Scholar
  42. van Deursen, A., Nederlof, A., & Bouwers, E. (2013). Teaching software architecture: With GitHub!. Retrieved August 24, 2015, from
  43. Whittaker, J. A., Arbon, J., & Carollo, J. (2012). How Google tests software. Boston: Addison-Wesley.Google Scholar
  44. Zagalsky, A., Feliciano, J., Storey, M. D., Zhao, Y., & Wang, W. (2015). The emergence of github as a collaborative platform for education. In Proceedings of the 18th ACM conference on computer supported cooperative work & social computing, CSCW 2015, Vancouver, BC, Canada (pp. 1906–1917), March 14–18, 2015. doi: 10.1145/2675133.2675284.

Copyright information

© Springer Science+Business Media New York 2016

Authors and Affiliations

  1. 1.HanoverGermany
  2. 2.University of VictoriaVictoriaCanada

Personalised recommendations