Software Quality Journal

, Volume 25, Issue 4, pp 1091–1143 | Cite as

Industry–academia collaborations in software testing: experience and success stories from Canada and Turkey

  • Vahid Garousi
  • Matt M. Eskandar
  • Kadir Herkiloğlu
Article
First Online:

Abstract

Collaboration between industry and academia supports improvement and innovation in industry and helps to ensure industrial relevance in academic research. However, many researchers and practitioners believe that the level of joint industry–academia collaborations (IAC) in software engineering (SE) is still relatively very low, compared to the amount of activity in each of the two communities. The goal of the empirical study reported in this paper is to characterize a set of collaborative industry–academia R&D projects in the area of software testing conducted by the authors (based in Canada and Turkey) with respect to a set of challenges, patterns and anti-patterns identified by a recent Systematic Literature Review study, with the aim of contributing to the body of evidence in the area of IAC, for the benefit of SE researchers and practitioners in conducting successful IAC projects in software testing and in software engineering in general. To address the above goal, a pool of ten IAC projects (six completed, two failed and two ongoing) all in the area of software testing, which the authors have led or have had active roles in, were selected as objects of study and were analyzed (both quantitatively and qualitatively) with respect to the set of selected challenges, patterns and anti-patterns. As outputs, the study presents a set of empirical findings and evidence-based recommendations, e.g.: it has been observed that even if an IAC project may seem perfect from many aspects, one single major challenge (e.g., disagreement in confidentiality agreements) can lead to its failure. Thus, we recommend that both parties (academics and practitioners) consider all the challenges early on and proactively work together to eliminate the risk of challenges in IAC projects. We furthermore report correlation and interrelationship of challenges, patterns and anti-patterns with project success measures. This study hopes to encourage and benefit other SE researchers and practitioners in conducting successful IAC projects in software testing and in software engineering in general in the future.

Keywords

Industry–academia collaborations Software engineering Software testing Empirical study Experience report Evidence Challenges Success factors (patterns) Anti-patterns 
This is a preview of subscription content, log in to check access.

Notes

Acknowledgments

Vahid Garousi was partially supported by several internal grants by the Hacettepe University and the Scientific and Technological Research Council of Turkey (TÜBİTAK).

References

  1. Adler, M., & Ziglio, E. (1996). Gazing into the Oracle: The Delphi method and its application to social policy and public health. London: Jessica Kingsley Publishers.Google Scholar
  2. Ahimbisibwe, A., Cavana, R. Y., & Daellenbach, U. (2015). A contingency fit model of critical success factors for software development projects. Journal of Enterprise Information Management, 28, 7–33.CrossRefGoogle Scholar
  3. Akdur, D., & Garousi, V. (2015). Model-driven engineering in support of development, test and maintenance of middleware communication protocols: A preliminary approach. In international conference on model-driven engineering and software development (MODELSWARD), Doctoral Consortium (pp. 11–19).Google Scholar
  4. Amannejad, Y., Garousi, V., Irving, R., & Sahaf, Z. (2014). A search-based approach for cost-effective software test automation decision support and an industrial case study. In Proceedings of international workshop on regression testing, co-located with the sixth ieee international conference on software testing, verification, and validation (pp. 302–311).Google Scholar
  5. Ankrah, S., & Al-Tabbaa, O. (2015). Universities–industry collaboration: A systematic review. Scandinavian Journal of Management, 31, 387–408.CrossRefGoogle Scholar
  6. Avison, D. E., Lau, F., Myers, M. D., & Nielsen, P. A. (1999). Action research. Communications of the ACM, 42, 94–97.CrossRefGoogle Scholar
  7. Baldassarre, M.T., Caivano, D., & Visaggio, G. (2013). Empirical studies for innovation dissemination: Ten years of experience. In: Presented at the Proceedings of the International Conference on Evaluation and Assessment in Software Engineering, Porto de Galinhas, Brazil.Google Scholar
  8. Biffl, S., Aurum, A., Boehm, B., Erdogmus, H., & Grünbacher, P. (2006). Value-based software engineering. Berlin: Springer.CrossRefGoogle Scholar
  9. Briand, L.C. (2011). Useful software engineering research-leading a double-agent life. In IEEE international conference on software maintenance (pp. 2–2).Google Scholar
  10. Briand, L. (2012). Embracing the Engineering Side of Software Engineering. IEEE Software, 29, 96.CrossRefGoogle Scholar
  11. Charette, R. N. (2005). Why software fails. IEEE Spectrum, 42, 42–49.CrossRefGoogle Scholar
  12. Chow, T., & Cao, D.-B. (2008). A survey study of critical success factors in agile software projects. Journal of Systems and Software, 81, 961–971.CrossRefGoogle Scholar
  13. Connor, A.M., Buchan, J., & Petrova, K. (2009). Bridging the research-practice gap in requirements engineering through effective teaching and peer learning. In International conference on information technology: new generations (pp. 678–683).Google Scholar
  14. Cruzes, D. S., & Dyba, T. (2011). Research synthesis in software engineering: A tertiary study. Information and Software Technology, 53, 440–455.CrossRefGoogle Scholar
  15. Deutsch, M. S. (1991). An exploratory analysis relating the software project management process to project success. IEEE Transactions on Engineering Management, 38, 365–375.CrossRefGoogle Scholar
  16. Eldh, S. (2013). Some researcher considerations when conducting empirical studies in industry. In International workshop on conducting empirical studies in industry (pp. 69–70).Google Scholar
  17. Enoiu, E.P. & Čaušević, A. (2014). Enablers and impediments for collaborative research in software testing: An empirical exploration. In Presented at the Proceedings of the international workshop on Long-term industrial collaboration on software engineering, Vasteras, Sweden.Google Scholar
  18. Fairhurst, M. (2013). Academia and industry collaboration to drive biometrics boom. Biometric Technology Today, 2013, 10–11.CrossRefGoogle Scholar
  19. Franch Gutiérrez, J., Ameller, D., Ayala Martínez, C. P., & Cabot Sagrera, J. (2012). Bridging the gap among academics and practitioners in non-functional requirements management: Some reflections and proposals for the future. In M. U. Vannerdat (Ed.), Modelling and quality in requirements engineering (pp. 267–273). Münster, Germany: Monsenstein und Vannerdat.Google Scholar
  20. Garousi, V. (2008). Empirical analysis of a genetic algorithm-based stress test technique for distributed real-time systems. In Proceedings of the genetic and evolutionary computation conference, search-based software engineering (SBSE) track (pp. 1743–1750).Google Scholar
  21. Garousi, V. (2010a). An open modern software testing laboratory courseware: An experience report. In Proceedings of the 23rd IEEE conference on software engineering education and training (pp. 177–184).Google Scholar
  22. Garousi, V. (2010b). A genetic algorithm-based stress test requirements generator tool and its empirical evaluation. IEEE Transactions on Software Engineering Special Issue on Search-Based Software Engineering (SBSE), 36, 778–797.Google Scholar
  23. Garousi, V. (2011). Incorporating real-world industrial testing projects in software testing courses: Opportunities, challenges, and lessons learned. In Proceedings of the IEEE conference on software engineering education and training (CSEE&T) (pp. 396–400).Google Scholar
  24. Garousi, V. (2012). Success stories in industrial software testing research and education: A Canadian perspective. In Marmara Research Center, Scientific and Technological Research Council of Turkey (BILGEM, TÜBITAK), Istanbul, Turkey.Google Scholar
  25. Garousi, V., Briand, L., & Labiche, Y. (2008). Traffic-aware stress testing of distributed real-time systems Based on UML models using genetic algorithms. Elsevier Journal of Systems and Software, Special Issue on Model-Based Software Testing, 81, 161–185.Google Scholar
  26. Garousi, V., Coşkunçay, A., Can, A.B., & Demirörs, O. (2013). A survey of software testing practices in Turkey. In Proceedings of the Turkish national software engineering symposium “Ulusal Yazılım Mühendisliği Sempozyumu” (UYMS).Google Scholar
  27. Garousi, V., Coşkunçay, A., Can, A. B., & Demirörs, O. (2015a). A survey of software engineering practices in Turkey. Journal of Systems and Software, 108, 148–177.CrossRefGoogle Scholar
  28. Garousi, V., Petersen, K., Özkan, B. (2015b). Industry–Academia collaborations in software engineering: A systematic literature review. In Technical report, Hacettepe University software engineering research group, HUSE-2015-01. https://drive.google.com/open?id=0B6dKdxaNjBENSWRwRlNJbExYUWc.
  29. Garousi, V., Petersen, K., & Özkan, B. (2015c). Challenges, success patterns and anti-patterns in industry–academia collaborations in software engineering: A systematic literature review. In Technical Report, Hacettepe University Software Engineering Research Group, HUSE-2015-01.Google Scholar
  30. Garousi, V., & Pfahl, D. (2015). When to automate software testing? A decision-support approach based on process simulation. In Press, Wiley Journal of Software: Evolution and Process.Google Scholar
  31. Garousi, V., & Varma, T. (2010). A replicated survey of software testing practices in the Canadian Province of Alberta: What has changed from 2004 to 2009? Journal of Systems and Software, 83, 2251–2262.CrossRefGoogle Scholar
  32. Garousi, V., & Zhi, J. (2013). A survey of software testing practices in Canada. Journal of Systems and Software, 86, 1354–1376.CrossRefGoogle Scholar
  33. Glass, R.L. (2006). Software creativity 2.0: Developer.* Books.Google Scholar
  34. Glass, R.L., & Hunt, A. (2006). Software conflict 2.0: The art and science of software engineering: Developer.* Books.Google Scholar
  35. Gorschek, T., Wohlin, C., Carre, P., & Larsson, S. (2006). A model for technology transfer in practice. IEEE Software, 23, 88–95.CrossRefGoogle Scholar
  36. Grunbacher, P. & Rabiser, R. (2013). Success factors for empirical studies in industry-academia collaboration: A reflection. In International workshop on conducting empirical studies in industry (pp. 27–32).Google Scholar
  37. Guruprasad, K. (2013). Multi-faceted industry–academia collaboration. In Evolving corporate education strategies for developing countries: The Role of Universities (pp. 90–102).Google Scholar
  38. Harrigan, M. T. (2003). Industrial biotechnology education: A model of collaboration between industry and academia. Biochemistry and Molecular Biology Education, 31, 142–144.CrossRefGoogle Scholar
  39. Jain, S., Ali Babar, M., & Fernandez, J. (2013). Conducting empirical studies in industry: Balancing rigor and relevance. In International workshop on conducting empirical studies in industry (pp. 9–14).Google Scholar
  40. Jolly, S.A., Garousi, V., & Eskandar, M.M. (2012). Automated unit testing of a SCADA control software: An industrial case study based on action research. In IEEE international conference on software testing, verification and validation (ICST) (pp. 400–409).Google Scholar
  41. Jugdev, K., & Müller, R. (2005). A retrospective look at our evolving understanding of project success. Project Management Journal, 36, 19–31.Google Scholar
  42. Kaindl, H., Brinkkemper, S., Bubenko, J. A, Jr, Farbey, B., Greenspan, S. J., Heitmeyer*, C. L., et al. (2002). Requirements engineering and technology transfer: Obstacles, incentives and improvement agenda. Requirements Engineering, 7, 113–123.CrossRefGoogle Scholar
  43. Kanso, A., & Monette, D. (2014). Foundations for long-term collaborative research. In Presented at the proceedings of the international workshop on Long-term industrial collaboration on software engineering, Vasteras, Sweden.Google Scholar
  44. Kato, Y. (2005). Industry–academia collaboration program for commercialization of new dry cleaning system. In AIST Today (International Edition) (p. 11).Google Scholar
  45. Krishnan, P., Ross, K., & Pari-Salas, P. (2009). Industry academia collaboration: An experience report at a small University. In Proceedings of conference on software engineering education and training (pp. 117–121).Google Scholar
  46. Lamprecht, S.J., & Van Rooyen, G.J. (2012). Models for technology research collaboration between industry and academia in South Africa. In Proceedings of the software engineering colloquium (pp. 11–17).Google Scholar
  47. Leszak, M., Perry, D. E., & Stoll, D. (2002). Classification and evaluation of defects in a project retrospective. Journal of Systems and Software, 61, 173–187.CrossRefGoogle Scholar
  48. Lunsford, T. R., & Lunsford, B. R. (1995). The research sample, part I: Sampling. Journal of Prosthetics and Orthotics, 7, 105–112.Google Scholar
  49. Martínez-Fernández, S. & Marques, H.M. (2014). Practical experiences in designing and conducting empirical studies in industry–academia collaboration. In Presented at the proceedings of the international workshop on conducting empirical studies in industry, Hyderabad, India.Google Scholar
  50. Maxwell, J. C., & Dornan, J. (2013). How to influence people: Make a difference in your world. Nashville: Thomas Nelson Inc.Google Scholar
  51. Misirli, A.T., Erdogmus, H., Juristo, N. & Dieste, O. (2014). Topic selection in industry experiments. In Presented at the proceedings of the international workshop on conducting empirical studies in industry, Hyderabad, India.Google Scholar
  52. Morris, P., Masera, M., & Wilikens, M. (1998). Requirements engineering and industrial uptake. In Proceedings of international conference on requirements engineering (pp. 130–137).Google Scholar
  53. Mujumdar, A. S. (2010). Editorial on industry–academia collaboration in R&D. Drying Technology, 28, 431–432.CrossRefGoogle Scholar
  54. Mullins, C. B. (1985). A university administrator’s view of academia–industry collaboration. Circulation, 72, I8–12.CrossRefGoogle Scholar
  55. O’Leary, C., Lawless, D., Gordon, D., Li, H., & Bechkoum, K. (2006). Developing a software engineering curriculum for the emerging software industry in China. In 19th Conference on software engineering education and training, 2006. Proceedings. (pp. 115–122).Google Scholar
  56. Osterweil, L. J., Ghezzi, C., Kramer, J., & Wolf, A. L. (2008). Determining the impact of software engineering research on practice. IEEE Computer, 41, 39–49.CrossRefGoogle Scholar
  57. Palkar, S. (2013). Industry–academia collaboration, expectations, and experiences. ACM Inroads, 4, 56–58.CrossRefGoogle Scholar
  58. Perkmann, M., Tartari, V., McKelvey, M., Autio, E., Broström, A., D’Este, P., et al. (2013). Academic engagement and commercialisation: A review of the literature on university–industry relations. Research Policy, 42, 423–442.CrossRefGoogle Scholar
  59. Petersen, K., & Engstre, E. (2014). Finding relevant research solutions for practical problems: The serp taxonomy architecture. In presented at the Proceedings of the international workshop on Long-term industrial collaboration on software engineering, Vasteras, Sweden.Google Scholar
  60. Petersen, K., Gencel, C., Asghari, N., Baca, D., & Betz, S. (2014). Action research as a model for industry–academia collaboration in the software engineering context. In Proceedings of the ACM international workshop on long-term industrial collaboration on software engineering (pp. 55–62).Google Scholar
  61. Pfleeger, S. L. (1999). Understanding and improving technology transfer in software engineering. Journal of Systems and Software, 47, 111–124.CrossRefGoogle Scholar
  62. Pinheiro, C., Garousi, V., Maurer, F., & Sillito, J. (2010). Introducing automated environment configuration testing in an industrial setting. In Proceedings of the international conference on software engineering and knowledge engineering, workshop on software test automation, practice, and standardization (pp. 186–191).Google Scholar
  63. Price, F. D. (1985). Industry and academia in collaboration: the Pfizer experience. Circulation, 72, I13–I17.Google Scholar
  64. Raschke, W., Zilli, M., Loinig, J., Weiss, R., Steger, C., & Kreiner, C. (2014). Embedding research in the industrial field: A case of a transition to a software product line. In Presented at the Proceedings of the international workshop on Long-term industrial collaboration on software engineering, Vasteras, Sweden.Google Scholar
  65. Reel, J. S. (1999). Critical success factors in software projects. IEEE Software, 16, 18–23.CrossRefGoogle Scholar
  66. Rombach, D., & Achatz, R. (2007). Research collaborations between academia and industry. In Symposium on future of software engineering (pp. 29–36).Google Scholar
  67. Rombach, D., Ciolkowski, M., Jeffery, R., Laitenberger, O., McGarry, F., & Shull, F. (2008). Impact of research on practice in the field of inspections, reviews and walkthroughs: Learning from successful industrial uses. ACM SIGSOFT Software Engineering Notes, 33, 26–35.CrossRefGoogle Scholar
  68. Runeson, P. (2012). It takes two to tango: An experience report on industry–academia collaboration. In Proceedings of international conference on software testing, verification and validation, (pp. 872–877).Google Scholar
  69. Runeson, P., Host, M., Rainer, A., & Regnell, B. (2012). Case study research in software engineering: Guidelines and examples. New York: Wiley.CrossRefGoogle Scholar
  70. Runeson, P., & Minör, S. (2014). The 4 + 1 view model of industry-academia collaboration. In Proceedings of the ACM international workshop on long-term industrial collaboration on software engineering, co-located with ASE 2014 (pp. 21–24).Google Scholar
  71. Runeson, P., Minör, S., & Svenér, J. (2014). Get the cogs in synch-time horizon aspects of industry–academia collaboration. In Proceedings of the ACM international workshop on long-term industrial collaboration on software engineering, co-located with ASE 2014 (pp. 25–28).Google Scholar
  72. Sahaf, Z., Garousi, V., Pfahl, D., Irving, R., & Amannejad, Y. (2014). When to automate software testing? Decision support based on system dynamics—An industrial case study. In Proceedings of international conference on software and systems process (pp. 149–158).Google Scholar
  73. Sandberg, A., Pareto, L., & Arts, T. (2011). Agile collaborative research: Action principles for industry–academia collaboration. IEEE Software, 28, 74–83.CrossRefGoogle Scholar
  74. Solingen, R. V., & Berghout, E. (1999). The Goal/Question/Metric Method: A practical guide for quality improvement of software development. New York: McGraw-Hill.Google Scholar
  75. Subrahmanyam, G.V.B. (2009). A dynamic framework for software engineering education curriculum to reduce the gap between the software organizations and software educational institutions. In CSEET ‘09. 22nd conference on software engineering education and training, 2009 (pp. 248–254).Google Scholar
  76. Sullivan, G. M., & Artino, A. R. (2013). Analyzing and interpreting data from likert-type scales. Journal of Graduate Medical Education, 5, 541–542.CrossRefGoogle Scholar
  77. Thomas, C. J., & McKew, J. C. (2014). Playing well with others! Initiating and sustaining successful collaborations between industry, academia and government. Current Topics in Medicinal Chemistry, 14, 291–293.CrossRefGoogle Scholar
  78. Wiederseiner, C., Garousi, V., & Smith, M. (2011). Tool support for automated traceability analysis in embedded software systems. In Proceedings of IEEE international conference on embedded software and system (ICESS) (pp. 1109–1117).Google Scholar
  79. Wiederseiner, C., Jolly, S.A., & Garousi, V. (2012). AutoBBUT—Automated code generation for black-box unit testing. http://code.google.com/p/autobbut/. Last accessed Sept. 2012.
  80. Wiederseiner, C., Jolly, S.A., Garousi, V., & Eskandar, M.M. (2010). An open-source tool for automated generation of black-box xunit test code and its industrial evaluation. In Proceedings of the international conference on testing: academic and industrial conference—Practice and research techniques (TAIC PART) (pp. 118–128).Google Scholar
  81. Wohlin, C. (2013a). Empirical software engineering research with industry: Top 10 challenges. In International workshop on conducting empirical studies in industry (pp. 43–46).Google Scholar
  82. Wohlin, C. (2013b). Software Engineering Research under the Lamppost. In Proceedings of the international joint conference on software technologies.Google Scholar
  83. Wohlin, C., Aurum, A., Angelis, L., Phillips, L., Dittrich, Y., Gorschek, T., et al. (2012). The success factors powering industry–academia collaboration. IEEE Software, 29, 67–73.CrossRefGoogle Scholar
  84. Wohlin, C., & Regnell, B. (1999). Strategies for industrial relevance in software engineering education. Journal of Systems and Software, 49, 125–134.CrossRefGoogle Scholar
  85. Wohlin, C., Runeson, P., Höst, M., Ohlsson, M. C., Regnell, B., & Wesslén, A. (2000). Experimentation in software engineering: An introduction. Berlin: Kluwer.CrossRefMATHGoogle Scholar
  86. Yeo, K. T. (2002). Critical failure factors in information system projects. International Journal of Project Management, 20, 241–246.CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2016

Authors and Affiliations

  1. 1.Software Engineering Research Group, Department of Computer EngineeringHacettepe UniversityAnkaraTurkey
  2. 2.Maral Software Engineering Consulting CorporationCalgaryCanada
  3. 3.MR Control Systems International Inc.CalgaryCanada
  4. 4.Quality, Test and Process Management DirectorateHAVELSAN A.Ş.AnkaraTurkey

Personalised recommendations