Skip to main content
SpringerLink
Log in

Industry–academia collaborations in software testing: experience and success stories from Canada and Turkey

  • Published:
Software Quality Journal Aims and scope Submit manuscript

Abstract

Collaboration between industry and academia supports improvement and innovation in industry and helps to ensure industrial relevance in academic research. However, many researchers and practitioners believe that the level of joint industry–academia collaborations (IAC) in software engineering (SE) is still relatively very low, compared to the amount of activity in each of the two communities. The goal of the empirical study reported in this paper is to characterize a set of collaborative industry–academia R&D projects in the area of software testing conducted by the authors (based in Canada and Turkey) with respect to a set of challenges, patterns and anti-patterns identified by a recent Systematic Literature Review study, with the aim of contributing to the body of evidence in the area of IAC, for the benefit of SE researchers and practitioners in conducting successful IAC projects in software testing and in software engineering in general. To address the above goal, a pool of ten IAC projects (six completed, two failed and two ongoing) all in the area of software testing, which the authors have led or have had active roles in, were selected as objects of study and were analyzed (both quantitatively and qualitatively) with respect to the set of selected challenges, patterns and anti-patterns. As outputs, the study presents a set of empirical findings and evidence-based recommendations, e.g.: it has been observed that even if an IAC project may seem perfect from many aspects, one single major challenge (e.g., disagreement in confidentiality agreements) can lead to its failure. Thus, we recommend that both parties (academics and practitioners) consider all the challenges early on and proactively work together to eliminate the risk of challenges in IAC projects. We furthermore report correlation and interrelationship of challenges, patterns and anti-patterns with project success measures. This study hopes to encourage and benefit other SE researchers and practitioners in conducting successful IAC projects in software testing and in software engineering in general in the future.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17

Similar content being viewed by others

References

  • Adler, M., & Ziglio, E. (1996). Gazing into the Oracle: The Delphi method and its application to social policy and public health. London: Jessica Kingsley Publishers.

    Google Scholar 

  • Ahimbisibwe, A., Cavana, R. Y., & Daellenbach, U. (2015). A contingency fit model of critical success factors for software development projects. Journal of Enterprise Information Management, 28, 7–33.

    Article  Google Scholar 

  • Akdur, D., & Garousi, V. (2015). Model-driven engineering in support of development, test and maintenance of middleware communication protocols: A preliminary approach. In international conference on model-driven engineering and software development (MODELSWARD), Doctoral Consortium (pp. 11–19).

  • Amannejad, Y., Garousi, V., Irving, R., & Sahaf, Z. (2014). A search-based approach for cost-effective software test automation decision support and an industrial case study. In Proceedings of international workshop on regression testing, co-located with the sixth ieee international conference on software testing, verification, and validation (pp. 302–311).

  • Ankrah, S., & Al-Tabbaa, O. (2015). Universities–industry collaboration: A systematic review. Scandinavian Journal of Management, 31, 387–408.

    Article  Google Scholar 

  • Avison, D. E., Lau, F., Myers, M. D., & Nielsen, P. A. (1999). Action research. Communications of the ACM, 42, 94–97.

    Article  Google Scholar 

  • Baldassarre, M.T., Caivano, D., & Visaggio, G. (2013). Empirical studies for innovation dissemination: Ten years of experience. In: Presented at the Proceedings of the International Conference on Evaluation and Assessment in Software Engineering, Porto de Galinhas, Brazil.

  • Biffl, S., Aurum, A., Boehm, B., Erdogmus, H., & Grünbacher, P. (2006). Value-based software engineering. Berlin: Springer.

    Book  Google Scholar 

  • Briand, L.C. (2011). Useful software engineering research-leading a double-agent life. In IEEE international conference on software maintenance (pp. 2–2).

  • Briand, L. (2012). Embracing the Engineering Side of Software Engineering. IEEE Software, 29, 96.

    Article  Google Scholar 

  • Charette, R. N. (2005). Why software fails. IEEE Spectrum, 42, 42–49.

    Article  Google Scholar 

  • Chow, T., & Cao, D.-B. (2008). A survey study of critical success factors in agile software projects. Journal of Systems and Software, 81, 961–971.

    Article  Google Scholar 

  • Connor, A.M., Buchan, J., & Petrova, K. (2009). Bridging the research-practice gap in requirements engineering through effective teaching and peer learning. In International conference on information technology: new generations (pp. 678–683).

  • Cruzes, D. S., & Dyba, T. (2011). Research synthesis in software engineering: A tertiary study. Information and Software Technology, 53, 440–455.

    Article  Google Scholar 

  • Deutsch, M. S. (1991). An exploratory analysis relating the software project management process to project success. IEEE Transactions on Engineering Management, 38, 365–375.

    Article  Google Scholar 

  • Eldh, S. (2013). Some researcher considerations when conducting empirical studies in industry. In International workshop on conducting empirical studies in industry (pp. 69–70).

  • Enoiu, E.P. & Čaušević, A. (2014). Enablers and impediments for collaborative research in software testing: An empirical exploration. In Presented at the Proceedings of the international workshop on Long-term industrial collaboration on software engineering, Vasteras, Sweden.

  • Fairhurst, M. (2013). Academia and industry collaboration to drive biometrics boom. Biometric Technology Today, 2013, 10–11.

    Article  Google Scholar 

  • Franch Gutiérrez, J., Ameller, D., Ayala Martínez, C. P., & Cabot Sagrera, J. (2012). Bridging the gap among academics and practitioners in non-functional requirements management: Some reflections and proposals for the future. In M. U. Vannerdat (Ed.), Modelling and quality in requirements engineering (pp. 267–273). Münster, Germany: Monsenstein und Vannerdat.

    Google Scholar 

  • Garousi, V. (2008). Empirical analysis of a genetic algorithm-based stress test technique for distributed real-time systems. In Proceedings of the genetic and evolutionary computation conference, search-based software engineering (SBSE) track (pp. 1743–1750).

  • Garousi, V. (2010a). An open modern software testing laboratory courseware: An experience report. In Proceedings of the 23rd IEEE conference on software engineering education and training (pp. 177–184).

  • Garousi, V. (2010b). A genetic algorithm-based stress test requirements generator tool and its empirical evaluation. IEEE Transactions on Software Engineering Special Issue on Search-Based Software Engineering (SBSE), 36, 778–797.

    Google Scholar 

  • Garousi, V. (2011). Incorporating real-world industrial testing projects in software testing courses: Opportunities, challenges, and lessons learned. In Proceedings of the IEEE conference on software engineering education and training (CSEE&T) (pp. 396–400).

  • Garousi, V. (2012). Success stories in industrial software testing research and education: A Canadian perspective. In Marmara Research Center, Scientific and Technological Research Council of Turkey (BILGEM, TÜBITAK), Istanbul, Turkey.

  • Garousi, V., Briand, L., & Labiche, Y. (2008). Traffic-aware stress testing of distributed real-time systems Based on UML models using genetic algorithms. Elsevier Journal of Systems and Software, Special Issue on Model-Based Software Testing, 81, 161–185.

    Google Scholar 

  • Garousi, V., Coşkunçay, A., Can, A.B., & Demirörs, O. (2013). A survey of software testing practices in Turkey. In Proceedings of the Turkish national software engineering symposium “Ulusal Yazılım Mühendisliği Sempozyumu” (UYMS).

  • Garousi, V., Coşkunçay, A., Can, A. B., & Demirörs, O. (2015a). A survey of software engineering practices in Turkey. Journal of Systems and Software, 108, 148–177.

    Article  Google Scholar 

  • Garousi, V., Petersen, K., Özkan, B. (2015b). Industry–Academia collaborations in software engineering: A systematic literature review. In Technical report, Hacettepe University software engineering research group, HUSE-2015-01. https://drive.google.com/open?id=0B6dKdxaNjBENSWRwRlNJbExYUWc.

  • Garousi, V., Petersen, K., & Özkan, B. (2015c). Challenges, success patterns and anti-patterns in industry–academia collaborations in software engineering: A systematic literature review. In Technical Report, Hacettepe University Software Engineering Research Group, HUSE-2015-01.

  • Garousi, V., & Pfahl, D. (2015). When to automate software testing? A decision-support approach based on process simulation. In Press, Wiley Journal of Software: Evolution and Process.

  • Garousi, V., & Varma, T. (2010). A replicated survey of software testing practices in the Canadian Province of Alberta: What has changed from 2004 to 2009? Journal of Systems and Software, 83, 2251–2262.

    Article  Google Scholar 

  • Garousi, V., & Zhi, J. (2013). A survey of software testing practices in Canada. Journal of Systems and Software, 86, 1354–1376.

    Article  Google Scholar 

  • Glass, R.L. (2006). Software creativity 2.0: Developer.* Books.

  • Glass, R.L., & Hunt, A. (2006). Software conflict 2.0: The art and science of software engineering: Developer.* Books.

  • Gorschek, T., Wohlin, C., Carre, P., & Larsson, S. (2006). A model for technology transfer in practice. IEEE Software, 23, 88–95.

    Article  Google Scholar 

  • Grunbacher, P. & Rabiser, R. (2013). Success factors for empirical studies in industry-academia collaboration: A reflection. In International workshop on conducting empirical studies in industry (pp. 27–32).

  • Guruprasad, K. (2013). Multi-faceted industry–academia collaboration. In Evolving corporate education strategies for developing countries: The Role of Universities (pp. 90–102).

  • Harrigan, M. T. (2003). Industrial biotechnology education: A model of collaboration between industry and academia. Biochemistry and Molecular Biology Education, 31, 142–144.

    Article  Google Scholar 

  • Jain, S., Ali Babar, M., & Fernandez, J. (2013). Conducting empirical studies in industry: Balancing rigor and relevance. In International workshop on conducting empirical studies in industry (pp. 9–14).

  • Jolly, S.A., Garousi, V., & Eskandar, M.M. (2012). Automated unit testing of a SCADA control software: An industrial case study based on action research. In IEEE international conference on software testing, verification and validation (ICST) (pp. 400–409).

  • Jugdev, K., & Müller, R. (2005). A retrospective look at our evolving understanding of project success. Project Management Journal, 36, 19–31.

    Google Scholar 

  • Kaindl, H., Brinkkemper, S., Bubenko, J. A, Jr, Farbey, B., Greenspan, S. J., Heitmeyer*, C. L., et al. (2002). Requirements engineering and technology transfer: Obstacles, incentives and improvement agenda. Requirements Engineering, 7, 113–123.

    Article  Google Scholar 

  • Kanso, A., & Monette, D. (2014). Foundations for long-term collaborative research. In Presented at the proceedings of the international workshop on Long-term industrial collaboration on software engineering, Vasteras, Sweden.

  • Kato, Y. (2005). Industry–academia collaboration program for commercialization of new dry cleaning system. In AIST Today (International Edition) (p. 11).

  • Krishnan, P., Ross, K., & Pari-Salas, P. (2009). Industry academia collaboration: An experience report at a small University. In Proceedings of conference on software engineering education and training (pp. 117–121).

  • Lamprecht, S.J., & Van Rooyen, G.J. (2012). Models for technology research collaboration between industry and academia in South Africa. In Proceedings of the software engineering colloquium (pp. 11–17).

  • Leszak, M., Perry, D. E., & Stoll, D. (2002). Classification and evaluation of defects in a project retrospective. Journal of Systems and Software, 61, 173–187.

    Article  Google Scholar 

  • Lunsford, T. R., & Lunsford, B. R. (1995). The research sample, part I: Sampling. Journal of Prosthetics and Orthotics, 7, 105–112.

    Google Scholar 

  • Martínez-Fernández, S. & Marques, H.M. (2014). Practical experiences in designing and conducting empirical studies in industry–academia collaboration. In Presented at the proceedings of the international workshop on conducting empirical studies in industry, Hyderabad, India.

  • Maxwell, J. C., & Dornan, J. (2013). How to influence people: Make a difference in your world. Nashville: Thomas Nelson Inc.

    Google Scholar 

  • Misirli, A.T., Erdogmus, H., Juristo, N. & Dieste, O. (2014). Topic selection in industry experiments. In Presented at the proceedings of the international workshop on conducting empirical studies in industry, Hyderabad, India.

  • Morris, P., Masera, M., & Wilikens, M. (1998). Requirements engineering and industrial uptake. In Proceedings of international conference on requirements engineering (pp. 130–137).

  • Mujumdar, A. S. (2010). Editorial on industry–academia collaboration in R&D. Drying Technology, 28, 431–432.

    Article  Google Scholar 

  • Mullins, C. B. (1985). A university administrator’s view of academia–industry collaboration. Circulation, 72, I8–12.

    Article  Google Scholar 

  • O’Leary, C., Lawless, D., Gordon, D., Li, H., & Bechkoum, K. (2006). Developing a software engineering curriculum for the emerging software industry in China. In 19th Conference on software engineering education and training, 2006. Proceedings. (pp. 115–122).

  • Osterweil, L. J., Ghezzi, C., Kramer, J., & Wolf, A. L. (2008). Determining the impact of software engineering research on practice. IEEE Computer, 41, 39–49.

    Article  Google Scholar 

  • Palkar, S. (2013). Industry–academia collaboration, expectations, and experiences. ACM Inroads, 4, 56–58.

    Article  Google Scholar 

  • Perkmann, M., Tartari, V., McKelvey, M., Autio, E., Broström, A., D’Este, P., et al. (2013). Academic engagement and commercialisation: A review of the literature on university–industry relations. Research Policy, 42, 423–442.

    Article  Google Scholar 

  • Petersen, K., & Engstre, E. (2014). Finding relevant research solutions for practical problems: The serp taxonomy architecture. In presented at the Proceedings of the international workshop on Long-term industrial collaboration on software engineering, Vasteras, Sweden.

  • Petersen, K., Gencel, C., Asghari, N., Baca, D., & Betz, S. (2014). Action research as a model for industry–academia collaboration in the software engineering context. In Proceedings of the ACM international workshop on long-term industrial collaboration on software engineering (pp. 55–62).

  • Pfleeger, S. L. (1999). Understanding and improving technology transfer in software engineering. Journal of Systems and Software, 47, 111–124.

    Article  Google Scholar 

  • Pinheiro, C., Garousi, V., Maurer, F., & Sillito, J. (2010). Introducing automated environment configuration testing in an industrial setting. In Proceedings of the international conference on software engineering and knowledge engineering, workshop on software test automation, practice, and standardization (pp. 186–191).

  • Price, F. D. (1985). Industry and academia in collaboration: the Pfizer experience. Circulation, 72, I13–I17.

    Google Scholar 

  • Raschke, W., Zilli, M., Loinig, J., Weiss, R., Steger, C., & Kreiner, C. (2014). Embedding research in the industrial field: A case of a transition to a software product line. In Presented at the Proceedings of the international workshop on Long-term industrial collaboration on software engineering, Vasteras, Sweden.

  • Reel, J. S. (1999). Critical success factors in software projects. IEEE Software, 16, 18–23.

    Article  Google Scholar 

  • Rombach, D., & Achatz, R. (2007). Research collaborations between academia and industry. In Symposium on future of software engineering (pp. 29–36).

  • Rombach, D., Ciolkowski, M., Jeffery, R., Laitenberger, O., McGarry, F., & Shull, F. (2008). Impact of research on practice in the field of inspections, reviews and walkthroughs: Learning from successful industrial uses. ACM SIGSOFT Software Engineering Notes, 33, 26–35.

    Article  Google Scholar 

  • Runeson, P. (2012). It takes two to tango: An experience report on industry–academia collaboration. In Proceedings of international conference on software testing, verification and validation, (pp. 872–877).

  • Runeson, P., Host, M., Rainer, A., & Regnell, B. (2012). Case study research in software engineering: Guidelines and examples. New York: Wiley.

    Book  Google Scholar 

  • Runeson, P., & Minör, S. (2014). The 4 + 1 view model of industry-academia collaboration. In Proceedings of the ACM international workshop on long-term industrial collaboration on software engineering, co-located with ASE 2014 (pp. 21–24).

  • Runeson, P., Minör, S., & Svenér, J. (2014). Get the cogs in synch-time horizon aspects of industry–academia collaboration. In Proceedings of the ACM international workshop on long-term industrial collaboration on software engineering, co-located with ASE 2014 (pp. 25–28).

  • Sahaf, Z., Garousi, V., Pfahl, D., Irving, R., & Amannejad, Y. (2014). When to automate software testing? Decision support based on system dynamics—An industrial case study. In Proceedings of international conference on software and systems process (pp. 149–158).

  • Sandberg, A., Pareto, L., & Arts, T. (2011). Agile collaborative research: Action principles for industry–academia collaboration. IEEE Software, 28, 74–83.

    Article  Google Scholar 

  • Solingen, R. V., & Berghout, E. (1999). The Goal/Question/Metric Method: A practical guide for quality improvement of software development. New York: McGraw-Hill.

    Google Scholar 

  • Subrahmanyam, G.V.B. (2009). A dynamic framework for software engineering education curriculum to reduce the gap between the software organizations and software educational institutions. In CSEET ‘09. 22nd conference on software engineering education and training, 2009 (pp. 248–254).

  • Sullivan, G. M., & Artino, A. R. (2013). Analyzing and interpreting data from likert-type scales. Journal of Graduate Medical Education, 5, 541–542.

    Article  Google Scholar 

  • Thomas, C. J., & McKew, J. C. (2014). Playing well with others! Initiating and sustaining successful collaborations between industry, academia and government. Current Topics in Medicinal Chemistry, 14, 291–293.

    Article  Google Scholar 

  • Wiederseiner, C., Garousi, V., & Smith, M. (2011). Tool support for automated traceability analysis in embedded software systems. In Proceedings of IEEE international conference on embedded software and system (ICESS) (pp. 1109–1117).

  • Wiederseiner, C., Jolly, S.A., & Garousi, V. (2012). AutoBBUT—Automated code generation for black-box unit testing. http://code.google.com/p/autobbut/. Last accessed Sept. 2012.

  • Wiederseiner, C., Jolly, S.A., Garousi, V., & Eskandar, M.M. (2010). An open-source tool for automated generation of black-box xunit test code and its industrial evaluation. In Proceedings of the international conference on testing: academic and industrial conference—Practice and research techniques (TAIC PART) (pp. 118–128).

  • Wohlin, C. (2013a). Empirical software engineering research with industry: Top 10 challenges. In International workshop on conducting empirical studies in industry (pp. 43–46).

  • Wohlin, C. (2013b). Software Engineering Research under the Lamppost. In Proceedings of the international joint conference on software technologies.

  • Wohlin, C., Aurum, A., Angelis, L., Phillips, L., Dittrich, Y., Gorschek, T., et al. (2012). The success factors powering industry–academia collaboration. IEEE Software, 29, 67–73.

    Article  Google Scholar 

  • Wohlin, C., & Regnell, B. (1999). Strategies for industrial relevance in software engineering education. Journal of Systems and Software, 49, 125–134.

    Article  Google Scholar 

  • Wohlin, C., Runeson, P., Höst, M., Ohlsson, M. C., Regnell, B., & Wesslén, A. (2000). Experimentation in software engineering: An introduction. Berlin: Kluwer.

    Book  MATH  Google Scholar 

  • Yeo, K. T. (2002). Critical failure factors in information system projects. International Journal of Project Management, 20, 241–246.

    Article  Google Scholar 

Download references

Acknowledgments

Vahid Garousi was partially supported by several internal grants by the Hacettepe University and the Scientific and Technological Research Council of Turkey (TÜBİTAK).

Author information

Authors and Affiliations

  1. Software Engineering Research Group, Department of Computer Engineering, Hacettepe University, Ankara, Turkey

    Vahid Garousi

  2. Maral Software Engineering Consulting Corporation, Calgary, Canada

    Vahid Garousi

  3. MR Control Systems International Inc., Calgary, Canada

    Matt M. Eskandar

  4. Quality, Test and Process Management Directorate, HAVELSAN A.Ş., Ankara, Turkey

    Kadir Herkiloğlu

Authors
  1. Vahid Garousi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Matt M. Eskandar
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Kadir Herkiloğlu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Vahid Garousi.

Appendix: Excerpts from the SLR study

Appendix: Excerpts from the SLR study

1.1 Challenges impeding success

We show in Table 12 the categories and the detailed list of all challenges. In the following we provide an overview of the findings for the categories.

Table 12 List of all challenges [from the SLR study (Garousi et al. 2015)]
Full size table

1.2 Best practices (success patterns): what to do to ensure success

See Table 13.

Table 13 List of all best practices (success patterns) [from the SLR study (Garousi et al. 2015a, b)]
Full size table

1.3 Anti-patterns: what not to do to ensure success

See Table 14.

Table 14 List of all anti-patterns [from the SLR study (Garousi et al. 2015a, b)]
Full size table

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Garousi, V., Eskandar, M.M. & Herkiloğlu, K. Industry–academia collaborations in software testing: experience and success stories from Canada and Turkey. Software Qual J 25, 1091–1143 (2017). https://doi.org/10.1007/s11219-016-9319-5

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11219-016-9319-5

Keywords

Search

Navigation