Characterizing industry-academia collaborations in software engineering: evidence from 101 projects

Research collaboration between industry and academia supports improvement and innovation in industry and helps ensure the industrial relevance of academic research. However, many researchers and practitioners in the community believe that the level of joint industry-academia collaboration (IAC) projects in Software Engineering (SE) research is relatively low, creating a barrier between research and practice. The goal of the empirical study reported in this paper is to explore and characterize the state of IAC with respect to industrial needs, developed solutions, impacts of the projects and also a set of challenges, patterns and anti-patterns identified by a recent Systematic Literature Review (SLR) study. To address the above goal, we conducted an opinion survey among researchers and practitioners with respect to their experience in IAC. Our dataset includes 101 data points from IAC projects conducted in 21 different countries. Our findings include: (1) the most popular topics of the IAC projects, in the dataset, are: software testing, quality, process, and project managements; (2) over 90% of IAC projects result in at least one publication; (3) almost 50% of IACs are initiated by industry, busting the myth that industry tends to avoid IACs; and (4) 61% of the IAC projects report having a positive impact on their industrial context, while 31% report no noticeable impacts or were “not sure”. To improve this situation, we present evidence-based recommendations to increase the success of IAC projects, such as the importance of testing pilot solutions before using them in industry. This study aims to contribute to the body of evidence in the area of IAC, and benefit researchers and practitioners. Using the data and evidence presented in this paper, they can conduct more successful IAC projects in SE by being aware of the challenges and how to overcome them, by applying best practices (patterns), and by preventing anti-patterns.


Introduction
"When companies and universities work in tandem to push the frontiers of knowledge, they become a powerful engine for innovation and economic growth" (Edmondson et al. 2012).
The global software industry and the academic world of Software Engineering (SE) are both large communities. However, unfortunately, a small ratio of SE practitioners and researchers collaborate with members of the other community, and the reality is that these two communities are largely disjoint (Glass 2006;Garousi et al. 2016a;Briand et al. 2017). For example, at an academic (industrial) SE conference, only a handful of practitioners (researchers) are usually present (if any), and vice versa. This is not a new problem. Since the inception of SE in the late 1960's, both communities have generally done little to bridge the "chasm" between them (Glass 2006), and the ratio of collaborative projects is thus relatively small compared to the number of research projects in the research community and SE activities in the industry. Various reasons have been suggested to explain the low number of industry-academia collaborations (IAC), e.g., difference of objectives between the two communities, industrial problems lacking scientific novelty or challenges, and low applicability or lacking scalability of the solutions developed in academia (Garousi et al. 2016a;Briand 2012). Yet, for the SE research community to have a meaningful future, there is a critical need to better connect industry and academia.
As we, members of the SE community, pass and celebrate the "50 years of SE" (as of this writing in 2018) (Ebert 2018;Broy 2018), many members of the SE community highlight the need for (more) industry-academia collaborations in SE (Carver and Prikladnicki 2018;Basili et al. 2018).
This need comes as no surprise to the SE community, because, being an applied discipline, it has long seen industrial relevance and impact of research activities to be of outmost importance. An indicator for this importance to the SE research community is the ACM SIGSOFT Impact project (n.d.; Osterweil et al. 2008), which was conducted in the years from 2002 to 2008. This project measured and analyzed the impact of SE research on practice. To stress the importance of IAC and to discuss success stories on how to "bridge the gap", various workshops and panels are regularly organized within international research conferences. An example is the panel "What industry wants from research" conducted at the International Conference on Software Engineering (ICSE) 2011, in which interesting ideas from companies such as Toshiba, Google and IBM, were presented. Another international workshop on the topic of long-term industrial collaborations on SE (called WISE) was organized in 2014, which hosted several noteworthy talks. In 2016, a conference panel was held on "the state of software engineering research" (Storey et al. 2016), in which several panelists discussed the need for more IAC in SE. Similar activities have been continuing up to the present day.
While the disconnect between research and practice perhaps hurts practitioners less than researchers, they too have recognized this missed opportunity. The classic book "Software Creativity 2.0" (Glass 2006) dedicated two chapters to "theory versus practice" and "industry versus academe" and presented several examples (which the author believes are "disturbing") on the mismatch of theory and practice (referring to academia and industry, respectively). An interesting blog called "It will never work in theory" (www.neverworkintheory.org) summarized the status-quo on the issue of the IAC as follows: "Sadly, most people in industry still don't know what researchers have found out, or even what kinds of questions they could answer. One reason is their belief that software engineering research is so divorced from realworld problems that it has nothing of value to offer them". The blog further stated that: "Instead of just inventing new tools or processes, describing their application to toy problems in academic journals, and then wondering why practitioners ignored them, a growing number of software development researchers have been looking to real life for both questions and answers".
Another recent trend among practitioners, perhaps indicating their willingness to leverage high-quality research, is the creation of reading groups specifically designed to read, discuss, and present academic papers that could impact their work. This movement, broadly known as "Papers we love" (www.paperswelove.org), has groups in over forty major cities. However, after reviewing the papers read and presented in the above community, at least as of this writing, we found that almost all papers are on theoretical computer sciences topics (such as databases and algorithms) and we did not find any papers on SE being the subject of presentation/discussions among that community.
In summary, we observe that, while perhaps our communities' history of collaboration has been weak, the enthusiasm on both sides makes this an ideal time to systematize and increase our efforts. Towards this end, the challenges, patterns (i.e., the best practices that promise success), and anti-patterns (what not to do) in IAC projects were recently synthesized in a Systematic Literature Review (SLR) (Garousi et al. 2016a). Taking those results as an input, the goal of the study reported in this article is to characterize IAC projects with respect to the challenges, patterns, and anti-patterns identified by the SLR. To address this goal, we conducted a worldwide opinion survey to gather the data from researchers and practitioners. In summary, this article makes the following contributions: & A comprehensive IAC-focused empirical study based on evidence and quantitative assessments of challenges, patterns, and anti-patterns (Garousi et al. 2016a) & A quantitative ranking of the challenges, patterns, and anti-patterns in a large set of IAC projects internationally (across 101 projects and in 21 countries) & A set of evidence-based recommendations to ensure success and to prevent problems in

IAC projects
The rest of this article is structured as follows. In Section 2, we present a review of the related work. In Section 3, we describe the context of our study and review existing process models for IACs in SE. In Section 4, we introduce the study goal, research questions and research methodology. In Section 5, we discuss demographics of our study's dataset. In Section 6, we present the answers to our study's RQs. Finally, in Section 7, we draw conclusions and suggest areas for further research.

Background and Related Work
In this section, we first provide an overview of the related work. Afterwards, to establish a theoretical foundation for our work, we review the existing theories and models of IACs.

Related work
A recent SLR (Garousi et al. 2016a) synthesized the body of literature on the subject of IAC projects in SE by reviewing a set of 33 papers in this area, e.g., (Petersen et al. 2014a;Sandberg et al. 2011;Lamprecht and Van Rooyen 2012). The SLR derived a list of 64 challenges, 128 patterns, and 36 anti-patterns for IAC projects. Among the 33 papers reviewed in (Garousi et al. 2016a), 17 studies reported the number of projects that the observations were based on. There were on average 4.8 projects reported in each paper (the range was from 1 to 22 projects). While the SLR shared insightful experience and evidence on the topic, we believe that the SE community still lacks the following two types of empirical evidence: (1) most of the experience is reported by focused (single) teams of researchers and practitioners and there is a need for evidence based on a larger, more distributed set of IAC projects to reduce the sampling bias; (2) challenges, success patterns, and anti-patterns in IAC projects have been reported rather sparsely and sporadically and there is a need for more systematic synthesis.
Aside from the SLR, while many other studies, e.g., (Petersen et al. 2014a;Sandberg et al. 2011; Lamprecht and Van Rooyen 2012), discuss challenges and success patterns in IAC projects, they report results from one or a few projects in local contexts. The current article aims to provide a larger-scale snapshot on the state of IAC projects, sampled from several countries.
In another survey, Wohlin et al. (2012) investigated the success factors for IAC in SE. Overall, 48 researchers and 41 practitioners from Sweden and Australia participated in the survey. The most important lessons from the study are that (1) buy-in and support from company management is crucial, (2) there must be a champion on the industrial side (company), i.e., someone who is passionate about the IAC and is driving it, and not only a person who merely has been "assigned" the responsibility to coordinate with the research partner, (3) different categories of people have different views on the purpose and goals of the IAC, and (4) social skills are important, particularly if a long-term collaboration shall be established. Different from Wohlin et al.'s survey (Wohlin et al. 2012), the units of analysis in our dataset are research projects, and not individuals. Furthermore, our study is not limited to success factors but, in addition, investigates challenges, success patterns, and anti-patterns.
Other empirical studies on IAC have been reported in other fields such as management (Barnes et al. 2002;Barbolla and Corredera 2009). For example, the study presented in (Barbolla and Corredera 2009) assesses the most influential factors for success or failure in research projects between university and industry. The study is based on interviews with 30 university researchers. It concludes that the company's real interest and involvement during an IAC project, its capacity to assimilate new knowledge, and a confident attitude towards the participating university researchers are the crucial factors for assuring a successful collaboration.
Another study published in 2017 (Ivanov et al. 2017) is a paper entitled: "What do software engineers care about? Gaps between research and practice". The authors surveyed software engineers with regard to what they care about when developing software. They then compared their survey results with the research topics of the papers recently published in the ICSE/FSE conference series. The authors found several discrepancies. For example, while software engineers care more about software development productivity than software quality, papers on research areas closely related to software productivitysuch as software development process management and software development techniqueshave been significantly less often published than papers on software verification and validation, which account for more than half of publications. The study also found that software engineers are in great need for techniques for accurate effort estimation, and they are not necessarily knowledgeable about techniques they can use to meet their needs.
One of the research questions (RQs) in this article (see Section 4.1) assesses the industrial impacts of the surveyed IAC projects. Previous efforts to this issue have been reported, e.g., the ACM SIGSOFT Impact project (n.d.;Osterweil et al. 2008), which, according to its website (ACM SIGSOFT, "SIGSOFT Impact project n.d.), was active in the period of [2002][2003][2004][2005][2006][2007][2008]. Several papers were authored in the context of the Impact project which synthesized and reported the impact of research on practice, e.g., one in the area of software inspections, reviews and walkthroughs (Rombach et al. 2008), and another about the impact of research on middleware technology (Emmerich et al. 2007).
This article is a follow-up to a recent conference paper (Garousi et al. 2017a) and extends it substantially in the following ways: (1) our previous study was based on data from only 47 projects while, based on a follow-up survey, this article is based on a larger dataset (101 projects); and (2) only a few aspects of data and demographics were previously reported, while more detail is reported in this article.
The current work also builds upon another paper co-authored by the first author and his colleagues (Garousi et al. 2016b) in which a pool of ten IAC projects conducted on software testing in two countries (Canada and Turkey) were analyzed with respect to challenges, patterns, and anti-patterns. A set of empirical findings and evidence-based recommendations have been presented in (Garousi et al. 2016b). For example, the paper reports that even if an IAC project may seem to possess all the major conditions to be successful, one single challenge (e.g., confidentiality disagreements) can lead to its failure. As a result, the study recommended that both academics and practitioners should consider all the challenges early on and proactively work together to eliminate the risk of encountering a challenge in an IAC project. While there are slight similarities between (Garousi et al. 2016b) and the current article, the set of RQs and the foci of the two publications differ. Paper (Garousi et al. 2016b) was based on ten IAC projects in software testing in two countries, while this paper is based on 101 projects in all areas of SE in 21 countries.

Theories and Models of Industry-Academia Collaborations
There exists a large body of literature about IAC in fields like management science and research policy, e.g., (Vedavyas 2016;Al-Tabbaa and Ankrah 2016;Lin 2017;Huang and Chen 2017), and also in SE (see the survey paper in (Garousi et al. 2016a)). A search for the phrase "(industry AND academia) OR (university AND industry)" in paper titles in the Scopus academic database (www.scopus.com), on March 15, 2018, returned 3371 papers, denoting the scale of attention to this important issue in the scientific community in general. Papers on IAC could be classified into two categories: (1) papers that propose heuristics and evidence to facilitate IAC, e.g., (Al-Tabbaa and Ankrah 2016;Huang and Chen 2017); and (2) papers that propose theories and models for IAC, e.g., (Nagaoka et al. 2014;Shimer and Smith 2000;Carayol 2003;Mindruta 2013;Banal-Estañol et al. 2017;Banal-Estañol et al. 2013;Ankrah and Al-Tabbaa 2015;Simon 2008).
To establish and conduct an effective IAC, the collaboration partners (researchers and practitioners) need to understand the underlying important concepts and theory (how, why, and when) behind the motivations, needs, and factors involved in a typical IAC. In their paper entitled "Where's the theory for software engineering,", Johnson et al. write that "To build something good, you must understand the how, why, and when of building materials and structures" (Johnson et al. 2012). Also, understanding and utilizing theories of IAC provides us with a solid foundation for designing our own research method and opinion survey used in this study (see Section 4). Johnson et al. state that most theories (explanations) in SE have three characteristics (Johnson et al. 2012): (1) they attempt to generalize local observations and data into more abstract and universal knowledge; (2) they typically represent causality (cause and effect); and (3) they typically aim to explain or predict a phenomenon. On a similar note, a highly-cited study in the Information Systems (IS) domain, which assessed the nature of theory in information systems, distinguished several types of theories (Gregor 2006): (1) theory for analyzing, (2) theory for explaining, (3) theory for predicting, and (4) theory for design and action. Thus, having an initial theoretical basis for IAC in SE can help us explain and characterize IAC as a phenomenon, and facilitate analysis of causality (cause and effect), e.g., helping us decide what practices have the potential of yielding more success in IAC.
We provide in the following a review of the existing studies that proposed theories and models for IAC (Nagaoka et al. 2014;Shimer and Smith 2000;Carayol 2003;Mindruta 2013;Banal-Estañol et al. 2017;Banal-Estañol et al. 2013;Ankrah and Al-Tabbaa 2015;Simon 2008). The study reported in (Nagaoka et al. 2014) focused on sources of "seeds" and "needs" in IAC and their matching process. Seeds were defined as "the technology which served as the base for cooperative research" and needs were defined as "specific use envisaged for the output of the joint research" (Nagaoka et al. 2014). The study focused on several research questions including: (1) how important are the seeds and needs for initiating IACs?; and (2) does matching based on efficiency criteria (the research capability of a partner and the good fit between industry and academic) result in a successful IAC? It then argued that there often exist specific seeds and needs motivating a given IAC project and presented a simple analytic model to quantify the output from collaboration between industry and academic partners. The study also used the "assortative matching" theory (Shimer and Smith 2000) to characterize the matching process between partners. Assortative matching is a matching pattern and a form of selection in which partners with similar objectives match with one another more frequently than would be expected under a random matching pattern (Shimer and Smith 2000).
In 2003, a paper published in the Journal on Research Policy proposed a typology of IAC and argued that firms involved in high (low) risk projects are matched with academic teams of high (low) excellence (Carayol 2003). The authors collected a list of 46 IAC projects in Europe and the United States. An outcome of the study was a typology of IAC built on a formal procedure: a multi-correspondence analysis followed by an ascendant hierarchical classification. The typology exhibited five types of collaborations, positioned inside circles on a 2D plane, in which the x-axis is the risk, novelty and basicness of research, and the y-axis corresponds to the research platform (number of partners), which goes from bilateral research to networked research.
A study published in 2012, entitled "Value creation in university-firm research collaborations: a matching approach", explored the partner attributes that drive the matching of academic scientists and firms involved in IAC (Mindruta 2013). The study modeled the formation of IAC as an endogenous selection process driven by synergy between partners' knowledge-creation capabilities and identified ability-based characteristics as a source of complementarity in IAC.
Banal-Estañol et al. developed a theoretical matching model to analyze IACs (Banal-Estañol et al. 2017). The model predicts a positive assortative matching (Shimer and Smith 2000) in terms of both scientific ability and affinity for type of research. The study suggests that "the most able and most applied academics and the most able and most basic firms shall collaborate rather than stay independent". Before deciding whether to collaborate, academics and firms weigh the benefits in terms of complementarities and the costs in terms of divergent interests. Recent evidence stresses the importance of the characteristics of the matched partners in assessing collaboration outcomes. Banal-Estañol et al. showed in (Banal-Estañol et al. 2013) that the research projects in collaboration with firms produce more scientific output than those without them, if and only if the firms in the project are research-intensive.
The theoretical model developed in (Banal-Estañol et al. 2017) considers and analytically models all the important factors in IAC, e.g., investment (time and money) levels and outcome of projects, which were modeled as follows. When an academic or firm runs a project on their own, the number of positive results (or the probability of obtaining a positive result) depends on its own ability and investment. It was modeled by T A I A and T F I F , where T A (resp. T F ) represents the academic's (resp. firm's) technical ability, or efficiency, and I A (resp. I F ) represents the academic's (resp. firm's) investment level. The parameter T A measures the technical and scientific level of a given academic, her publications, the patents and know-how she owns, the quality of the research group (lab) she works in, etc., whereas the parameter T F measures the scientific level of a given firm, its absorptive capacity, the level of its human capital, etc. The theoretical model was then applied to a set of 5855 projects in a project database of the UK's Engineering and Physical Sciences Research Council (EPSRC) and the predictions provided by the model received "strong support" by the teams of involved academics and firms (Banal-Estañol et al. 2017).
In management science, a SLR on the topic of IAC was published in 2015 (Ankrah and Al-Tabbaa 2015). The SLR reviewed a pool of 109 primary studies and investigated the following RQs: (1) What are the organizational forms of IACs?; (2) What are the motivations for IACs?; (3) How are IACs formed and operationalized?; (4) What are the factors that facilitate or inhibit the operation of IACs?; and (5) What are the outcomes of IACs? The SLR identified five key aspects that underpin the theory of IAC: necessity, reciprocity, efficiency, stability, and legitimacy. The SLR showed that, in the IAC literature, researchers emphasize the role of interdependency and interaction theories in the genesis, development and maintenance of IAC. Interdependency theories stress the impact of the external environment on the formation of IAC, while interaction theories explore the internal development and maintenance of IAC. Furthermore, the SLR (Ankrah and Al-Tabbaa 2015) argued that various perspectives and theories have been widely used in IAC, including transaction costs economics, resource dependency, strategic choice, stakeholder theory, organizational learning, and institutional theory. Transaction Cost Economics (TCE) assumes that transaction (or economic exchange) is the basic unit of analysis for an organization's economic relationships, where these relationships are sought to reduce production cost and increase efficiency. Therefore, it may provide an explanation why universities and companies are inclined to engage in IAC, i.e., minimize the sum of their technology development costs. Finally, by synthesizing the pool of 109 primary studies, the SLR (Ankrah and Al-Tabbaa 2015) presented a conceptual process framework for IAC, as shown in Fig. 1.
Another study, published in the European Journal of Innovation Management, proposed a process model for IAC which "can be utilized by practitioners from both academia and industry in order to improve the process of research collaboration and facilitate more effective transfer of knowledge" (Simon 2008). This study highlighted the social interactions assuming that "social capital can be regarded as an important factor when developing collaborations". The process model, as proposed in (Simon 2008), is shown in Fig. 2. This process model resembles the process framework presented in (Ankrah and Al-Tabbaa 2015) (Fig. 1) in terms of the process flow, with the exception that the former has an extra phase called "Terrain mapping" in the beginning. As discussed in (Simon 2008), mapping of IAC terrain is the initial process stage where industry and market analysis is undertaken in order to develop a detailed understanding of the "collaboration opportunity landscape". This analysis should initially be broad-based, but as requirements are understood in more detail this should lead to more focused activities. If possible, this information gathering exercise should be extended to include industry's current needs that could be gained through person-to-person interactions and networking. Some of the authors of the current paper have experience with terrain mapping activities, e.g., in a set of 15+ software-testing projects in Canada and Turkey (Garousi et al. 2016b), and experience with selecting the "right" topics for an IAC (Garousi and Herkiloğlu 2016).
Furthermore, compared to the model in Fig. 1, the model in Fig. 2 has four additional components: (1) social capital, (2) technical mission, (3) business mission, and (4) collaboration agent. In this context, social capital corresponds to the networks of relationships among participants in an IAC who collaborate and enable the IAC to execute effectively. It includes factors such as familiarity, trust, a common understanding, and a long-term commitment to collaboration. Technical mission and business mission are quite self-explanatory, i.e., an IAC should create "value" both in terms of technical and business missions. Collaboration agent is a role or individual who personally drives forward the collaboration and is responsible for achieving the required objectives in order to initiate and deliver the collaboration. In the recent SLR on IAC in SE (Garousi et al. 2016a), the term "champion" was used as a synonym for the term "collaboration agent". Albeit the slight difference in the terminology used, there are other semantic similarities between the two models in Figs. 1 and 2, e.g., the process flow are almost the same as an IAC usually starts from "proposition "or "formation" phase. In this stage, parties aligned the university's research offering to the company's strategy and needs and specifically to the technology development plans for the relevant products and services that are delivered by the company (Simon 2008). The IAC then continues to the next phases and finished in the "evaluation" or "outcomes" phase, in which benefits of IAC are actually implemented and, usually, a formal or informal post-project evaluation is conducted by both sides. "Motivations" are the need drivers of an IAC in Fig. 1, while in Fig. 2, the terms "technical" and "business missions" are used to refer to the same concept.
Another interesting model to assess research "closeness" of industry and academia was proposed by Wohlin in (Wohlin 2013a). In a talk entitled "Software engineering research under the lamppost "(Wohlin 2013a), Wohlin presented, as shown in Fig. 3, five levels of closeness between industry and academia, which could be seen as a maturity model. IAC projects usually take place in Level 5 (One team) and sometimes in Level 4 (Offline).
In Level 5, the IAC is indeed done in "one team", a specific industrial challenge is identified, draft solutions are evaluated and validated in iterations and final solutions are usually implemented (adopted) in practice. In Level 4, the IAC is offline and often "remote". As in Level 5, a specific industrial problem is identified but the solution is done offline, or rather remotely, in academia. Once ready, a "pre-packaged" solution is offered that is challenging to implement (adopt) in industry due to its generality.
In Level 1 (Not in touch), Level 2 (Hearsay), and Level 3 (Sales pitch), the linkage between industry and academia is non-existent or weak, and thus one cannot refer to the linkage as a proper IAC.

Initial Context and Process Models for Industry-Academia Collaborations in SE
To put our study in context, we present a domain model (context diagram) and provide definitions for the terms used in this context. Figure 4 depicts a UML class diagram representing a typical domain model (context diagram) for IAC projects. Note that, for brevity, this diagram does not include the cardinality details. Researchers and practitioners participate in a given IAC project. Either of them or both could act as the initiator. There is usually a need that drives the project offering one or more solutions with impact. Solutions are, in fact, the contributions of an IAC project to its industrial partner(s). Solutions are expected to have (positive) impact in the industrial context, e.g., an example solution could be a new software refactoring method for a company providing positive impact by saving software maintenance costs. To keep our study focused, we only consider industrial impact and do not consider "academic" impact (Poulding et al. 2015) of an IAC.
There is at least one object of study in the form of a (software) process or a software system. For example, an IAC project may target improving software testing processes of a given company. Papers are usually written as a result of the project. Funding sources may support the project. Partners involved in an IAC project naturally expect the project to be successful. The level of success is assessed by a set of success criteria, which are defined (at least implicitly) by the partners, e.g., publication of papers, training of PhD students and young researchers, getting insights, lessons learned, or new research ideas, and solving the need that triggered the project in the first place.
In terms of conceptual terminology, the scope of a typical IAC project might not be immediately clear. We use in this study the "project" notion in the same way as typically used by funding agencies, e.g., national agencies, such as the NSERC 1 in Canada or TÜBİTAK 2 in Turkey, or international agencies, such as the European Commission's Horizon-2020 program. 3 An IAC project can take various forms, e.g., technology transfer and consultancy, but there should be some sort of research involved in it, to make it within the scope of our definition in this paper. A SE IAC project is a project in which at least one academic partner and at least one industrial partner formally define a SE-related research topic.
The trigger for an IAC project is usually a real industrial "need" (or challenge), e.g., improving test automation practices in a company (Garousi and Herkiloğlu 2016), or is based on academic research, e.g., assessing the benefits of software documentation using UML. As a concrete example may serve one of the authors' action-research IAC (Santos and Travassos 2009;Petersen et al. 2014b;Stringer 2013) conducted with a software company in Turkey. Early in the collaboration process, the partners systematically scoped and defined a set of topics to work on (Garousi and Herkiloğlu 2016), e.g., (1) increase test automation, (2) assess and improve an in-house test automation framework, (3) establish a systematic, effective and efficient GQM-based measurement program for the testing group, and (4) assess and improve test process maturity using TMMi (Garousi et al. 2018a).
The presented overview of existing work about IAC theory (Nagaoka et al. 2014;Shimer and Smith 2000;Carayol 2003;Mindruta 2013;Banal-Estañol et al. 2017;Banal-Estañol et al. 2013;Ankrah and Al-Tabbaa 2015;Simon 2008) enables us to lay a solid foundation for designing our research method (see Section 4). Various models have been presented (e.g., see Figs. 1 and 2) and while there are many similarities across different models, there does not exist one single unified model. We should mention that each model usually takes a certain view (perspective) on the nature of IAC or highlights certain aspects. For example, both (Ankrah and Al-Tabbaa 2015; Simon 2008) took a process view but while the former highlighted the issues of motivations and facilitating/impeding factors, the latter highlighted social capital and collaboration agent.
In our study, we focus on the process aspect and on cause/effect-relationships in IAC within the SE domain. In addition, we incorporate the set of challenges and patterns provided by the IAC SLR published in (Garousi et al. 2016a).
Thus, we consider the models presented in (Ankrah and Al-Tabbaa 2015; Simon 2008) as our baseline and extend/adapt them to fit our purpose, as illustrated in Fig. 5. We synthesized our process model from three sources: (1) the models presented in (Ankrah and Al-Tabbaa 2015;Simon 2008); (2) our experience in IAC, e.g., (Garousi et al. 2016b); and (3) the SLR study published in (Garousi et al. 2016a). In our study, we use this process model to understand and characterize IAC in a way inspired by the authors of (Kemper 2013) who stated that "… a way to evaluate a theory is by its derivations, that is, what does the theory help us to understand?". Note that our model is not a collaboration model (like those discussed in (Petersen et al. 2014a;Sandberg et al. 2011;Gorschek et al. 2006)) but a process model for IAC projects, including important factors of interest to our study (e.g., collaboration need, challenges and patterns). We do not claim this model to be a unified complete model for IAC within the SE domain. We rather see it as an initial step towards such a model corresponding to our needs in this study.
According to the grounded theory technique (Corbin and Strauss 2014), if the dynamic and changing nature of events is to be reflected in a process model, then both structure and process aspects must be considered. Therefore, the model in Fig. 5 is centered in a linear process for collaboration but also supported by the structural elements, i.e., the cross-cutting concerns such as challenges and patterns, need for collaboration, outputs, results and contributions to the literature, and impact on the software project or product under study. The process model has  (Garousi et al. 2016a;Garousi and Herkiloğlu 2016). Throughout the project, partial or full solutions are developed which are expected to address that need (represented by the link between "solution" and "need" in

Research Goal and Method
We discuss in this section the study goal, research questions, study context, and research method.

Goal and Research Questions
Formulated using the Goal, Question, Metric (GQM) approach (Solingen and Berghout 1999), the overall goal of this study is to characterize a set of IAC projects in SE, with respect to the challenges, patterns, and anti-patterns identified by the SLR study (Garousi et al. 2016a). Our study contributes to the body of evidence in the area of IAC, for the benefit of SE researchers and practitioners in conducting successful projects in the future. Based on the overall goal, we raised the following research questions ( Note that, compared to our previous paper (Garousi et al. 2017a), RQ 1 has been added. Both RQ 2 and 3 were partially addressed in (Garousi et al. 2017a) but without in-depth analysis. Also, the analyses in (Garousi et al. 2017a) were based on a smaller dataset compared to the current article. Furthermore, we conduct and report additional analyses in this paper, e.g., an in-depth analysis of the demographics of the dataset, and an in-depth analysis of how the challenges affected the projects. Thus, this article is a substantial extension of (Garousi et al. 2017a). Furthermore, we believe this work makes a novel contribution to the community by studying both the technical SE aspects of a large set of IAC projects via RQ 1 (needs, solutions and impacts), as well as their operational aspects and characteristics via RQs 2 and 3 (challenges, patterns and anti-patterns).

Research Method (Survey Design)
To answer the study's RQs, we designed and conducted an opinion survey. Our goal was to gather as many data points (opinions) from researchers and practitioners world-wide. Table 1 shows the structure of the questionnaire used to conduct the survey. To provide traceability between the questionnaire and the RQs, we also show in Table 1 the RQs addressed by each part of the questionnaire. Furthermore, we designed the survey in a way to fully match the IAC process model in Fig. 5. Due to space constraints, we do not provide the full survey questionnaire, as presented to participants, in this paper, but it can be found as a PDF file in an online source (Garousi et al. 2016c).
In designing the survey, we benefitted from the survey guidelines in SE (Molleri et al. 2016). Some example survey guidelines that we utilized from (Molleri et al. 2016) are as follows: (1) identifying the research objectives, (2) identifying and characterize target audience, (3) designing sampling plan, (4) designing the questionnaire, (5) piloting test questionnaire, (6) distributing questionnaire, and (7) analyzing results and writing the paper. We also used the recommendations from w.r.t characterizing units of observation, units of analysis, establishing the sampling frame and recruitment strategies (Mello et al. 2014a, b).
We were also aware of validity and reliability issues in survey design and execution (Molleri et al. 2016). One aspect of the survey's validity, in this context, is how well the survey instrument (i.e. the questions) measures what it is supposed to be measured (construct validity). External validity of a survey relates to the representativeness of the results for the population from which respondents are sampled. The reliability of a survey refers to the question whether a repeated administration of the questionnaire at different points in time to the same group of people would result in roughly the same distribution of results each time. We dealt with those validity issues in both the survey design and execution phases. It was intended that respondents would respond to the questionnaire with respect to each single IAC project they had participated in. The unit of analysis in our survey is a single IAC project. Therefore, a participant could provide multiple answers; each one for a single project he or she was involved in. The considered IAC projects could be completed, (prematurely) aborted or ongoing (near completion). We included aborted projects in the survey and its dataset so that we could characterize the factors leading to abnormal termination of IAC projects.
Part 1 of the questionnaire has 11 questions about demographics (profile) of the respondent and the IAC project. Part 2 of the questionnaire asked about the needs, developed solutions, and impact of the IAC project. Part 3 asked about the challenges, patterns, and anti-patterns in the project, as adopted from the SLR study published in (Garousi et al. 2016a). For example Q3.1 asked the participants to "rate the extent of the negative effect each of the following challenges had on the industry-academia collaboration in the project" and listed ten categories of challenges (again, adopted from the SLR ( For more details about each of the above patterns, the reader may refer to the SLR (Garousi et al. 2016a). Part 4 of the survey included five questions about outcome, success criteria and success levels of the projects. To keep the current paper focused, we are not including any data nor raise any RQs about those aspects, and plan to analyze those parts of our dataset in future papers.

Validation of the Survey Design
As mentioned above, construct validity was an important issue and we ensured that the survey instrument (i.e., the questions) would measure what our study intended to measure. Parts 2 and 3 of the survey were explicitly designed to ensure a direct mapping with the study goal and its associated RQs (Section 4.1). We designed questions in each survey section (part) to gather data about the following aspects of a typical IAC project: Part 2 focused on need, developed solutions, and impact of the project. Part 3 focused on challenges, patterns and anti-patterns in the projects.
To ensure construct validity, we conducted two rounds of pilot applications of the questionnaire used in the survey, first among the authors and then, in addition, with five practicing software engineers selected from our industry network. The main issue we considered in the pilot phase was to ensure that the survey questions would be understood by all participants in the same manner.
We were also aware about the importance of reliability/repeatability of the survey instrument. We applied the test-retest reliability check (Kline 2013) for this purpose. We asked two participants (who had provided their emails addresses in the main data collection phase), one practitioner and one researcher, to re-fill the survey. The second time of filling the survey by those two participants was about 1 year after the first time (Fall 2018 versus Fall 2017, see the next section). For measuring reliability for two tests, we calculated the Pearson correlation coefficient of the numerical data fields in the survey (e.g., challenge Likert scales), as suggested in the statistics sources (Kline 2013). The correlations for the two participants were 0.85, and 0.72; and the average value was 0.78, which is interpreted as an "acceptable" reliability measure for survey instruments (Kline 2013).

Survey Execution and Data Collection
For data collection, we sent invitations by email to SE researchers and practitioners who were known in the community to be active in IAC projects and to the authors of the primary studies reviewed in the SLR (Garousi et al. 2016a). The survey was anonymous, but the participants could provide their names and emails if they wanted to receive the results of our study. The total number of invitations and the resulting response rates are discussed further below.
Our sampling method was convenience sampling which is the dominant approach chosen in survey and experimental research in SE (Sjoeberg et al. 2005). Albeit its drawbacks and potential risk of bias in the data, this does not mean that convenience sampling is generally inappropriate (Ferber 1977). Convenience sampling is also common in other disciplines such as clinical medicine and social sciences (e.g. (Kemper and Stringfield 2003)).
We are aware of the importance of external validity and reliability of the survey results and instruments, i.e., representativeness of the dataset and appropriateness of the data sampling (Gobo 2004). There have been many discussions about advantages and disadvantages of convenience sampling, e.g., (Gobo 2004;Sackett and Larson 1990). Regarding its limitations, it has been said that "because the participants and/or settings are not drawn at random from the intended target population and universe, respectively, the true representativeness of a convenience sample is always unknown" (Sackett and Larson 1990). At the same time, researchers have recommended two alternative criteria to explore the external validity of convenience samples: sample relevance and sample prototypicality (representativeness) (Sackett and Larson 1990). Sample relevance refers to the degree to which membership in the sample is defined similarly to membership in the population. For instance, an example of sample irrelevance taken from a field outside SE, would be a study of executive decision making conducted with a sample of university student (Sackett and Larson 1990). There exist also studies about this issue in SE, e.g., . Sample prototypicality refers to the degree to which a particular research case is common within a larger research paradigm. An example of prototypicality would be a study exploring the benefits of software design patterns; although a sample of senior executives completing such a survey could be collected, a sample of "technical staff", i.e., software developers, would be more prototypical of when such benefits would actually be observed. With sample representativeness, sample relevance is assumed (Sackett and Larson 1990). In summary, while using convenience sampling in our work (similar to many other survey studies in SE) the representativeness (and thus external validity) of our study results could be limited, we still ensured meeting the other two external validity aspects, i.e., relevance and sample representativeness, since we sent the survey to researchers and practitioners who have been active in IAC projects and have first-hand experience of initiating and conducting IAC projects.
Data collection via the online survey was conducted in two phases. The first phase was conducted in Fall 2016. The second phase was conducted in Fall 2017. In the first phase, we sent invitations to a large number of SE researchers and practitioners (about 100) who were known in the community to be active in IAC projects and the authors of the primary studies reviewed in the SLR (Garousi et al. 2016a). About two-thirds (2/3) of the (100) invitations were sent to researchers, while the rest (1/3) were sent to the practitioners in our network. Unfortunately, we received a response rate from the SE community (only 11 data points). The response rate was 9.1%. Since we (the authors of this study) have also been active in IACs, we also provided data points related to our past/current projects. In total, during the first phase, the authors of this study contributed 36 data points, creating a dataset with a total of 47 data points. We reported an initial analysis based on those 47 data points in a conference paper (Garousi et al. 2017a).
The second phase of data collection was conducted in Fall 2017, in which we sent 150 invitations. Similar to the phase #1, the recipients were again the researchers and practitioners who were known in the community to be active in IAC projects. About 100 of the invitations were sent to the same pool of the recipients, as we had sent in phase #1. We developed an additional set of 50 researchers and practitioners in the phase #2. Similar to the first phase, about two-thirds (2/3) of the 150 invitations in the second phase were sent to researchers, while the rest (1/3) were sent to the practitioners in our network. In the second phase of data collection, we were more proactive in our survey invitation strategy (e.g., we personally emailed and reminded our collaborators to fill out the survey) and the response rate (32.7%) increased compared to the first phase (9.1%). In the expanded dataset, 60 data points were from our invited participants in addition to the 47 data points provided by the author team. Since the study's authors all have been active in IACs throughout their careers, it was natural for us to also include the data points from the author team, since we could get a more enriched dataset. When we entered the data into the questionnaire, we ensured treating it with outmost care and seeing ourselves as independent participants to prevent any bias in data collection. Each co-author contributed between 3 and 19 data points. Details about the composition and evolution of the dataset are shown in Table 2.
To ensure the quality of data, we screened the 107 raw data points. One data point was excluded since one respondent had entered one single data point as a proxy (aggregate) for her/ his IAC projects in all her/his entire career and thus the provided measures were not valid for our survey. We excluded five more data points since the only reported research method was a practitioner-targeted survey, which cannot be considered an actual IAC. Thus, the final dataset contained 101 projects after screening.
As shown in Table 2, in the final dataset, 46 data points were from the study authors, and 55 data points from the community at large (i.e., not from the authors of this study). In the rest of this article, we refer to the projects using the labeling of Pi, with i ranging from 1 to 101. These IDs are indicated in the dataset file.
We also wondered about how many respondents provided the information on the 101 projects. We had some identifying information of the respondents (e.g., emails) and used them to gather this information. In total, 64 respondents provided the information on the 101 projects. Each respondent provided between 1 and 19 data points. A majority of the respondents (57 people) provided only one data point, thus we can say that a large number of data points came from different people.
For transparency and to benefit other researchers, after removing identification and sensitive information about the projects, we have shared the raw dataset of the survey publicly in online sources (phase #1 in (Garousi et al. 2018b), and phase #2 in (Garousi et al. 2018c)). The full survey questionnaire can be found as a PDF file in an online source (Garousi et al. 2016c).

Data Analysis Method
We used both quantitative and qualitative analysis techniques. Many questions in our survey instrument are closed questions. Thus, we could apply simple descriptive statistics and visualization techniques (e.g., bar charts, histograms, boxplots, and individual value plots) to analyze and visualize the data received from the survey participants. Answers to open-ended questions were analyzed using the qualitative coding (synthesis) technique (also called, "open/axial" coding) (Miles et al. 2014). For one of the questions ("needs" addressed in the projects), we also using the word clouds technique to visualized the responses (results in Section 6.1.1).
Qualitative coding of each open-ended question was done by one co-author, and was peer reviewed by one other author at least, to ensure highest quality and to prevent bias. In the case of conflicting opinions by different authors, we had planned to conduct group discussions to reach consensus (but this never happened).
We provide below an example of how we conducted the qualitative analysis by showing how the analysis was done on one of the open-ended about industrial impacts of the projects (Section 6.1.3). Free-text responses for 92 of the 101 data points were provided for that question by the respondents. We used qualitative coding (Miles et al. 2014)  Qualitative coding (synthesis) (Miles et al. 2014) is a useful method for data synthesis and has been recommended in several SE research synthesis guidelines, e.g., (Cruzes and Dybå 2010; Cruzes and Dybå 2011; Cruzesa and Dybåb 2011). Table 3 shows examples of how we conducted qualitative analysis of data received for one survey question on several projects. For example, for project P18, the respondent wrote: "The industry partner did not adopt the approach, to the best of our knowledge" and thus it was easy to classify it under the "No impacts on industry" group.
As we were also interested in the SE topics of the IAC projects in the dataset, another task in our data analysis was to extract the SE topic(s) of each IAC project. We did not have a specific question about this aspect in the survey (Section 4.2) but we were able to derive the SE topics of each IAC project by looking at its need (tackled challenges). To classify the SE topics, we used the latest version (3.0) of the Software Engineering Body of Knowledge (SWEBOK) (Bourque and Fairley 2014). The SWEBOK describes 15 Knowledge Areas (KAs), out of which 12 KAs are focused on technical aspects of SE, e.g., requirements, design, construction, and configuration management. Three other SWEBOK KAs cover mathematical, computer and engineering foundations. For example, two respondents mentioned the following needs for their projects: "supporting change impact analysis" for project P3 in the pool, and "to enable/improve quality assurance and engineering process improvement" for project P4. Based on the data, project P3 was classified under the KA "configuration management" and P4 was classified under the KAs "process" and "quality". Note that each project could be classified under more than one SWEBOK KA.

Demographics of the Dataset
We present the demographics of the dataset in this section.

P18
The approach was evaluated on a case-study industrial-like system in the 'lab'. The industry partner did not adopt the approach, to the best of our knowledge x P19 Based on quantitative measurements, the approach optimized cost of developing and maintaining software documentation.
x P20 Based on quantitative case study, the optimization-based intelligent software system was able to reduce the pumping cost of oil pipelines. % values are in the papers x P21 The performance of the web application improved and there were no problems with high user loads x P23 Many of the challenges were addressed and improving case studies are now underway to quantitatively measure the benefits and highlight the areas for further improvements x P24 No quantitative measurements have been done, only qualitative. They are suggesting the + impact of the project x

Breakdown by Affiliation Types and Countries
One of the questions asks about the respondent's affiliation types (researcher or practitioner). 83 respondents said they are working at universities or research centers, 21 respondents said they had industry affiliations, and three respondents said they had both affiliation types (such as graduate students who also work part-time in companies).
Another question asked about the country (or countries) in which the IAC project was conducted. Figure 6 shows the data. 21 different countries are represented in the dataset. Although the authors' countries of residence influenced the country distribution, the survey sample is diverse in terms of geographical distribution, covering Europe, Asia, North and South America.

Research Methods Used in IAC Projects
To understand the demographics of the IAC projects under study, we asked about the research methods used in each project. We used the classification of empirical SE research methods provided by Easterbrook et al. (Easterbrook et al. 2008) for this purpose, as shown in Fig. 7. Figure 7 shows that the most used research methods are industrial case studies (Runeson and Höst 2009) and "action-research" (Santos and Travassos 2009;Petersen et al. 2014b;Stringer 2013). This observation is as expected since these methods are, perhaps, better suited than other methods for IAC projects. Action research is "research initiated to solve an immediate problem" (Stringer 2013) and thus many industry-focused researchers are increasingly adopting it as the research method of choice, e.g., (Petersen et al. 2014b;Doležel and Buchalcevová 2015;Christensen et al. 2008;Viikki and Palviainen 2011). By being rooted in industrial practice, these research methods typically generate rich contextual information and many qualitative insights (Santos and Travassos 2009). As per our experience, e.g., (Garousi et al. 2016b), industrial case studies usually apply either the "exploratory" or the "improving" type, or both, rather than other case study types (descriptive, explanatory) (Runeson and Höst 2009). This helps, for example, to understand how and why certain practices, techniques, tools and methods actually work in a specific context. In projects using as their research method controlled experiments with practitioners, initiation of the IAC usually starts from the researchers' side (refer to the process model in Fig. 5). In these cases, typically, researchers have a research method or tool that they wish to apply in practice to assess its benefit and usefulness (Runeson and Höst 2009).
& Observation 1: Industrial case studies and action-research are the most popular research methods used in IAC projects.

Number of Resulting Papers from each Project
We asked in another question about papers resulting from each project. Figure 8 shows that more than 90% of the projects have at least one publication. This means that there is a fair chance that others can learn from the project experience and that knowledge about successfulor unsuccessful -IAC projects is being conveyed to a larger community. However, since most publication venues are of academic nature, there is a high risk that the lessons learnt from the reported projects are mainly shared within academia. This is due to the fact that most industrial practitioners do not read scientific journals or conference proceedings on a regular basis (Chakrabarti and Lindemann 2015).
& Observation 2: Over 90% of IAC projects, in our dataset, resulted in at least one publication.

Who initiated the Projects
IAC projects can be initiated by academic partners, industrial partners, or jointly by both sides (Fig. 5). In our dataset, the situation was quite balanced, i.e., the academic partners initiated 24 projects (22.4%) and the industry partners also initiated 24 projects. However, 52 projects (48.6%) were initiated by both sides jointly. No response was provided for the remaining one project. We justified the above breakdown as follows. There can be different reasons for having an IAC project initiated by just one partner. One possibility is that the motivation for starting a collaboration is heavily academic-driven. For example, an academic might approach a company in order to validate a new SE method. On the other hand, the trigger for starting a collaboration might be heavily practice-driven. For example, a company might approach a university or research institute in order to get support for improving its SE practices. Thus, in spite of the myth that industry has in general lower motivation, compared to academia, to start collaborations (Garousi et al. 2016a), we see that many projects were initiated by industry.
Finally, there might exist established industry-academia relationships among a set of collaborators. In this situation, partners who have already a track record of successful collaborations might simply initiate the next IAC project jointly, building upon the trustful established relationship and capitalizing upon results of past IAC projects.
One could do further in-depth (correlation) analysis on collaborator type who initiated the project versus other variables in the dataset, e.g., country, and looking into a question such as: Are there differences between countries in terms of who initiates the IAC projects? However, for space constraint and also since our dataset is not large enough, we do not conduct and report such an analysis in this paper. But as discussed in Section 4.4, the entire raw dataset file (Garousi et al. 2018c) can be used for such extended analyses.
& Observation 3: Industry is as motivated as academia to start IAC projects.

Project Duration
The "individual-value" plot in Fig. 9 shows the durations of IAC projects in our dataset (in months). The mean and median values of the IAC project durations are about around 22 and 15 months, respectively. About 4% of the projects took at most 13 months, whereas the durations of only about 5% of the projects were larger than 48 months. Overall, the durations of the IAC projects, in most cases, range between 4 and 48 months (i.e., 4-6 months: 18%, 7-12 months: 28%, 13-24 months: 20%, 25-36 months: 19%, and 37-48 months: 7%).
& Observation 4: More than 90% of the IAC projects in our dataset lasted from 4 months to 4 years.

Analysis and Results
We analyze the data and answer each of the RQs one by one in the following sub-sections.

RQ 1-Industrial Needs, Solutions Developed by, and Impacts of the IAC Projects
Since RQ 1 spans over three aspects, we review the results related to each of the three aspects separately in the next three sub-sections. Table 4 shows the needs and problems to be solved, the solutions developed, and the achieved impact of a representative subset of projects. Respondents provided qualitative narratives of the needs (problems) addressed in their projects. A number of projects were initiated to improve various aspects, e.g., P1's goal was to improve test models used for model-based testing, P5 aimed at improving requirements specifications, and P13 aimed at improving the tool-support traceability analysis in embedded software. Other projects were set up based on needs to develop approaches or tools for other purposes, e.g., need for P3 was to provide and deploy an approach to support change-impact analysis, and P81 focused on the need to provide a practical way to perform regression test selection.

Needs (Problems) Addressed in the IAC Projects
Of course, we would expect that there would be direct relationships among the needs, solutions, and impacts of a given IAC project. After all, a project is initiated with the goal of providing a solution (or improvement) to a given need and it is expected that the solution will have (positive) impact on the context (e.g., software process or product under analysis). We discuss examples of such expected linkage among the three aspects in the following section.
Given the diversity and wealth of data capturing IAC project needs, we also extracted a high-level picture of the needs by using word-cloud visualization, as shown in Fig. 10. As we can see, "testing", "improvement", "quality", "process", "automation", and "systems" are among the most popular needs addressed in the projects. Let us note that we are not using word-cloud as a "formal" data analysis tool in this paper, but rather as a simple visualization to show a high-level trend of the topics.  "A bundle of concepts and methods for QA and software process and product improvement in multi-disciplinary engineering context for improving defect detection and engineering processes" "Enabling collaboration between disciplines (which was not possible before) and increase data synchronization steps (before manually and if required), afterwards on a weekly basis and automation supported." P5 Improve requirements specifications "A method for causal analysis using defect data and a Bayesian network on common causes for requirements engineering problems" "Reduction of 50% in defect rates.
In the first iteration, reviewers found 1 defect per function point and 2.4 defects per inspection hour. In the third (last) iteration reviewers found 0.5 defects per function point and 2.8 defects per inspection hour." P13 Improve tool-support traceability analysis in embedded software "A traceability analysis tool-set was developed and released to the industrial context" "Based on results from the improving case studies, the traceability analysis toolset was found useful." P14 Manual troubleshooting of environmental configuration issues was tedious and error prone.
"An automated environment configuration testing was developed and released to the industrial context" "The staging environment instability issues were automated detected by the tool and corrected in minutes. The service downtime reduced to 0-10 min per week." P15 Cost of manual testing was too high and too many regression faults were observed "A tool for automated test code generation for black-box unit testing was developed and released to the industrial context" As stated in Section 4.4, the entire raw dataset is available online in a file (Garousi et al. 2018c). The IAC project needs/solutions data can be reviewed in that online file and may be used for further analysis. For brevity, we only present summaries of each analyzed aspect in this article.
& Observation 5: The IAC projects under study have covered a variety of needs / problems and have developed different types of solutions (contributions) with a variety of industrial impacts. Figure 11 shows the distribution of our data points across SE topics. Each data point corresponds to one IAC project. With the exception of the KA "professional practice", all SWEBOK KAs have been modestly covered. Testing and software quality are the most frequently mentioned topics in the dataset. We initially felt that the authors and their research areas had some impact on the dataset in this regard, but then we noticed that the survey participants outside the author team also frequently mentioned projects on testing. Several data points were classified under more than one KA. Figure 12 shows the number of KAs per IAC project in a histogram. More than half of the projects (57 out of 101) focused on a single KA, while the rest focused on more than one KA. This indicates that some IAC projects addressed multiple SE subject areas. For example, the goal of project P64 was to "better understand how people deal with code quality when using Continuous Integration", which made us classify it under three KAs: Configuration management, Construction, and Quality.
One project (P20) focused on seven KAs, since its goal was to develop "intelligent software systems to reduce the pumping cost of oil pipelines", engineering "scientific software" (Segal and Morris 2008;Farhoodi et al. 2013). Project P20 was an IAC project among SE researchers and mechanical/electrical engineers addressing the entire process of developing an optimization software for oil pipelines with focus on requirements, design, development, testing and other KAs of the SWEBOK. Four other projects (P33, P35, P36 and P37) focused on six KAs, each. Thus, we see that a mix of projects with respect to coverage of SE topics is included in our study pool. & Observation 6: Most of the IAC projects seem to focus on exactly one topic of SE, i.e., one SWEBOK KA. Fewer projects considered multiple SE topics, e.g., an IAC project was to develop a large software system applying best practices in seven KAs. Fig. 10 Word-cloud of "needs" (problems) addressed in the projects (created using www.wordle.net)

Contributions of (Solutions Developed by) the IAC Projects
Respondents were asked to provide qualitative narratives of the solutions developed and offered in their projects (see a glimpse of the corresponding data in Table 4). Solutions are in fact the contributions of an IAC project to its industrial partner(s). Analysis of solutions showed that they usually are about development or customization of (new) tools or approaches and are related to analysis, testing, measurement and processes.
For example, in P14, "an automated environment configuration testing was developed and released in the industrial context". In project P15, to address the problem that "cost of manual testing was too high and there were too many regression faults", "a tool for automated test code generation for black-box unit testing was developed and released in the industrial context".
To get a better understanding of the solution/contribution types developed in the IAC projects, we classified the solutions according to the following popular taxonomy of contributions used in SLR studies, e.g., (Zhi et al. 2015;Garousi et al. 2015;Petersen et al. 2015  approach (method, technique), tool, empirical study only, process, model, metric, any other contribution (solution type). Figure 13 shows the resulting classifications. As we can see, approaches and tools are the two most common project contributions (solutions), which appeared in 35 (32.7%), and 32 (29.9%) projects, respectively.
Contributions of 15 projects (14.0% of the dataset) were empirical studies only. In these IAC projects new approaches or tools were not developed but existing ones taken from the literature were applied in industrial settings. For example, in project P12, existing defect taxonomies were adopted from the literature to link requirements to test artifacts and to use those links to improve effectiveness and efficiency of tests as well as the requirements review process. In P24, researchers and practitioners did not develop a novel approach or a tool, but instead conducted an empirical assessment and improvement of the software test processes in the company using existing maturity models, like Test Maturity Model integration (TMMi) (TMMI Foundation 2017) and Test Process Improvement (TPI) (Koomen and Pol 1999).
In 14 IAC projects (13.1%), solutions were process-related. For example, in P4, efforts were spent on software process improvement (SPI) in multi-disciplinary engineering contexts for improving defect detection and engineering processes. P8 introduced risk-based testing into existing test processes. In P29, a company-wide measurement process based on the "Goal, Question, Metric" (GQM) paradigm was developed.
In ten IAC projects (9.3%), solutions were model driven. For example, in P5, a Bayesiannetwork model for identifying common causes of requirements engineering problems was proposed. In P46, a meta-architecture model was developed to improve architectural design practices in a company. Participants of P95 developed a knowledge map model.
In two IAC projects (1.9%), solutions were metric-focused. For example, motivated by the need to identify agile metrics for assessing the performance of IT development teams, project P55 developed a set of metrics to distinguish leading and lagging aspects in agile teams (Cohn 2009). & Observation 7: Most IAC projects in the pool tend to focus on approach or tool contributions.

Industrial Impacts of the IAC Projects
Similar to project needs/contributions, respondents provided qualitative narratives of the industrial impacts of their projects. To give a glimpse into the related qualitative data, Fig. 14 shows a wordcloud of the textual data provided by participants. We can see in the word-cloud that terms like "improved", "effectiveness", "change", and "effort" were mentioned several times. Approach, method, technique Tool Empirical study "only" Process Model Metric # of data points Fig. 13 Classifications of project contributions based on the taxonomy of contribution types used in SLR studies (Zhi et al. 2015;Garousi et al. 2015;Petersen et al. 2015) A closer look at their occurrence in the raw data reveals that many projects improved aspects such as effectiveness and effort. Some participants provided concrete quantitative impact (improvement) measures in their feedback. For example, the participant reporting on P81 mentioned that "Based on measurements, our technique reduced end-to-end testing time for about 65% in industrial setting", and for P14, it was mentioned that "The development environment instability issues were automatically detected by the tool and corrected in minutes. [Thanks to the project solution], the service downtime reduced to 0-10 minutes per week." Furthermore, as discussed in Section 4.5, we used a qualitative coding approach (Miles et al. 2014) to classify industrial impacts of the projects into three categories: (1) Positive impacts on the industry partner, backed by quantitative evidence (measures) in the data point; (2) Positive impacts, backed by qualitative statements; and (3) No impacts on industry (yet), work in the lab only, or "not sure". Figure 15 shows the results according to this classification. For a given project, a respondent could provide quantitative and/or qualitative impacts (none, either or both). Since not all respondents provided this information, the sum of the classified data points only added up to 97, and not to 101 (the total number of data points).
73 IAC projects (~68%) reported at least one type of positive impact (quantitatively or qualitatively measured). 47 IAC projects reported quantitative positive impact. For example, a tool-supported approach for extending/refining test models based on execution traces collected during exploratory test activities was developed in P1 and it was reported that "The number of faults that can be detected by model-based test was increased by at least 30% for the conducted case studies". For P50, the developed approach "improved test [activities] by automating it and reduced it by 10% for a year". 30 IAC projects reported positive qualitative impact. For example, it was reported for P7 that "Based on three case studies from large enterprises, four success factors for introducing risk-based testing, i.e., (1) risk drivers like an inhomogeneous distribution of risks associated to the various parts of the tested software system, (2) an improvement goal with regards to effectiveness or efficiency as well as, (3) a proper risk identification, and (4) risk analysis approach were identified".
For a large percentage of the IAC projects in our dataset respondents reported positive impact and outcome. We attributed this to the widely-discussed "reporting" bias, which refers to researchers (survey participants) "under-reporting unexpected or undesirable results [IAC projects in our case], while being more trusting of expected or desirable results" (McGauran et al. 2010).
It was surprising to observe that 20 projects (19.8%) had "no impacts on industry". Respondents gave various reasons for such a lack of (measured) impact, stating, e.g., that industrial impact was not formally assessed (e.g., in P58), respondents were "Not sure [about impact]" (P77, P78, P80), "The approach was evaluated on a case-study industrial-like system in the 'lab'. The industry partner did not adopt the approach, to the best of our knowledge" (P18), "Project is still in progress (towards the end)" (P46), "The UML tool prototype showed feasibility, but was not used in the end (the tool was discontinued)" (P67), and "So far, it [the project] has not led to a change in the use of safety analysis techniques" (P84).
& Observation 8: A large share of IAC projects (76.2%) reported positive impacts on their industrial contexts. In about 20% of the projects, industrial impacts were not formally assessed or the project outcomes did not have noticeable impacts.

RQ 2-Challenges and their Negative Impacts in the IAC Projects
As discussed in the section on survey design (Section 0), one survey question asked about the impacts of ten types of challenges (adopted from the SLR (Garousi et al. 2016a)) in the projects, e.g., "Lack of research relevance" and "Lack of training, experience, and skills (of researchers or practitioners)". Responses were provided using the following five-point Likert scale: (0): no negative impact, (1): minor negative impact, (2): moderate negative impact, (3): high negative impact, and (4): very high negative impact. As shown in Fig. 16, all histograms are skewed towards the left indicating that the reported impacts of the challenges are low in general. In all histograms, the number of "0 -No negative impact" responses is higher than or equal to each of the other responses. This may denote the experience and expertise of the participants, in general, in carrying out IAC projects.

Ranking the Negative Impacts of Challenges
As the above ranking shows, mismatch between industry and academia (MIA) is the challenge with the highest observed negative impact. It has been reported in previous studies, e.g., (Sandberg et al. 2011;Wohlin 2013b;Rombach and Achatz 2007), that academia and industry have indeed different cultures, goals and objectives, e.g., academics' main goal is to conduct research and publish papers, while industry's main goal is revenue generation and to excel in their (software) business. Thus, it is very likely that this challenge is to some degree always present in IAC projects.
An aspect of mismatch between industry and academia is choosing the "right" topics for collaboration (what problem to focus on in an IAC project?). This issue has been extensively discussed in the literature. For some researchers, industrial problems lack scientific novelty or challenges, while some practitioners believe that most of the solutions developed in the academia have low applicability and scalability (Garousi et al. 2016a;Briand 2012). The first author and some of his industry colleagues had success in addressing this challenge in (Garousi and Herkiloğlu 2016), in which they reported a set of guidelines and a grounded-theory-based approach for selecting the right topics in IAC projects with focus on software testing. This approach could be easily transferred to other areas of SE.
& Recommendation 1: Choose the "right" topic for your IAC, since one aspect of mismatch between industry and academia is failing to choose the right topics for collaboration. Readers are encouraged to review guidelines such as (Garousi and Herkiloğlu 2016) to tackle this problem.   Furthermore, the SLR study (Garousi et al. 2016a) divides the challenge of industry-academia mismatch into those sub-challenges: Different terminology and ways of communicating; and Different communication channels and directions of information flow. To address those challenges, the SLR study suggested the following practices: (1) Having prior projects between researchers and practitioners and positive experience (which facilitates communication in current projects); (2) Personal interaction among researchers and practitioners during data collection; and (3) Running workshops and seminars (increases visibility across organizations, allows to show relevance, strength and ability of both sides).
& Recommendation 2: Tackle industry-academia mismatch in your projects using the variety of remedies proposed in the SLR study (Garousi et al. 2016a).
To address the second most important types of challenges, i.e., "human and organizational challenge"s, we observed that in IAC projects it is considered a good idea to involve the key staff within the company to clarify the common goals, to support the knowledge transfer, and to validate the ideas. The third most important challenge, i.e., "lack or drop of interest/commitment", should also be properly addressed. A crucial aspect in many human endeavors is commitment. Whenever organizations are involved, it is very important to guarantee the commitment from top managers. For example, ensuring the direct participation of CEOs of small companies (such as software startups) was indicated by one of our participants (P8) as being relevant for boosting the interest/commitment. In many contexts, having a champion is all that is needed to have success. It is also important to ensure engagement and to manage commitment.
The challenge with the lowest negative impacts is "lack of research relevance" (LRR, average = 0.90 out of 4), which means that the IAC projects included in our study, in their vast majority, are indeed based in industrial problems that require a research-oriented approach. In fact, in almost half of the projects (48 projects, 47.5%), the LLR challenge had no adverse impact at all (see the LRR histogram in Fig. 16). If we consider the cases where the LLR challenge had no negative or just a minor negative impact, the percentage of projects goes up to almost 75%. However, we should note that industrial relevance of academic research in general, has been criticized in many papers in SE, e.g., (Briand 2011;Jain et al. 2013;Lo et al. 2015), and in other disciplines (Mason 2001;Panda and Gupta 2014;Fox and Groesser 2016;Tan and Tang 2016;Olson et al. 2008). Since our dataset is in fact made up of IAC projects in which research had to be aligned with industrial needs, it is not surprising that this challenge has been rated very low.
& Observation 9: Mismatch between industry and academia (MIA) and Human and organizational challenges (HOC) have the highest negative impacts on projects.
When assessing challenges of the projects, we were aware of the ratio of the projects (~68%) for which positive impacts (outcomes) were reported (as discussed in Section 6.2). One would expect that the projects with positive outcomes are less likely to be impacted by the challenges, compared to projects with no or negative impacts in the sample. For this purpose, we divided the two groups of projects and calculated for each group the sum of the reported challenge levels (as gathered in the 5-point Likert scale introduced in Section 4.2). For example, for project P1, the levels of the ten considered challenges were: 0, 0, 2, 1, 1, 3, 3, 3, 3, and 0, which sums up to 16. Figure 17 shows two boxplots for the sums over challenge levels of the projects having positive impact (on the right) versus those without (on the left). We can see that there is no significant difference between the two cases, thus not confirming our hypothesis that the projects with positive outcomes were less likely to be impacted by the challenges, compared to projects with no or negative impacts in the sample.

How the Challenges Affected the IAC Projects
In one survey question we asked how the challenges impacted the projects. 70 of the 101 data points provided answers to this question. The full list of verbatim responses to this question is provided in Appendix 1.
To make sense of the textual data, we conducted qualitative coding and mapped the explanations provided by respondents to the list of challenges. Each explanation could refer to one or more challenges. We provide a few examples below:  & Explanation provided in P5: "After causal analysis sessions, we came up with a set of Improvement actions, but not all of them could be adopted, since this would change organizational standards beyond the scope of the project. Therefore, most of the reported benefits actually came from institutionalizing inspections and categorizing defects (and not from the causal analysis itself).": This was mapped to the challenge category: Human and organizational challenges (HOC) & Explanation provided in P6: "Organizational commitment and resources to implement [the] approach was difficult to get": This was mapped to two challenge categories: Lack or drop of interest / commitment (LDRC), and Resource-related challenges (RRC) We compared the number of times each of the challenges appeared in the provided explanations of how the challenges affected the projects versus the average negative impact of challenges (based on rubric data) as discussed in Section 6.2.1. The related visualization in the form of a scatterplot is shown in Fig. 18. The Pearson correlation of the two data series in this figure is 0.606, denoting a moderate correlation. We were actually expecting a higher correlation. The reason for the relatively low correlation is that some respondents mentioned negative impact for certain challenges (based on rubric data) but provided explanations related to other challenge types.
We then looked at the coded data (the groups under each challenge type),and synthesized the responses. We report a summary of that synthesis below, grouped by the type of challenges and ordered according to the frequencies with which challenge types occurred in the provided explanations. To support our discussion, we provide verbatim quotes of the received responses. For the case of P4, "the availability of industry contact persons made it challenging, because there are unavailable for a certain time interval or it took quite long to get answers". In many cases, unfortunately, the IAC project gets low priority when compared to other business-critical tasks. A project employee was involved in other important commercial projects during the research collaboration and therefore had to quit.
In addition to time and human resources, lack or shortage of financial resources was also a recurring challenge. For example, the data point P12 reported that "the project was not formally funded". We actually had a question in the demographics section asking about source(s) of funding (if any) for each project. As per our counting, 14 of the 101 projects were not funded (yet). A non-funded IAC is often executed out of curiosity and passion from both sides, or when there is a Master or PhD student who already works in the company and conducts her/his thesis on the topic that becomes the IAC. The first author had several such students in Turkey which resulted in successful theses and IACs.
Often, researchers work for free when there is no research funding for an IAC, sometimes only using the internal funding and resources of their universities. An example is P26, a project that was an industrial case study in automated visual GUI testing (Garousi et al. 2017b). In such IAC projects, due to lack of funding, there are inherent challenges to run the project, e.g., hard to hire additional PhD students, and inability to attend conferences to present papers, etc.
For P34, it was mentioned that "The project ended up being more a development/ engineering one, with a minimal necessity to include research efforts. The management of the project was interested in creating research evidences, like patents and papers. This, the team decided to write scientific papers to address the more research-oriented perspectives of the project". For 67, it was mentioned that "industry wanted solutions that would easily integrate in their current processes. academics wanted to propose their own methods".
For P50, it was mentioned that "It was very hard to find previous research done on this topic. And whenever we talked to other researchers, they didn't think it was a problem. Whereas when talking with practitioners, we often got the question if/when they would be able to give it a try. I guess you could say both sides perceive different challenges for (or their impact on) the industry".
For P51, it was mentioned that "The long-term research scope of researchers at universities does not link with short-term goals of many companies". For P34, it was mentioned that "Academics wanted to use UML, but no industrial partners were using it". For P86, it was mentioned that "the organization and the researchers had different objectives and backgrounds. so, each side tried to make the resulting products be like what they like. finally, the results did not make either of them happy".

Contractual, Privacy and IP Concerns (CPC)
Challenges related to CPC were the third most frequently discussed challenge and reported in eleven cases. For example, for P16, it was mentioned that "Security and privacy concerns were not discussed early on in the project. The only cancelation reason of the project was just that which was a non-technical issue, i.e., inability to get security clearance for two graduate students planned to be involved in the project". Thus, while contractual and privacy concerns may look trivial for people new to IAC projects, they sometimes get very critical and lead to cancelation of a project in which a lot of effort has already been spent.
A similar, but less critical, situation was reported for P49. The respondent mentioned his/ her disappointment with the fact that, although "rich findings and results from the industry" were generated, their research team could not "exhaustively discuss it and share it with the community. All the discussion and knowledge sharing [instead had] to be [done] through open-source, and rather smaller subjects". The same issue was reported for P68: "Privacy concerns made the process for publication harder and caused the paper to not have specific measurements that would have strengthened the arguments". Similarly, for P80, it was mentioned that "Not surprisingly, enterprises are paranoiac when it comes to security. This poses several challenges, e.g., from accessing to the company software for experiments to even name the company in published papers". In other projects (such as P69), due to security concerns, "all the development had to be done on-site [in industry]".
Issues w.r.t. IP were mentioned for several projects. For example, for P75, it was mentioned that "Multiple small to medium sized enterprises [were] involved in the project, each wanting to retain their own IP".
In some cases, team members had to go through long negotiations, e.g., in P3, "long negotiations [were held] before being able to deploy [the] tool".
Lack of Training, Experience, and Skills (LTES) Ten of the 70 provided explanations discussed negative impacts of challenges related to LTES. In P15, "lack of SE training in industry side caused the industry folks to undervalue the novelty of test techniques". In P35, "the project required knowledge of the software test process. A lack of experience, by the team, on the software development lifecycle made the team's velocity lower and a slight delay in the development of the application". Also, in P36, "development approaches required a multidisciplinary team with a significant learning curve on business context and tools to be adopted".
Several respondents mentioned technical and training challenges w.r.t. knowledge of UML in their projects, e.g., in P77, "industrial practitioners often need training to understand them [UML models], as they do not [often] use UML during their regular work".
Another project (P91) whose topic was on software process improvement, mentioned a similar challenge: "(…) their [the practitioners'] lack of process modeling knowledge and difficulty in explaining the processes while communicating the required information, their difficulty in reading the models".
Another similar issue was raised in P99: "The resistance of managers who do not have the sufficient background to understand the research ideas but still have a huge impact on the decision-making process".
Lack or Drop of Interest / Commitment (LDRC) Eight of the 70 provided explanations discussed negative impacts in projects related to LDRC. For P22, it was reported that "interest / commitment dropped early on". Such a situation could happen due to many reasons, e.g., when expectations are not met or when they are not clearly communicated.
In P32, the following observation was reported: "Much of the research published by academia was not of interest to the companies". That project's goal was to "understand problems in mobile application testing and addressing those problems by transferring solutions proposed in the literature". When the researchers provided a synthesized summary of the existing mobile testing literature to their industry partners, it seemed that the industrialists did not show much interest in reviewing them.
In general, many projects reported that organizational commitment and resources to implement research approaches were "difficult to get", e.g., P6. For P85 also, it was reported that "it was hard to keep the motivation of the key personnel in the organization motivated, since they did not have much incentive to contribute to an innovative project as it required more effort than their regular job descriptions".
Human and Organizational Challenges (HOC) Human and organizational challenges were reported in six cases. In some cases, "relocation of key people within the company" (P2) caused issues in the IAC project. It often becomes an issue when a given IAC project's industry contact point (also called "champion" (Wohlin et al. 2012)) changes the position within the company or moves to another company. In the former case, since the champion is still in the company, the same person may be able to continue her/his role in the project, or someone else may "take over" the role of contact point. If no proper arrangements are made, the project will most likely be paused or terminated (e.g., P13).
There were also organizational challenges, e.g., in P5, the team members "came up with a set of improvement actions, but not all of them could be adopted, since this would impact changing organizational standards beyond the scope of the project".
Resistance to change, by staff and organizations, was another mentioned challenge, e.g., in P88 ("additional effort was needed to be spent to overcome the resistance of employees") and P91.

Management-Related Challenges (MRC)
Six of the 70 provided explanations discussed negative impacts due to management-related challenges. Changes in company management or priorities often produce challenges in IAC projects, e.g., P39 and P44.
IAC projects may also struggle to get management to "believe in the business value of the collaboration" (P53). It is also usually not easy to engage management "because academia is basically slow and [often] unwilling to adapt" (P57).

Lack of Research Relevance (LRR)
Five responses discussed negative impacts due to lack of research relevance. In P13, "the connection link between industry and academia got weak in middle of project due to turn-over in the company, and thus gradually led to lack of research relevance".
For P38, "the research relevance of the project lost its significance during the project. Although the project had focused on relevant research issues related to test processes, those research issues were not exploited since the defined test process had not been sufficiently studied".
In P41, "the idea was to build a measurement system in [an] EU project. Parts of the measurement where not novel and those got implemented. All the other goals were unrealistic and did not get implemented. [There were] lots of bureaucracy involved".
Unsuitable Research Methods (RM) Four responses discussed negative impacts due to unsuitable research methods. For P67, it was reported that "Industry wanted solutions that would easily integrate in their current processes. Academics wanted to propose their own methods, tools/techniques that they are fond of, without considering the actual needs of the industrial partners". Such a research approach is often called "advocacy research" (also called the "research-then-transfer") (Potts 1993;Glass 1994) which is often not received well by industry. Researchers have commented on how to address such an issue. For example, (Tichy et al. 1993) called for "a paradigm shift ... from purely theoretical and building-oriented to experimental". (Griswold and Opdyke 2015)suggested to researchers to "see the world [SE practice] as it is". (Glass 1994) suggested using the "industry-as-laboratory" approach, in which "researchers identify problems through close involvement with industrial projects, and create and evaluate solutions in an almost indivisible research activity" (Potts 1993).
For P47, it was reported that "We [the team members] missed a major flaw to validity through combined lack of understanding related to test flakiness", denoting issues in the research method. A challenge in P53 was that "[the] tension between scientific quality and practical applicability cannot be avoided but can be balanced with experience".
For P57, it was reported that "they [researchers] come in with a hammer and look for nails", again denoting fundamental issues in research methods. Similarly, for P67, it was reported that: "academics wanted to propose their own methods, tools/techniques that they are fond of, without considering the actual needs of the industrial partners", again highlighting the negative impact of the "advocacy research" (also called "research-then-transfer") (Potts 1993;Glass 1994).

Communication-Related Challenges (CRC) Three responses discussed negative impacts due
to communication-related challenges. Communication habits/patterns of the involved parties might be different, e.g., speed in responding to emails or technical tasks (such as data collection). If both sides (researchers and practitioners) do not show "understanding" for such different styles, there could be challenges.
For P53, it was reported that "Communication habits/patterns are challenges and need champions on both sides to be overcome". For P38, it was reported that" Due to some connection problems (Skype mainly), there was some challenges related to communication". Thus, we can that, for the case of remote collaborations, it is important to ensure smooth and reliable teleconferencing facilities.
P20 was a project on engineering (development) of scientific software (Segal and Morris 2008;Farhoodi et al. 2013). For that project, it was reported that "Since software engineers worked with other types of engineers in the oil industry (e.g., chemical and mechanical engineers), as expected, there was considerable gap of knowledge in the two sides w.r.t. the other side. The two sides had to struggle to communicate and understand each other in many occasions".
One Challenge could Lead to One or More Additional Challenges By analyzing the data, we also came across a few cases in which a challenge could lead to one or more additional challenges. For example, for P10, there was a "demanding industry partner" and expectations of both sides were not clearly communicated and thus one side put more demand on the other side. Such a situation often negatively impacts the relationship (the so called "social capital" (Al-Tabbaa and Ankrah 2016)) which in turn degrades the performance of the IAC project, and could result in the Drop of interest/commitment" challenge.
As another example, for P13 "the connection link between industry and academia got weak in middle of project due to turn-over in the company". Such a challenge "gradually led to lack of research relevance". & Recommendation 3: Review how the challenges affected the projects in the dataset of this study (summarized above), and the suggested best practices provided in the SLR study (Garousi et al. 2016a) to be able to cope with similar challenges in your IAC projects.

Other Challenges Observed
Another survey question (see Q3.3 in (Garousi et al. 2016c)) asked participants about any other challenge(s) they might have observed in their projects, in addition to the ten categories of challenges that we had asked in a previous question (as discussed above in Section 6.2.1). This question was answered for 24 of the 101 projects.
After analyzing the provided responses, we found that 20 of those 24 "other challenges" could actually be classified under our own ten challenge types (see the above sections). We show a few of those responses that we classified under our own ten challenge types below: The remaining four of those 24 "other challenges" were related to getting funding in support of the IAC project. We itemize those responses below: & "Time and effort spent for bureaucratic issues due to the involvement (communication with) multiple funding partners" (P1) & "Lack of understanding for empirical approach by sponsors. Terminology was also a problem as people often tend to make (negative) associations with certain terms. We simply talked about technology transfer when actually meaning technical action research (to not be associated with ivory tower research)" (P66). The first author also recently experienced these challenges in the context of a recent grant proposal in which he partnered with three industrial companies and two other research institutes on the topic of "testing scientific software". After submitting the proposal to the Dutch national funding agency (called: NWO), it was rejected and the main comment was that most of the work was empirical and there was little "scientific" merit in the proposal. & "To get funding, in the grant applications you need to promise the Moon (a grant application in Software Engineering that does not also cure AIDS and cancer as byproduct is not worthy to fund). But what you get in the end (i.e. in terms of PhD students and postdocs to hire) is not enough to even cover 1/10th of what promised... the industrial partners might not be happy about it" (P67) & "It was not easy to get the project funded, given the bureaucracy at a Brazilian federal university. I just moved to a privately held university [name removed to keep anonymity] that has a huge tradition in promoting university-industry collaborations. Promoting and facilitating industry collaborations was the main motivation for my move". (P71)

RQ 3-Practices (Patterns and Anti-Patterns) and their Impacts in the IAC Projects
For RQ 3, we measured the impacts of applying practices in the IAC projects. We intentionally did not classify practices into patterns (what to do to ensure success) or anti-patterns (what not to do to avoid failure) as it was done in the survey (Garousi et al. 2016a), since we wanted to see whether extracted data for impacts related to each practice would denote it as a practice resulting in positive or negative outcomes (i.e., patterns and anti-patterns). As discussed in Section 4.2 (survey design), responses were provided using the following five-point Likert scale: (−2) -Very negative impact; (−1): Negative impact; (0): Neutral (the practice did not have any impact); (+1): Positive impact; and (+2): Very positive impact. An "empty" answer would denote that "The practice was not applied / NA". Figure 19 shows the reported impacts of practices on projects, classified using the five-point Likert scale. For example, the "Ensuring engagement and managing commitment (ENMC)" practice had largely positive (48 votes) or very positive impact (29 votes).
To be able to compare the impacts of practices, we calculated the average values for all the 101 data points. For example, if there were equal number of −2 and + 2 values for a given practice (and not other values), the average would be 0, denoting that considering all data points, that practice had overall a neutral impact. Figure 20 shows the average values.

Patterns (What to do to Ensure Success)
Based on the values in Fig. 20, the three practices with the highest positive impacts are: (1) Working in (as) a team and involving the "right" practitioners (WTI), average impact = 1.39 (in range of [−2, 2]) (2) Having mutual respect, understanding and appreciation (HMRU), average impact =1.37 (3) Considering and understanding industry's needs, and giving explicit industry benefits (CUIN), average impact =1.28 & Observation 10: The three most important practices in IAC projects to increase chances of success are: (1) Working in (as) a team and involving the "right" practitioners, (2) Having mutual respect, understanding and appreciation, and (3) Considering and understanding industry's needs, and giving explicit industry benefits.
We noticed interesting "contrasting" situations for some of the practices. For example, the HMRU practice had an overall positive impact (i.e., it ranked the second most helpful practice above), however, in the case of one data point (P22), HMRU had a "high" negative impact. The following supporting comment was entered by the corresponding participant for P22: "the industry side did not appreciate the need for software testing research". For another project (P21), "moderate" negative impact was reported for the HMRU practice, with the following comment: "appreciation from the industry side decayed by time". Thus, we can see that while most of the helpful practices were supporting IAC projects, there could be exceptional opposing cases due to differences in the project contexts. Note that, in such cases, it is not appropriate to say that the practice itself is an anti-pattern or had a negative impact. One should rather say that not being able to satisfy (implement) the practice yields a negative impact (like the examples of projects P21 and P22). We had two follow-up questions asking for a narrative explanation of the effect of the practice with the most positive impact. We reviewed those explanations about manifestations of the patterns and matched them with the corresponding patterns. We show a subset of the matching data in Table 5, which presents qualitative data and lessons learned.
& Recommendation 4: Review and consider using the patterns (as listed in Table 5) in your IAC projects.
Based on the qualitative data provided by the participants, we give the following recommendations with respect to the top-3 reported patterns to increases chances of success in IAC projects: & Having mutual respect, understanding and appreciation: Recommendation 5: Develop a relationship of trust with the involved practitioners. Trustful relationship is key in the successful execution of many IAC projects. Often, the physical presence of the researchers at the industrial partners' facilities, during some periods of time, proves helpful in developing this relationship of trust. & Working in (as) a team and involving the right practitioners: Recommendation 6: Develop a team spirit in your IAC projects. Like in collective sports, team spirit is crucial for a successful IAC project. A good practice, indicated by Table 5 Patterns and their manifestations in the sample data points (quotes from the data)

Patterns
Manifestations (Example quotes from the survey data) Proper and active knowledge management (PAKM) • The fact that we can get feedbacks right away! Knowledge transferring to the industrial practitioners is the key. Ensuring engagement and managing commitment (ENMC) • We keep including the practitioners in the loop of the project day by day and I think that's the most importance practice. • In our project, SME's were involved. Management commitment was therefore essential. Indeed, two CEOs even participated directly in the project. Considering and understanding industry's needs, and giving explicit industry benefits (CUIN) • The academia side carefully considered and understood industry's needs, and gave explicit industry benefits during the project. • Giving explicit industry benefits and solving the right problem, in our case we categorized defects found during peer reviews and focused directly on them. Thus, we were using real and relevant data, in which the industry partner had a direct interest. One of the researchers was a trainee at the organization, which gave us on-site presence and easy access to all the data. Finally, the improvements were measured precisely in terms of defects per function point and also defects per inspection hour." • Keeping the companies happy and having constant discussion with them. This in turn is associated to a precise measurement approach to enable clearly showing the achieved benefits to the industry partner stakeholders. Also, the new technology that was applied during the project had been evaluated before, in academy and with industry representatives, before applying it in the real case study. Having mutual respect, understanding and appreciation (HMRU) • The most positive practice was having mutual respect, understanding and appreciation. • When having a project with multiple partners (and limited resources), goal alignment becomes essential. • Developing a relationship of trust with the involved practitioners was the key in the successful execution of study steps. Being Agile (BA) • "We applied the Agile practices and since the project was based on action-research, we reacted to changes and needs ASAP. This led to success." • Being agile in organizing research • Allowing the researcher full reign after describing the problem and enabling them to go wherever the data took them. Working in (as) a team and involving the "right" practitioners (WTI) • "Tight and regular knowledge exchange during the project (typically bi-weekly)" • Finding the right people in the right environment is the key of success of such a project, in my experience.

Considering and manage risks and limitations (CMRL)
No discussion quotes in the data.
Researcher's on-site presence and access (ROSP) • "Appointing researchers who have domain knowledge and full-time access to subject systems" • "Build trust and spend time on site" • The physical presence of the researcher for some time at the company was quite helpful in developing this relationship of trust. Following a proper research/data collection method (FPRM) • The research method (experiment) was well understood and, due to that, the management had trust in our results. Managing funding/recruiting/partnerships and contracting privacy (MFRP) No discussion quotes in the data.
Understanding the context, constraints and language (UCCL) • In order to get quickly to the point and to engage with the company experts, it is important that the researcher has solid technical (and practical) background and speaks the language of the practitioners. Efficient research project management (ERPM) No discussion quotes in the data. Conducting measurement/ assessment (CMA) No discussion quotes in the data. Testing pilot solutions before using them in industry (TPS) • Testing pilot solutions in lab settings before using them in industry was a good practice Providing tool support for solutions (PTS) No discussion quotes in the data. many of our respondents, to boost this team spirit is to consider the presence of the academics at the facilities of the industrial partners. This practice seems to strengthen the bond between the two sides. Recommendation 7: Organize frequent and regular meetings during the IAC project. One recommendation is to meet twice a week, to ensure that the knowledge is exchanged among the team members. Another guideline is to involve academics with domain knowledge (i.e., knowledge on the subjects being addressed by the IAC project). In fact, researchers should have solid technical and practical background and be able to speak the language of the practitioners. & Considering and understanding industry's needs, and giving explicit industry benefits: Recommendation 8: Make sure that each side adapts, as much as possible, to the other one (e.g., in terms of terminology, needs, etc.). Academia and industry have different mindsets and distinct time frames. A good recommendation is for academics to give explicit industry benefits and solve the right problem. In some cases, this entails simple things such as using real and relevant data, in which the industry partner has a direct interest.
Recommendation 9: Follow iterative approaches in IAC projects: Adopting iterative approaches, such as the ones suggested by agile methods, seem useful as they allow industrial partners to iteratively adapt their needs, a practice that is followed in some contexts (Hicks and Foster 2010).

Anti-Patterns (What not to do to Avoid Failure)
In terms of anti-patterns (the last four practices in Fig. 20), the participants reported negative impacts for them, as was expected. The anti-pattern with the highest negative impact is "Following self-centric approach" (FSCA) in conducting IAC projects, i.e., each side (industry and academia) focuses only on its needs and objectives in the project. We have had personal experience in how this particular anti-pattern can ruin an IAC, for example, we have been involved in an IAC in which a researcher just focused on the academic objectives (publishing papers) after the project started and did not consider the industrial partner's objective of decreasing test costs in practices (although it was mentioned in the project description for the funding agency). This anti-pattern led to a decrease of mutual trust and loss of "social capital" (Al-Tabbaa and Ankrah 2016) on the side of the industrial partner which unfortunately led to a poor execution of the IAC project at the end.
As Fig. 20 shows, the other three anti-patterns were also empirically observed in the dataset: Poor change management; Ignoring project, organizational, or product characteristics; Unstructured decision structures. We reviewed the quotes from the dataset that provided narrative explanation on manifestations of the anti-patterns and matched them with the four anti-patterns types. We show a subset of the matching data in Table 6. As we can see, there is a huge amount of qualitative data and lessons learned in these quotes, which we present and hope that readers can benefit from.
& Recommendation 10: Review and take the anti-patterns (as listed in Table 6) as cautionary lessons in your IAC projects.
Based on the qualitative data provided by the participants, we provide the following recommendations with respect to the top-three reported anti-patterns to increase chances of success in IAC: & Poor change management: Recommendation 11: Be more flexible to change in the IAC project. We have observed in a few projects of our own that many academics are usually work in big up-front design (planned) fashion and are not flexible to changes. However, industry in general is becoming more flexible to change and we have seen that this can sometimes cause issues when things do not go as planned, e.g., when the project's main contact point in the company leaves the company without notice. & Ignoring project, organizational, or product characteristics: Recommendation 12: Adopt and pay attention to needs of both sides of the IAC project. In some cases, researchers are interested in just proving the concept, while the industry partners are often looking for tools with full functionality and relevant quality attributes (usability, portability, maintainability).
Recommendation 13: Pay attention to organizational characteristics (norms). Researchers are encouraged to have the "soft" skills to follow the specific written and nonwritten organizational norms. Following self-centric approach • "Lack of [active] collaboration (between industry and research partners) can make the project challenging (and hinder successful projects)" • "A self-centric approach not taking the specific situation of the 5 SMEs [involved in the project]" • "The industry side did not appreciate the need for SW test research" • "A PhD student was the bridge between the two sides (he worked for the company). Self-centric approach was slightly the case" Ignoring project, organizational, or product characteristics • "Underestimating the fluctuations regarding workload at the industry side due to deadlines, crisis situations, relocation of employees, etc." • "Underestimating the rigidness of the development process under study" • "Due to the team lack of experience in the test domain, in the beginning of the project the team had some difficulty understanding the project context." • "To sit down and get acceptance to conduct measurements in an organization/group (as part of a large organization) that has "little time" is often futile (since their connection to a specific measurement is often vague, it's good measurement for the company, but not for the group that needs to collect measurements). Therefore, new measurements are hard to get started, or old measurements might get removed." Poor change management • "The connection link between industry and academia got weak in middle of project due to turn-over in the company, and from that point on, the project was led mostly by the academia side. The industry side 'just stayed on paper'" • "Strictly following the research plan that made no sense. Wrong partners" Unstructured decision structures • "Lack of full control on the resources provided by the industry" • "Unclear and non-transparent decisions" & Following a self-centric approach: Recommendation 14: Avoid self-centric approach. We have observed in a few projects of our own that many researchers follow self-centric approach in that they consider their ideas and method unquestionable and thus act in a self-centric fashion. This usually turns off industry partners and should be avoided.

Discussion
In this section, we first discuss the implications of our findings and recommendations and then the limitations and potential threats to validity of this study.

Implication of Findings
Based on the synthesis of all the data provided by the participants in the previous sections, we offer four take-away messages in this article that should ensure successful IAC projects. They are the following: (1) establishing a common goal; (2) recognizing and understanding the fundamental difference; (3) understanding and team work; and (4) managerial topics.
A common goal is a cornerstone of any IAC project. In the histogram showing the impact of challenges (Fig. 16), we see that lack or drop of interest is the challenge with the highest negative impact. Mismatch between industry and academia is the challenge with the second highest impact. Both of these challenges reflect that a common goal for the research project is missing. Without a common goal that is both interesting and beneficial for both parties, IAC projects have a little chance of succeeding. This is highlighted by the quote stating that "Finding common ground is hard but important" (P52).
The fundamental difference denotes the fact that, ultimately, academics want to get top journal publications and material for Ph.D. degrees from the projects. Industry wants a solution that is useful and comes with low maintenance cost. For academics, making improvements to a tool or process or writing a publication is a common research goal. Industry, on the other hand, wants long-term solutions, as they have to live with the new tool or process for the years to come, while the academics move to the next challenge. For example, a new tool that is correct 90% of the time might be a great improvement in a specific industry setting and thus the industry partner would in a next step focus on making sure that the tool is usable and maintenance free (able to handle all special cases, for example). In contrast, academics would focus on developing a new prototypical tool that is 95% correct. Thus, the academic interest would be to continue to improve the tool's correctness no matter whether the new prototype renders the tool less usable and increases maintenance cost or learnability (for example requiring longer running time, more manual steps, more information from different data sources). Our suggestion for this fundamental issue is that both sides recognize that such a point exists where interest would differ and define what to do at this point. In an ideal case, industry would start to ensure tool usability and low maintenance, while academics would be allowed to work on the tool until prior work is beaten.
Understanding and team-work ensure that IAC projects move smoothly and the common goal is not lost in the process. Industry and academia have different cultures, backgrounds, and objectives. Thus, it is natural that all of the top-3 ranked success criteria deal with the topic of gaining understanding and forming a team: 1. Having mutual respect, understanding and appreciation (HMRU) 2. Working in (as) a team and involving the "right" practitioners (WTI) 3. Considering and understanding industry's needs, and giving explicit industry benefits (CUIN) Even when a common goal has been found and a team with a mutual understanding and respecting has been formed, an IAC project can fail due to management related issues. For example, contractual and privacy concerns need to be taken into account. Getting and keeping higher levels of management commitment is important, as otherwise top-level mangers can abruptly abort IAC projects if they think the company employees are wasting their time.
Internal company policies need to be understood; for example, some units or sites of a large company may not have permission (or have limitations) to be involved in research activities. Some of these managerial topics could be impossible to bypass and may lead to ending a research project before it has even begun. We should also mention in this context that challenges of an IAC project should not be analyzed in isolation, as challenges are often inter-related with patterns and anti-patterns. Figure 21 shows a cause-effect diagram of inter-relations of challenges, patterns, and antipatterns (adopted from (Garousi et al. 2016b)). Challenges and anti-patterns are expected to negatively impact success, while applying patterns is expected to positively impact success. Challenges necessitate the need for applying patterns that address those challenges. Patterns and anti-patterns could neutralize each other. For example, the anti-pattern "Following selfcentric approach (FSCA)" could damage (neutralize) the benefits gained by applying the pattern "Working as a team and involving practitioners (WTI)".
Furthermore, anti-patterns usually bring more challenges. Thus, participants of an IAC usually apply patterns (which we discuss later in the context of RQ 2) to address challenges. For example, to decrease the cultural and objective mismatch between industry and academia, various practices, such as those grouped under the "Understanding the context, constraints and language (UCCL)" pattern (Garousi et al. 2016a), are usually applied.

A Checklist for IAC Projects
In order to help researchers and practitioners who plan to conduct an IAC, we suggest a checklist as shown in Table 7. The checklist is based on the recommendations extracted in Sections 6.1 and 6.2. The first ten items in the checklist apply to both researchers and practitioners. The 11th item is particularly important to be checked by researchers. After each checklist item, we indicate the recommendation(s) from which it is derived.

Limitations and Potential Threats to Validity
In this section, we discuss potential threats to the validity of our study and steps we have taken to minimize or to mitigate them. The threats are discussed in the context of the four types of threats to validity based on a standard checklist for validity threats presented in : internal validity, construct validity, conclusion validity and external validity.
Internal Validity Internal validity is a property of scientific studies which reflects the extent to which a causal conclusion based on a study and the extracted data is warranted . A threat to internal validity in this study lies in the selection bias (i.e., lack randomness of the projects participating in our survey).  (Garousi and Herkiloğlu 2016).
x x 1 2. In case you identified a mismatch of expectations and interests under Item 1, try to resolve the mismatch using IAC best practices such as those proposed in (Garousi et al. 2016a).
x x 2 3. Review the list of challenges listed in Sec 6.1.1 and check whether you will be able to cope with similar challenges in your IAC project.
x x 3 4. Review the patterns listed in Table 5 and check which ones are the most usable in (beneficial to) your IAC project.
x x 4 5. Check whether you have established a relationship of trust between the actors involved in your IAC project.
x x 5 6. Check whether you think that you will be able to develop team spirit among the actors in your IAC project.
x x 6 7. Check whether you will be able to organize frequent and regular meetings during your IAC project (e.g., once or twice a week).
x x 7 8. Check whether each partner (researchers and practitioners) has sufficient benefit from your IAC project. The expectations should be made explicit and aligned as much as possible.
x x 8 , 1 2 9. Check whether the IAC project will follow a flexible, iterative approach allowing each partner to mutually adapt and align their needs and expectations.
x x 9 , 1 1 10. Review the anti-patterns listed in Table 6 and check whether there is a risk that any of those anti-patterns apply to your IAC project.
x x 1 0 11. Check whether each partner has the soft skills to pay enough attention to organizational characteristics and norms important for the other side in your IAC project. Avoid a self-centric attitude.
x 1 3 , 1 4 Construct Validity Construct validity is concerned with the extent to which the objects of study truly represent theory behind the study . In other words, it relates to whether we actually measured industry academy collaboration project in our study. One form of industry academy collaboration project we might have covered only partially are Master's thesis student projects, which in many countries are made while the students work on the industry payroll. A typical goal of such thesis is to improve a tool or a process used by a company. These projects might have been missed because they do not often result in academic papers, the university does not formally manage them, and the academic contributions are limited. At the same time, based on discussion with industry, it seems that these projects deliver timely and real benefits to industry and industry is also continuously willing to invest their money for thesis student projects. Let us recall from Section 4.4 that, in total, 64 respondents provided the information on the 101 projects. Each respondent provided between 1 and 19 data points. A majority of the respondents (57 people) provided only one data point, thus we can say that a large number of data points came from different people. The data points from the same respondent correspond to different projects and thus there is some level of independence between them. It is also common for people to deflect their answers when they feel being evaluated and based on what they think is the intended result of a study. To mitigate this, we informed participants prior to the survey that our motive in this study was to take a snapshot of IAC projects and that we did not intend to publish any identifying information so that participants will remain anonymous.
Conclusion Validity Conclusion validity of a study deals with whether correct conclusions are reached through rigorous and repeatable treatment . We analyzed, qualitatively, challenges and success criteria of IAC projects. For each RQ, we attempted to reduce the bias by seeking support from the data gathered in the survey and subsequent statistical analysis. Thus, all the conclusions that we present in this article are strictly traceable to data.
External Validity External validity is concerned with the extent to which the results of this study can be generalized . Given the moderate number of 101 projects included in the analysis (after screening), the external validity is somewhat limited, but still the largest one reported so far in the literature. Also, the results might be more or less representative, depending on SE area in the scope of an IAC project. In the set of projects we used for our analysis the fields "testing", "quality" and "process" are the most frequent ones, while "professional practice", "configuration management" and "maintenance" are the least frequent ones. This could indicate that some SE areas are more appropriate and relevant for conducting successful IAC projects than others.
As discussed in Section 4.4, we were vigilant about survey reliability, i.e., representativeness of the dataset and sampling (Gobo 2004). Although we used "convenience" sampling, we took various measures to ensure survey reliability. In summary, by using convenience sampling in our work (similar to many other survey studies in SE), although representativeness could be limited, but we increased the external validity of the survey by ensuring sample relevance and sample prototypicality (representativeness) (refer to Section 4.4 for details).
Furthermore, since the unit of analysis in this study is an IAC project, the true population is all IACs in SE. In our survey execution, to ensure getting as many data points as possible, we have asked each respondent to provide as many data points (IAC projects) as possible, thus we have taken the convenient sampling approach for sampling respondents conveniently, and also sampling projects conveniently, which is away from true random sampling. Therefore, the external validity is impacted with this convenient sampling approach.
Also, as discussed in Section 6.1.3, a large percentage of the sample included projects reporting positive impact and outcome. We attributed this to the widely-discussed "reporting" bias, which refer to researchers (survey participants) "under-reporting unexpected or undesirable results [IAC projects in our case], while being more trusting of expected or desirable results" (McGauran et al. 2010). Approaches to address this potential validity threat could be to increase ratio of less-successful projects by removing from the dataset a randomly-selected subset of projects with positive impact, or to encourage survey participants to also report projects with no impact. We had actually made that encouragement in our invitation emails. As for the former approach to increase ratio of less-successful projects, we decided not to remove from the dataset any of the projects with positive impacts as to not decrease our dataset size.

Conclusions and Future Work
This article makes four contributions. Firstly, we report the largest survey, ever in the Software Engineering (SE) community, on industry-academia collaborations (IAC) projects. Our results are based on 101 different projects from 21 different countries, and covering 11 (out of 12) SWEBOK KAs. Secondly, we show that lack or drop of interest/commitment (LDRC) is the most highly observed challenge in the projects. On the other hand, the challenge with the highest perceived negative impact is resource-related challenges (RRC) followed closely by LDRC. Thirdly, our statistical analysis unexpectedly shows that perceived challenges were not correlated with project success or failure. Fourthly, in order to ensure success in IAC projects, we suggest focusing on three issues: (1) Finding and maintaining a common goal; (2) focusing on mutual understanding (between academia and industry) and teamwork; and (3) making sure that managerial topics do not prevent project success. In fact, good management in research projects cannot guarantee success, but poor management can prevent or shutdown a project, therefore effectively preventing any type of success to occur.
Our future work directions include the following possibilities: (1) to use findings from this study with respect to challenges, patterns, and anti-patterns in our upcoming IAC projects; (2) to adapt the paper to have industry practitioners as the targeting audience; and (3) to analyze which issues are most likely to be relevant in order to guarantee that a given IAC project is successful, taking into account its characteristics (SE topic, culture of the country where it is being developed, project initiators, etc.). This sort of result can provide important indicators for SE researchers and practitioners, so that they can analyze them before progressing with IAC projects.
Furthermore, we plan to conduct methodological and empirical studies for improving IAC in SE by adopting novel ideas from other fields, e.g., the concept of social capital and its role in IAC (Al-Tabbaa and Ankrah 2016), the contingent role of collaboration-specific attributes in IAC (Lin 2017), and focusing on "innovation performance" in IAC (Huang and Chen 2017 & A project employee was involved in other important commercial projects during the research collaboration and therefore had to quit the project & Academics wanted to use UML, but no industrial partner was using it. So most of the time was spent in the learning the software system of the industrial partners to be able to model it, which of course remove important resources from building the solutions (e.g. tools) that industrial partners need additional effort was needed to be spent to overcome the resistance of employees & After causal analysis sessions we came up with a set of Improvement actions, but not all of them could be adopted, since this would change organizational standards beyond the scope of the project. Therefore, most of the reported benefits actually came from institutionalizing inspections and categorizing defects (and not from the causal analysis itself). & As for any IAC project involving UML, lot of time is needed to prepare the models, and the industrial practitioners often need training to understand them, as they do not use UML during their regular work & Availability of company for checking results & Boss controlling everyone. Multiple small to medium sized enterprises involved in the project each wanting to retain their own IP. & Building usable tools take a lot of time, especially when you also have to write academic papers and run experiments. But industrial partners couldn't care less about these latter, and have the wrong expectation that academics would code full time like normal engineers & Challenges such as the lack of interest from the bank employees to the project, their lack of process modeling knowledge and difficulty in explaining the processes while communicating the required information, their difficulty in reading the models. Resource assignment was not done in time when compared with their commitment. & Confidentiality constraints and lawyers in general were by far the most harmful problems & Contracts had to be signed before we started, which significantly slowed down planning. & Contractual and confidentiality related challenges such that it was forbidden to work in weekends in the organization and organization's network was a must to be used in the project. There was no access to the organization network & Demanding industry partner & Development approaches required a multidisciplinary team with a significant learning curve on business context and tools to be adopted & Different time horizons between industry and academia. Industry is interested in solutions in a timeframe of 3-6 months. This is too short to start research projects together with university. For instance, it is hard to manage master thesis projects part of an industry research collaboration. Ph.D. projects are not possible at all. & Difficult to get time for dissemination of results and necessary feedback & Difficult to invest in external academic research, because 1) they know less (!) about testing than industry -"Why should we teach academics-I thought they would HELP us"? 2) learning curve in general difficult on industrial systems and industrial practitioners are busy, unwilling to change and have other targets -it must make the everyday work betterif it does not -not interested 3) management hard to engage because different time -resource and feedback -academia is basically to slow and ""lazy""and unwilling to adapt. They come in with a hammer and look for nails, instead addressing issues at hand. Researchers should more be generalists but still know coding. Also, most of my collaborations are Ph.D. students from Industry. Their issues are a bit different. For pure academic (Ph.D. students) their work is most often considered a ""waste""of effort for Industry. Senior researchers hard to engage. Some universities in Sweden are fighting about IPR and do not sign NDA deals -a deal-breaker. In general. We I am always surprised when I get collaboration to really work. But the effort internally is usually VERY large (10-15 meetings before deal is done). & Time horizons is so different. Industry want results within a few months, but academics are.... SLOW. Takes years. So, problem is already solved and obsolete by the time we really got the academic to understand all issues. And of course -resources are always very scares for this ""extra curricula""activities as research is seen as." & Due to security concerns, all the development had to be done on-site. & Especially in large companies as this one it is difficult to get the support. In our case, strangely, we lacked the support by higher management while we managed to get champions at project level. Those champions get us the necessary contacts in product development to complete the project successfully. The end results than convinced higher management. Wouldn't we have had the champions, we'd never be able to get through the project & I think the different vision is the real turning point, although I don't see any real solution.
Industry deals with "white papers" which would never pass a peer review. & In this particular collaboration there was a huge interest by the industry partner. They wanted to improve their requirements engineering practices and in a previous contact we, from the university side, had already informed them that we had the know how to support them on this matter. The main challenge was highlighted as "Human and organizational challenges" because we applied defect causal analysis and as a result had a list of improvement actions (listed in the paper). There is a natural resistance to change in process improvement in general. Therefore, not all the actions that we would like to implement could actually be implemented in practice. Nevertheless, they were happy with the outcome of the project and future collaborations are planned. & Industry wanted solutions that would easily integrate in their current processes. Academics wanted to propose their own methods, tools/techniques that they are fond of, without considering the actual needs of the industrial partners & Intellectual property rights and privacy limited access to data (challenge C61), were not discussed early one. The cancelation reason of that project was one single purely nontechnical issue, i.e., inability to get security clearance for two graduate students planned to be involved in the project. Thus, we observe that, even if an IAC project does not possess challenges from many aspects, one single major challenge is enough to lead to its halt/failure. & Interest / commitment dropped early on & It seemed that much of the research published by academia was not of interest to the companies. & It was difficult to identify the right contact points in the companies for the survey; interest to participate was generally low; the topic RE is not really in the focus for most companies (as they ae often technology-driven) & it was hard to keep the motivation of the key personnel in the organization motivated, since they did not have much incentive to contribute to an innovative project as it required more effort than their regular job descriptions. Also, since the personnel was not trained and experienced in software engineering and BPM, they were uncomfortable in critical making decisions. & It was quite difficult to find participants at the company for the study since the number of experts was quite small. Also, reviewers at academic conferences were not understanding of industrial constraints e.g. not understanding why pilot studies of "moon shot" ideas are useful. & It was very hard to find previous research done on this topic. And whenever we talked to other researchers, they didn't think it was a problem. Whereas when talking with practitioners, we often got the question if/when they would be able to give it a try. I guess you could say both sides perceive different challenges for (or their impact on) the industry. & Lack of background/experience in research from important stakeholders & Lack of commitment translates into people becoming unavailable, or at least delays; different time horizons, communication habits/patterns are challenges and need champions on both sides to be overcome; the management needs to believe in the business value of the collaboration; tension between scientific quality and practical applicability cannot be avoided but can be balanced with experience. challenges, e.g. from accessing to the company software for experiments to even name the company in published papers & Organizational commitment and resources to implement approach was difficult to get & Originally, much more participants were committed. Due to other, more important things, the number decreased dramatically. & Privacy concerns made the process for publication harder and caused the paper to not have specific measurements that would have strengthened the arguments. & Privacy issues Different priorities (industrial partners had deadlines and no specific resources to invest in the research we were doing). & Project was not formally funded & RE: Mismatch between industry and academia: the industry side was not that interested to continue the work after one single tool development and paper. & Relocation of key people within the company & Since software engineering worked with types of engineers in the oil industry (e.g., chemical and mechanical engineering). As expected, there was considerable gap of knowledge in the two sides with respect to the other side. The two sides had to struggle to understand each other in many occasions & Some of the consultants we worked with were not trained in qualitative research and observation; management did not value the research outcomes and focused on the business objectives. & The availability of industry contact persons made it challenging, because there are unavailable for a certain time interval or it tool quite long to get answers. & The connection link between industry and academia got weak in middle of project due to turn-over in the company, and thus gradually led to lack of research relevance, e.g., Results produced by research are not measurable and exploitable (mechanisms for exploiting them are missing) & The fact that with a huge and rich findings and results from the industry, we cannot exhaustively discuss it and share it with the community. All the discussion and knowledge sharing need to be through several open source, and rather smaller subjects. & The goal was to apply published modeling approaches to company data; not all published approaches were sufficiently described to be able to reproduce them & The lack of training etc. was a challenge since several of the engineers were domain experts and not testers. This posed a challenge since it was testing we were doing, but otoh this was also the main objective with the project, i.e., that an interactive search-based tool would help domain experts in testing the software. & The long-term research scope of researchers at universities does not link with short-term goals of many companies. Researchers should organize their research in a more iterative manner, including the financial part. Instead of asking budget for 4 year research they should organize for 3 months research goals, with an option on extension when successful. & The main issue is the lack of willingness to go the extra mile to make the approach really applicable: e.g. addressing most of C++, taking care of the proper build system. & The most challenge thing on the project was the connection with other systems or receiving information from other partners. Unfortunately it wasn't easy to guarantee that but the team found alternatives ways to receive the necessary information. & The most significant challenge is that industry partners did not have much time to dedicate to the project. They had to do their own work since company was not willing to invest much time of their employees on the project. & The most significant challenge was to get the resources required to move the research from prototype/proof of concept to production & The organization and the researchers had different objectives and backgrounds. So, each side tried to make the resulting products be like what they like. Finally the results did not make either of them happy. & The organization personnel resisted to change and had a hard time to understand the goals & The project ended up being more a development/engineering one, with a minimal necessity to include research efforts. The management of the project was interested in creating research evidences, like patents and papers. This, the team decided to write scientific papers to address the more research-oriented perspectives of the project. & "The project required knowledge of the software testing process. & This lack of experience, by the team, on the software development lifecycle made the team's velocity lower and a slight delay in the development of the application." & The project was performed in an organization in defense sector and the information confidentiality was high. The academicians needed to be within the organization facilities to run the data analysis. That was not always feasible as the academicians had time constraints. & The research relevance of the project lost its significance during the development of the tool. Although the project has focused on relevant research issues related to testing processes, those research issues were not exploited and the defined testing process has not been sufficiently studied. & Most of the team members had no experience in testing, test tools and test processes. Part of the team members had no knowledge of SCRUM approach. These two topics have caused some delays in the initial development of the project. & Some elements of the development team were in a different company facility, and due to some connection problems (Skype mainly) there was some challenges related to communication, particularly in the Sprint Reviews and Daily Meetings. & One element of the team, due to software license issues had to use personal laptop to develop the project." & The resistance of managers who do not have the sufficient background to understand the research ideas but still have a huge impact on the decision making process. & Things developed in the project genuinely helped the companies. However, when customers who pay the companies what something that takes precedence over anything. Company people are busy and having them to run e.g. RCA workshops completely on their own is challenging. When we run it for them they were very happy and active participants in the sessions. Changes in company management produced challenges. When the company side management of the research project changed then the researchers had to "resell" their research to the new managers. New managers in general are eager "to prove their worth" and thus getting rid of anything extra that takes money or resources can be seen as an achievement. We were successful in re-selling our project to the new managers. & This was not really a research project with clear focus. The idea was to build a measurement system in EU project. Parts of the measurement where not novel and those got implemented. All the other goals were unrealistic and did not get implanted. Lot of bureaucracy involved & Time pressure & We had research deadlines different than project deadlines & We had issues in the meetings, the company folks wanted full agile dev, while some researchers wanted waterfall & We had to significantly censor many of our discoveries. & We missed a major flaw to validity through combined lack of understanding related to test flakiness. The technique preferred scheduling flaky tests. & We started with some groups/projects and then after a while they changed the managers and we had to start over to "sell" the model, why we do this etc. This happened several times during the years Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
Michael Felderer is a professor at the Department of Computer Science at the University of Innsbruck, Austria and a guest professor at the Department of Software Engineering at the Blekinge Institute of Technology, Sweden. His research interests are in the areas of software and security engineering with a focus on testing, quality, processes, data analytics, requirements, and empirical research methodology. Michael Felderer has more than 15 years of practical experience and performs research in close collaboration with industry. He is as an editorial board member of the journal Information and Software Technology (IST), organizer of conferences (e.g., General Chair of PROFES 2017, Program Co-Chair of SEAA 2017, Industry Co-Chair of ESEC/FSE 2019) and regular PC member of premier software engineering conferences. Furthermore, he is also a member of the steering committee of PROFES, RET, TAIC PART, and ISERN. For more information visit his website at mfelderer.at.
Mika Mäntylä is a professor of Software Engineering at the University of Oulu, Finland. He received a D. Sc. degree in 2009 in software engineering from the Helsinki University of Technology, Finland. His research interests include empirical software engineering, software testing, software maintenance, mining software repositories, and behavioral software engineering. He has previously worked as a post-doc at the Lund University, Sweden and as an assistant professor at the Aalto University, Finland. His studies have appeared in journals such as IEEE Transaction on Software Engineering, IEEE Software, Empirical Software Engineering, and Information and Software Technology. He currently serves as an associated editor for IEEE Software. For more information http://mikamantyla.eu/ David C. Shepherd is a Senior Principal Scientist & Research Area Coordinator with ABB Corporate Research where he conducts research on topics from developer productivity to end-user programming for robots. He is passionate about increasing the impact of software engineering research, which has manifested itself in his professional activities. His day job serves as a ongoing personal experiment on how to, and how not to, transfer technology. His open source work has created a tool downloaded by 47,000+ developers. His recent chairing of five industrial tracks in software engineering (e.g., ICSE, FSE, ICPC, SANER, ICSME) gave him deep insight into the challenges and successes of others' transfer efforts. His recent project, the FlowLight, which had exposure on venues like The Wall Street Journal and BBC Radio, opened his eyes to the impact of popular publications. Finally, his position as the Co-Editor-In-Chief of the Journal of Systems and Software gives him a more permanent platform to encourage industrial publications.
Andrea Arcuri is a Professor of Software Engineering at Kristiania University College, Oslo, Norway. His main research interests are in software testing, especially test case generation using evolutionary algorithms. Having worked 5 years in industry as a senior engineer, a main focus of his research is to design novel research solutions that can actually be used in practice. Dr. Arcuri is the main-author of EvoMaster and a co-author of EvoSuite, which are open-source tools that can automatically generate test cases using evolutionary algorithms. He received his PhD in software testing from the University of Birmingham, UK, in 2009.