Data collection in global software engineering research: learning from past experience

Abstract

Global Software Engineering has become a standard in today’s software industry. Research in distributed software development poses severe challenges that are due to the spatial and temporal distribution of the actors, as well as to language, intercultural and organizational aspects. These challenges occur in addition to “traditional” challenges of the domain itself in large-scale software projects, like coordination and communication issues, requirements volatily, lack of domain knowledge, among others. While several authors have reported empirical studies of global software development projects, the methodological difficulties and challenges of this type of studies have not been sufficiently discussed. In this paper, we share our experiences of collecting and analysing qualitative data in the context of Global Software Engineering projects. We discuss strategies for gaining access to field sites, building trust and documenting distributed and complex work practices in the context of several research projects we have conducted in the past 9 years. The experiences described in this paper illustrate the need to deal with fundamental problems, such as understanding local languages and different cultures, observing synchronous interaction, or dealing with barriers imposed by political conflicts between the sites. Based on our findings, we discuss some practical implications and strategies that can be used by other researchers and provide some recommendations for future research in methodological aspects of Global Software Engineering.

Appendix A—The Experiences of the Authors

This appendix provides details on the aims and methodologies of the various case studies and ethnographies which formed the basis of our experiences. Table 3 also shows the details of each of the experiences.

Table 3 Details of the experiences of the authors

Study 1

In Avram’s first study (2007a, b), Avram et al. (2009), the field site was an Irish subsidiary of a Fortune 500 multinational company involved in software development. Following a first contact with R&D managers, the request of following a software development team over an extended period of time was presented to a number of development managers, and one manager volunteered, with the agreement of his team. The study began in January 2006 and the research team adopted an ethnographically informed approach. The first 3 months allowed the two researchers to familiarize themselves with the context and the work being done. In the following 15 months, they spent more than 70 full days in the field, observing the activity of the team in its own work environment, participating in meetings and group activities and occupying desks in the open plan in the team’s area. 26 interviews were realised with members of the team based in Ireland, Germany and in the US, including the team lead, the two software architects, the lead of the QA team, software developers and quality engineers. Some of the team members were interviewed more than once. The research team was granted access to the company intranet, to the project’s document repository and to the team’s mailing list. They were also allowed to make use of the company’s own instant messaging system, useful both as an awareness mechanism and as a communication channel with the members of the observed team. Observation and interaction continued online when the researchers were not present on site. Participation in teleconferences allowed them to observe directly the team members’ interactions with people in various other locations (US, Germany, India). A good working relationship developed between the research team and the software development team and the researchers found many opportunities for conducting both formal interviews, and informal discussions on various topics—without disrupting the actual work. The researchers kept diaries, taking detailed notes on every day spent in the field. Remote collaborators of the observed team were also interviewed, either via instant messaging/phone or face-to-face. One of the researchers travelled to one of the company’s sites in Germany in November 2006 and interviewed five people with different roles in the collaboration between the two sites.

The data collected from the field was periodically discussed and analyzed on a weekly basis by the extended research team, in order to identify topics, trends and problems and compare the findings to those from other similar sites where fellow research team members were observing similar processes and activities. The findings were shared with the members of the software development team, their managers and remote collaborators in conversations, specially designed workshops and in the form of draft reports and papers.

Study 2

In Avram’s second study (Richardson et al. 2008; Boden et al. 2009, 2011), one of the cases presented is that of an Irish company with a development division in Romania. The method chosen for this study was that of a case study. One of the researchers, of Romanian origin, found the company website on the Internet. She contacted the Irish manager and organized an interview at the company headquarters in September 2007. After 2 months, she had the opportunity to travel to Romania, visit the Romanian branch and interview the Romanian manager (and co-owner) and 3 of the developers. At the time, the company was employing 3 project managers based in Ireland and Romania and 19 developers, all of them being located in Romania. Besides interviews, a limited amount of observation was undertaken in the two locations and a few documents were collected. Both the intermediary report and the finished paper were shared with the two managers and received positive feedback.

Study 3

In Boden’s study (Boden et al. 2007, 2008), the researchers conducted an ethnographically informed case study in two German SMEs engaging in offshore software development with partners in Russia. The goal was to understand how software developers in distributed teams organize their development work in terms of Articulation Work, and how organizational learning can be organized in distributed settings.

The researchers started with an exploratory analysis of the literature on offshoring, covering discourses of various communities of practitioners and scholars. Based on these findings, relevant research questions were identified, focusing on informal and situated coordination practices of developers. On this basis, the researchers conducted 15 semi-structured interviews with managers and developers of different German SMEs. From this sample, two companies (named Alfa and Beta for the purpose of this study), which had offshored part of their software development to subsidiaries in Eastern Europe, were selected for closer investigation in the form of case studies.

For the case studies, the researchers conducted a triangulation of several ethnographic research methods, comprising of further semi-structured interviews, participant observation, as well as artifact analysis. For the participant observation, each of the German SMEs was visited for a period of ten to twenty working days. A third participant observation period was conducted at the Russian partner company of company Alpha—in Tomsk, Siberia. Beta’s partner company could not be visited because of an abrupt end of the cooperation, but the researchers were able to cover the perspective of the Russian developers by conducting interviews via Skype with the Russian senior developer. During their time spent at the companies, the researchers had ample opportunities to observe local and distributed articulation processes, in the context of meetings, individual work situations and cooperative tasks. Several informal interviews were conducted and the researchers were allowed to analyze artifacts such as e-mails, chat protocols, internal work papers and whiteboard sketches.

The findings were documented by means of field notes and photos that were taken during the research. For validation, the findings were correlated with the literature focusing on articulation processes in software development. Furthermore, the identified concepts and topics were discussed with the participants of the study during a workshop.

Study 4

In Prikladnicki’s first study (Prikladnicki et al. 2007, 2008) the authors have conducted an exploratory case study of distributed projects in five multinational companies. The data collection methods included interviews and document reviews. The documents reviewed were project plans, lessons learned, and documents describing the software development process. A total of 20 individual interviews (lasting 1 h each) included technical leaders, project managers, IT managers and directors. Interviews were conducted face-to-face -in Brazil and Canada, and over the telephone—with informants in the U.S. In Brazil, the access to the companies was facilitated by the researcher’s previous contact. In Canada, the contact was facilitated by a professor living in that country, who had previous contact with the two companies. The professor was also part of the research team. In the U.S., the contact was made through the Brazilian subsidiary director, who has made the interview possible. The respondents were selected by convenience. Informants from three management levels: project management, information technology and portfolio management, and organization management were interviewed. Among them, there were six site directors, five information technology managers, seven project managers, and two technical leaders. The unit of analysis was the subsidiary. Findings were shared in the form of technical reports, papers, and also included in a book (the first book published in Portuguese on this theme).

Study 5

In Prikladnicki’s second study (Prikladnicki et al. 2003, 2006), a case study was conducted in two software development subsidiaries, each one owned by a multinational organization with worldwide spread units. The first organization worked mainly in consulting, software development projects and training and had external clients. At the time when the research was conducted, it had nine software development subsidiaries located in Brazil. The organization also had offices located in Brazil and other countries in Latin America, as well as in the U.S. and Europe. Its headquarters were located in Brazil. The second organization supported and manufactured computers. It had three software development subsidiaries located in two continents that were responsible for internal client demand worldwide. The headquarters were located in the U.S. The data collection included primary and secondary sources. We conducted 22 individual interviews (11 in each organization), covering four projects, two in each organization. All interviews were conducted in Brazil, facilitated by previous connections between the companies and the research team. Secondary sources were also used: document reviews, meetings minutes, and software development process descriptions, together with public information available on the homepage of each organizations. For data analysis a content analysis protocol was defined and applied.

The respondents were selected according to the unit of analysis and the purpose of the study. Among the interviewees, there were project team members, development managers, quality assurance team members, software process improvement responsible and individuals representing the organizations’ strategic level. Two questionnaires were developed, each considering a specific dimension to be explored: “organizational,” containing information about the organization as a whole, and “project,” with information on the four projects included in this study. One development manager was interviewed in each organization, whereas five interviews were conducted for each project. The convenience sample was not probabilistic, although the research team tried to get a good representation of all groups involved. Data collected was evaluated by practitioners in both companies—with positive feedback. Findings were used as an input for a training program on Global Software Engineering within one of the companies under study.

Study 6

In the de Souza’s study (De Souza and Redmiles 2011), fieldwork was conducted in a large software development company that will be called LAR for the purpose of this article. LAR was one of the largest software development companies in the United States, with products ranging from operating systems to software development tools, including e-business and tailored applications. The project studied was responsible for developing a mobile application that had not been released yet. The project staff was divided into three major groups: user interface (UI) designers, software developers, and the quality assurance (QA) team. The staff was distributed over five different sites, spread in three different countries: North Carolina, US; Massachusetts, US; Beijing, China; Shanghai, China; and Taipei, Taiwan. To be more specific, user interface design and evaluation was performed by six professionals in North Carolina, and the implementation was performed in all other sites distributed as follows: nine developers in Massachusetts, five in Shanghai, five in Beijing, and four in Taipei. The quality assurance team was divided between U.S. and Chinese sites: three engineers were located in Massachusetts and six engineers in Beijing. The main coordination of the project and the project manager were located in Massachusetts, where all the data were collected.

Data was collected through document analysis and semi-structured interviews. Among other documentation, artifacts, emails and instant messages exchanged among the software engineers were collected. Access to shared discussion databases used by the software engineers was also granted for the research team. All of this information was used in addition to the notes generated by the interviews. We conducted 17 semi-structured interviews with members of all teams from the different sites: some interviews were conducted face to face, and others were conducted by telephone, with one interview conducted via instant messaging. The interview questions were designed to encourage the participants to talk about their everyday work, including work processes, problems, tools, communication, collaboration, and coordination efforts between their collocated and distributed colleagues. The interviews also aimed to explore the relationship between software dependencies and the coordination of software development projects, or, to be more specific, the potential usage of dependency information to facilitate collaborative software development. Interviews lasted between 20 and 70 min. The interviews with some of the Chinese team members were conducted while they were visiting one of the US sites.