Collaborative traceability management: a multiple case study from the perspectives of organization, process, and culture

2.1 Empirical studies on traceability approaches

While a number of empirical studies have been published on traceability, few studies are aimed at describing the state of the practice of traceability and practical challenges in general. Most studies focus on validating specific technical approaches [1, 28, 53], or specific aspects of traceability such as assessment [43] and benefits of traceability [29].

Klimpke and Hildenbrand [25] conducted five interviews to identify how practitioners use traceability. The majority of the companies were “high-end users” of traceability, aiming to support verification and to track project progress. In all five companies, traceability starts from the requirements level. In contrast, our study includes a significantly larger sample as well as a wider spectrum of roles. We found that while such requirements-centered approaches are common in practice, there also exist traceability approaches focusing on implementation artifacts such as issues and code changes.

Mäder et al. [30] identified four categories of traceability users: (1) the regulated group, where traceability is mandated by standards, (2) the sub-contractor group, which inherits traceability requirements passed from the companies they supply to, (3) the consultants, who design traceability strategies for their clients, and (4) the enthusiasts, who practice traceability not because it is mandatory but because they see its benefit. They conclude that there is a need for more guidance on traceability, better tools, and more empirical studies on how traceability is established and used in practice, especially across boundaries.

With respect to traceability goals, Bouillon et al. [6] report traceability usage scenarios and conclude that traceability is mainly used for tracking the origin and rationale of requirements, for tracking project progress, and for certification. The authors also report on the use of trace links for collaboration: Trace links help to identify relevant stakeholders and notify them about changes.

Gotel et al. [18] collected data from more than 100 practitioners focusing on the requirements traceability problem. Besides reporting on challenges, we also investigate detailed practices and their situatedness in development processes found within the companies. Ramesh [39] investigates how organizational, environmental, and technical factors influence requirements traceability practices. In his study, he distinguishes between high-end users and low-end users of traceability and gives a list of factors that affect traceability and recommendations for how companies can transition from low-end to high-end traceability practices. While some traceability practices are reported, the rise of the agile paradigm calls for a new study of these aspects.

Arkley and Riddle [2] investigate traceability challenges and conclude that it is essential to select appropriate roles for traceability management. Due to their knowledge of the system, developers are better suited to create high-quality links than, for example, separate quality assurance teams. However, while developers could create high-quality links, they often do not see the direct benefits of traceability. The authors suggest that organizations should ensure that goals and needs of developers are addressed in traceability development contracts that define traceability management of shared artifacts. The findings of our study suggest that the potential of using traceability for collaboration can in fact motivate developers to see a benefit of investing in traceability.

2.2 Traceability in agile

As pointed out by Cleland-Huang [7], traceability in projects following agile methodologies is just as important as in non-agile ones. The goals are essentially the same, but the available artifacts are different and some of the documentation that is enforced in non-agile projects to, e.g., prevent embellishment, is omitted in favor of trust within the team and direct communication. Trace links commonly exist between requirements and test cases and can be accompanied by references to code in commit messages and between requirements. Cleland-Huang also suggests a classification of agile projects [7] in (i) small-to-medium-sized agile projects, (ii) large-scale, distributed, or long-lived projects, and (iii) safety-critical projects. These classes differ in their traceability needs and in the applicable techniques, e.g., relying only on direct communication is not feasible in large-scale distributed projects. In this paper, we not only investigate the impact of process on traceability, but also conduct a broader investigation on characteristics of the development process on collaboration and traceability management (RQ4).

Espinoza and Garbajosa expand on the idea of specific traceability concerns in agile projects [13] and argue that the lack of formal documentation and formal requirements calls for traceability practices that go beyond those of non-agile projects. Accordingly, the key is to closely connect requirements and tests via trace links. To do this, they propose a traceability information model, concrete roles involved in managing traceability, and rules that determine if and when links have to be established. According to our findings, even companies following more rigorous agile methodologies are not yet embracing such formally defined traceability approaches. However, the suggestions made by Espinoza and Garbajosa could be the basis for adopting traceability practices in such cases.

Gayer et al. [17] give a concrete example of integrating traceability in an agile context. In their example, trace links between different artifacts are established in a rigorous manner (e.g., user stories, test cases, and architecture models). The authors argue that the time required to create trace links was offset by the benefits, including easier sprint planning as well as “a structured way to communicate requirements among stakeholder groups.” This corresponds to our findings that traceability can indeed be an enabler for collaboration.

Furtado and Zisman [16] focus on how traceability can help organizations transition to the agile paradigm. In particular, they argue that traceability can help to mitigate the typical lack of understanding of a project’s “high-level scope” or issues with control by management. Trace links between artifacts created following the plan-driven paradigm and artifacts created following the agile paradigm can help to address these challenges. Our findings suggest additional traceability management-related differences and similarities between agile and “traditional” projects.

2.3 Collaborative traceability management

The topic of traceability management can be targeted with different levels of human involvement. For instance, trace links can be automatically recovered, recommended by a classifier [42], or evolved over time using tool support [38]. However, it was found to be beneficial to also involve human decision making in trace link evolution [38]. An underlying issue is that practitioners are often hesitant because of the quality of automatically generated trace links—and “prefer no links at all to (possibly) inconsistent links” [31]. We found the importance of seeing traceability management as a collaborative effort of stakeholders who leverage it to support collaboration and create incentives for individuals to create, maintain, and use trace links together.

Demuth et al. [11] conducted a study on how to use traceability for systems engineering to facilitate change notification and consistency checking of artifacts. They describe how traceability can facilitate communication between departments enabling them to identify what artifacts and stakeholders will be affected by a change. According to the authors, the main challenge in providing traceability in such a scenario is the diversity of tools that complicates traceability between artifacts stored in different tools.

With respect to tooling, Figueiredo et al. [14] describe a tool that facilitates collaboration in a distributed setting by identifying developers affected by a change via trace links. Helming et al. [21] present a tool that also uses trace links to notify users of changes; this technique is able to generate a lower number of relevant notifications compared to standard notification techniques by leveraging modifier, creator, and assign information. Strašunskas [56] discusses traceability in collaborative development environments with a strong focus on technical aspects such as how to version and store artifacts (including trace links). While Strašunskas focuses on how such artifacts can be edited in a collaborative environment, our focus is on traceability practices and challenges in such environments. Cleland-Huang et al. [9] identify a need for further empirical studies that shed light on aspects of social collaboration tools and their impact on traceability management. This is related to our investigation of how collaboration is affected by traceability management.

3.1 Research validity

We discuss six categories of threats to validity in qualitative research presented by Maxwell [33, 34]:

Descriptive validity Although one cannot completely avoid this threat, member checking [50] is one way to mitigate descriptive validity. Moreover, it was helpful to create transcripts to allow for a word-by-word analysis of the interviews. Aspects such as gestures, pauses, or irony were, however, not captured in the transcripts which might have resulted in misinterpretation of the statements. In cases of doubt, we therefore referred to the recordings to clarify confounding parts in the transcripts.

Interpretative validity At the beginning of the interviews, we ensured that there was a common understanding of the terminology of traceability. Most interviews were conducted in the native language of the interviewees or with interviewees with an advanced level of English. While this helped prevent misunderstandings, we faced another difficulty: Selected quotes were translated from German to English. We tried to maintain the meaning of the translated statements in the translations and discussed critical parts with other researchers to mitigate the threat of translation inaccuracies.

Theoretical validity One threat of validity is the potential of interpreting data from a biased perspective because of preconceptions we built up during this study. We potentially interpreted data differently that were not in line with our preconceptions. Feedback from fellow researchers helped to mitigate this threat. We discussed our methods and findings throughout the study and especially during the analysis.

Researcher bias When analyzing qualitative data, there is a threat of being biased by the researchers’ own background, values, and theories. However, in our study, we did not have any incentives to arrive at certain conclusions, which would have been different if we had evaluated a method or tool. Collecting feedback from fellow researchers helped to mitigate this threat, as well as using member checking to collect feedback from interviewees.

Reactivity: A researcher always influences the setting of an interview [34]. The way questions are phrased and how one reacts to answers influence subsequent answers in the interviews. Formulating the questions as unbiased as possible and validating them in pilot interviews was one strategy to mitigate this. We could also identify misleading questions and find appropriate ways to react to (surprising) answers during the pilot interviews.

Generalizability: Generalizability is concerned with the transferability of our findings to other situations or cases than the ones we focused on in this case study [33]. The objective of qualitative research is to describe the particular cases rather than arriving at generalizable conclusions [10].

We were able to study the phenomenon traceability from different angles using triangulation [46]. There might, however, be a sampling bias as we asked for participants who were knowledgeable in the area of traceability and probably had a positive attitude toward the topic.

4.1 Collaboration across boundaries

The necessity of coordinating traceability management activities across disciplinary and tool boundaries greatly complicates traceability management.

Open image in new window

4.2 Common goals and responsibilities

Responsibilities of individuals need to be coordinated, and the benefit of traceability has to be seen. We found this to be challenging in practice, for reasons explained in the following.

4.3 Collaborative trace link maintenance

Traceability maintenance is highly relevant to ensure high trace link quality. In this section, we present challenges related to traceability maintenance and collaboration.

5.1 Opportunities: collaboration across boundaries

We identified two opportunities with collaboratively addressing traceability across boundaries: (1) easier communication in distributed environments (Sect. 5.1.1), and (2) interdisciplinary engineering (Sect. 5.1.2).

5.2 Explicit documentation of decisions

5.3 Creating trace links to receive information

Another idea from our interviewees was to react to artifact changes by sending the person(s) responsible for connected artifacts an automatic notification and request to review all affected trace links.

6.1 Traceability goals’ relation to collaboration

6.2 Requirements-centered traceability management

We categorized eight of 15 cases as relying on a requirements-centered traceability management approach, characterized by a strong focus on requirements management when it comes to the use of traceability.

With the exception of Case 6 and Case 13, all of the requirements-centered cases rely on a dedicated requirements management tool for organizing and structuring the requirements. The tool most commonly used was IBM Rational DOORS³ (especially in automotive cases), used in four of the eight requirements-centered cases. Among other features, the tool allows managing requirements and test cases on different levels of granularity. It not only supports traceability between these artifacts, but also between other artifacts stored in external tools through interfaces such as OSLC.

An information management tool is also used in Case 6, storing requirements, analysis and design models, test cases, and trace links between requirements and other artifacts.

The usage of traceability management in requirements-centered approaches is characterized by fixed, plan-driven processes, often officially following the V-model. Many of these cases are part of large organizations or represent situations in which several companies work together. Often, customer–supplier relationships and collaboration with external organizations are relevant in these cases. This goes along with more organized processes and responsibilities.

In Case 15, for example, formal reviews are conducted before a milestone is completed, also making sure that trace links are set. In this particular case, stakeholders are invited for the review using a feature in DOORS that sends out e-mails automatically.

It should be noted that the majority of the requirements-centered cases belong to the automotive domain where safety criticality and OEM–supplier relationships require formal specifications and organized ways of working.

Interviewees following the requirements-centered approach stated that communication is typically handled via e-mail and document exchange. Meetings are often held using telephone or video conference systems, as well as regular face-to-face meetings when possible. However, generally, a rather formal approach to collaboration is followed.

In most of these cases, traceability was established in a top-down fashion from management side, motivated by quality or safety standards that the company wants to fulfill. We observed that the more relevant safety or quality certifications are for a company, the more formally defined are the processes for requirements-centered traceability management. It also relates to the formal way of collaborating in these cases. Trace links are typically created by project managers, test managers, software architects, and/or the responsible developers for features.

6.3 Developer-driven traceability management

Besides requirements-centered traceability management, we identified more developer-driven, ad hoc approaches, used in five of 15 cases.

In most of these cases, the approach involves the use of an application lifecycle management or software configuration management system. Team Foundation Server⁴ is used in two of five cases. In many cases, bug or project tracking tools such as JIRA⁵ together with a version control system like Git⁶ are used, e.g., to manage implementation tasks or reported defects of a project, collectively referred to as “issues” in the following.

The motivation for traceability is the potential to relate changes in the code to the connected requirement or issue. This is particularly used during troubleshooting. It is also used to track the status of the project.

The creators of trace links in developer-driven traceability management are typically developers with a focus on the actual software implementation. The approach is often only used in the development departments in an organization, sometimes only by a small team.

Communication is often handled in face-to-face conversations, (weekly) meetings, and via e-mail exchange. Compared to the requirements-centered approach, traceability is not discussed as much in conversations and meeting, and is not considered an essential organizational goal. Instead, informal ways of collaborating are frequently used, which makes it more difficult for stakeholders to see the need for formal documentation.

Developer-driven traceability management is especially common in cases with an agile paradigm. Three of the cases following the developer-driven approach use the agile paradigm, and the fourth agile case in our data follows a mix of approaches, as shown in Table 2. In these cases, the software configuration management system is also used to plan the work. Collaboration with other organizations is less relevant in these cases. For this reason, the natural collaboration approach is rather informal and focused on the needs of developers.

6.4 Mixed approaches

In two cases, there are attempts to handle traceability management in a structured, top-down way, and at the same time, bottom-up approaches are pursued by the developers. Note that while this suggests a best-of-both-worlds approach, we could not observe a systematic endeavor to combine tracing information from both parts.

Depending on the discussed issue, various means of communication are used. More formal communication is used for the requirements-centered part, and informal collaboration within developer teams. Our cases focus on direct conversations wherever possible, but also exchange e-mails. Traceability-related topics are typically not discussed a lot, given that they are not as central for the cases’ organizational goals as in the requirements-centered approaches. The collaboration with external stakeholders is less relevant in these cases.

7.1 Rigorous cases

There are several rigorous cases—both related to plan-driven approaches and to agile approaches. We describe both groups in the following.

Some of the plan-driven cases use a significant amount of reviews and assessments to ensure correct trace links or by more clearly defining and controlling responsibilities. Less rigorous cases allowed processes and responsibilities to emerge bottom-up. The only reported case in which end-to-end traceability was achieved was a plan-driven rigorous approach.

Besides rigorous plan-driven cases, we also found cases following rigorous agile paradigms. Rigorous agile cases are characterized by a strong focus on quality, a global definition of agile values, principles, and practices beyond individual development teams, and the purposeful use of traceability. The importance of face-to-face communication as an agile principle influences how stakeholders collaborate. The introduction of the agile paradigm is typically not done in an ad hoc way, but in a well-planned, long-term transition process.

One commonality in rigorous agile cases is that interviewees describe the trace link quality as “good, but not good enough” (a software quality manager from Case 9). The interviewees showed interest in improving their traceability approaches. They all state that tool support is missing to leverage traceability in a better way (e.g., a software quality manager from Case 9 and the Head of the Project Management Office from Case 12).

7.2 Balanced or lenient cases

We observed that a number of cases followed a less rigorous approach to traceability management and collaboration. Many of the balanced or lenient cases are currently undergoing a transition from plan-driven to agile paradigms. It was mentioned by several interviewees that the organization requires V-model processes, whereas software development teams or departments internally work in more agile ways. These cases reported difficulties with trace link quality and maintenance. Trace links are mainly used for troubleshooting, but “links are not always there when they should be there,” as mentioned by a team leader from Case 2. The interviewees reported that traceability is typically established in a bottom-up way inside the development department, but the scope is too limited to be beneficial for the company. From a collaborative perspective, it could be observed that collaboration typically works well in a smaller team, but organizational boundaries are not crossed to exchange knowledge in a satisfying way.

8.1 Challenges

We answer RQ1 based on our findings: What are practitioners’ challenges with collaboration in traceability management? We identified three groups challenges that we summarize and discuss in the following:

8.2 Effects of traceability management on collaboration

In the following, we discuss our results for RQ2: How can traceability management support collaboration?

We found that collaborative issues can be mitigated with adequate traceability management. This requires that collaborators have common goals, and see that the perceived effort of traceability is exceeded by the benefit of overcoming such boundaries. The issue of tool boundaries could be facilitated by tool suppliers that move toward common standards such as OSLC.⁷ Whereas tool boundaries and the lack of applicable tool support play a role in practice, they are not the only complicating factors to be blamed. More severe is the issue of invisible benefits of traceability which makes it difficult to leverage traceability in practice. Therefore, we see it as imperative to create direct, tangible advantages for engineers that actively maintain trace links by allowing them, e.g., to more easily identify the impacts of change or to engage other members of their distributed team, and thus ensure that a positive cost–benefit balance is perceived by all stakeholders. Our findings indicate that traceability can be used to enable collaboration and facilitate communication and knowledge management, especially in distributed teams.

Our findings indicate that traceability management can improve collaboration in the following four ways:

8.3 Traceability management approaches

In the following, we discuss our findings concerning RQ3: How does collaboration relate to different approaches of traceability management?

We found that there exist (1) requirements-centered traceability management, (2) developer-driven traceability management, as well as (3) mixed approaches, combining the former two. As the names suggest, requirements-centered traceability management is focused on requirements as the main artifacts, whereas developer-driven traceability is mainly conducted and leveraged by developers. Organizations typically introduce requirements-centered traceability to assure the quality or safety of their products in a top-down manner. Developer-driven traceability management originates from development teams.

In requirements-centered cases, collaboration is often achieved in a formal setting that helps regulate the interactions between a typically larger team and a customer who is often very involved in the projects. Much of this collaboration is facilitated through a requirements management tool, and both traceability and collaboration are focused on the requirements. In contrast, developer-driven cases are usually found in an environment where a small team communicates informally and face to face and where collaboration with external stakeholders is less relevant. An issue tracker is often used as the main locus of traceability, e.g., between commits and tickets.

These findings indicate that the concrete way that collaborative traceability management is implemented in an organization depends heavily on the established collaboration approaches and the team size. To realize the potential benefits of traceability management on collaboration as presented in Sect. 5, a traceability strategy in a more formal setting needs to be equally formalized with strong tool support and potential enforcement by the tool. On the other hand, in less formal settings it is easier to establish norms and rules regarding collaborative traceability management in an ad hoc fashion within the smaller team. Enforcement can be based on social pressure or a definition of done.

Our findings cannot confirm Cleland-Huang [7]’s finding that trace links between requirements and test cases in agile projects are used. In the agile cases of our case study, traceability of requirements to test cases was not mentioned. To some extent, we can confirm Mäder et al.’s classification [30]: There exist “regulated” traceability users, sub-contractors, consultants, and enthusiasts. For instance, the regulated type is in line with the requirements-centered approach, motivated by the need to comply with standards and enforcing processes. Our findings do not suggest that the concrete types of inter-organizational dependencies (working as a consultant or sub-contractor) strongly impacted the way traceability is used. However, collaboration across organizational boundaries did play a role and comes with challenges, as we presented in Sect. 4.1.2.

8.4 Contextual factors’ influence

In the following, we summarize and discuss RQ4: What characteristics of the development effort influence traceability management and collaboration?

Similar to earlier studies, we found that traceability management can also be leveraged in agile projects. Based on our data, we could not confirm differences between traceability management in agile contexts that depend on scale, complexity, and safety-criticality that were presented in [7]. This can be connected to the fact that the large-scale, complex, and safety-critical cases in our study came from the automotive domain where the transition to agile paradigms is still in progress. We found that traceability management can be used in both agile and V-model-based contexts. However, our findings indicate that it might be desirable to rather rigorously define and follow traceability management strategies. It is difficult to find a good cost–benefit balance if the practices are followed in an inconsistent way and the quality of trace links drops. We capture this point in the following principle: (P4) Strive for a rigorous culture with respect to traceability maintenance.

We identified several cultural, organizational, and process factors that impact how traceability management is conducted. Differences between approaches can be most prominently observed in the development paradigms and the rigor of following them. Our findings suggest that agile and plan-driven cases focus on different artifacts and purposes of traceability: Cases following agile paradigms tend to focus more on development artifacts, whereas plan-driven cases use requirements-centered traceability. It is crucial to analyze the traceability goals and choose and appropriate approach that fits to the purposes. We relate these points to the importance of stakeholders’ information needs and goals (P1), and capture them in the following sub-principles: (P1a) Choose a lightweight, developer-centric approach for goals related to tracking change to code. (P1b) Choose a formal, requirements-centric approach for goals related to regulation (e.g., safety or legal requirements).

Moreover, the trace link quality and maintenance appear to relate to the culture and rigor: Lenient cases and less well-defined paradigms reported difficulties with trace link quality and maintenance. In case trace links are not established in a systematic manner, stakeholders struggled with identifying the benefit and use of traceability, and thus with finding motivation to create trace links.

Our findings are in line with Rempel et al.’s finding that practitioners need to specify traceability goals, and find ways to implement and assess them [44]. In practice, this is often not the case, which results in inconsistencies between the goals of the development paradigm, traceability goals, and the actual trace links [44]. As a consequence, trace link quality is often too low to be suitable for practitioners.

We found successful traceability both in plan-driven and agile cases. However, only two of the cases followed agile paradigms and had a rather rigorous culture. During the analysis, we found that culture and rigor play a critical role, in addition to the development paradigm. Had we found these emerging aspects earlier, we could have aimed to include more agile/rigorous samples. Future research will have to scrutinize whether this finding can be generalized beyond our sample.