Keywords

1 Introduction

Non-Functional Requirements (NFR), also known as quality requirements, frequently express constraints to be imposed on functional requirements as well as general quality properties that will apply to software. Examples of NFRs are Privacy, Usability, Reliability, Security, and Trust.

Despite many new works going from elicitation to verification and validation [1], there is still no convergence towards an amalgamated approach to deal with NFRs. NFRs are frequently fuzzy in nature, hard to identify, and it is even harder to elicit how to satisficeFootnote 1 them. Choosing solutions to cope with one particular NFR frequently introduce synergies and, most likely, conflicts with other NFRs. Identifying and modelling these interdependencies is the most challenging task for effectively eliciting NFRs.

One way of representing knowledge on satisficing NFRs is to build catalogues using Softgoal Interdependency Graphs (SIG) [2, 3], where softgoals represent NFRs and their refinements can be further refined using an And/Or decomposition approach until it reaches possible operationalizations to satisfice the softgoals.

In a previous work [4], we detail an empirical study that suggests the use of NFR catalogues helps identifying a larger and more appropriate set of NFRs that should be used in a project as well as a broader set of possible solutions to operationalize these NFRs.

Other works such as [5, 6] also conducted empirical experiments pointing out to catalogues helping to elicit NFRs in different situations. However, these works recognize that the use of catalogues faces some challenges, among them, the scalability problem. Catalogues can grow quite fast and become difficult to visualize. Another problem relies on the ability to discover conflicts among solutions to different NFRs. Typically, NFR catalogues contain very few if any conflicts clearly typified. More importantly, when dealing with catalogs for several NFRs at the same time, which is a common situation, even when these conflicts are mentioned in one catalogue, realizing the connection among these catalogs for such interdependencies is a visual task quite prone to mistakes and omissions.

To mitigate these problems, we started to investigate a framework that can semi-automate the acquisition of knowledge linking possible interdependencies (positive and negative ones) portrait in several different catalogues while at the same time providing an environment that can facilitate searches to guide requirements engineer (RE) to identify possible NFRs and operationalizations that could be needed in a project. The retrieved set of NFRs and possible operationalizations would help the RE to model all possible solutions to only then evaluate which solution will fit best the interests of this specific project. We understand that NFRs solutions will vary from one project to another and hence, the best approach is to provide as many possible solutions as possible to guide a better cost vs. benefit analysis for each project. As such, our approach does not suggest one solution. Instead, it suggests a whole set of solutions to be studied as possible solutions to guide the cost vs. benefit analysis.

We have investigated the use of ontologies and semantic web techniques to represent SIGs in a machine-readable format [7]. It aims to facilitate the reuse of the knowledge captured in SIGs. Veleda [8] introduced the requirements for the Quality Requirements (QR) Framework and its supporting tool, which aims at eliciting knowledge on how to develop software that will deliver appropriated solutions for needed NFRs.

The objective of this paper is to present the initial results of implementing the first step of the QR Framework, which focus on providing an environment that can semi-automate the acquisition of knowledge and at the same time provide mechanisms to efficiently search for the knowledge embedded in this knowledge base. We have implemented these mechanisms on the QR tool and carried out an empirical study to evaluate how well the current approach helps to identify a more comprehensive set of possible NFRs and its operationalizations and correlations. Correlations model the impact, either positive or negative, of one NFR solution into other NFR solution [2], and it is one of the most challenging goals of eliciting and negotiating NFRs satisficing. Initial results suggest that participants using the QR tool were able to identify a much broader set of possible operationalizations, but perhaps more importantly, they were also able to identify a significantly higher number of possible conflicts.

The paper is structured as follow: Sect. 2 tackles some of the related work while Sect. 3 describes the QR Framework and details the QR tool which is central to support the QR Framework. Section 4 depicts the experiment used to validate our hypotheses followed by a discussion of the results. Section 5 concludes this work.

2 Related Work

SIGs were introduced by the NFR Framework [2] and have been used to describe quality attributes. Many works have used SIGs to represent knowledge related to NFRs such as [2, 9, 10]. Empirical works have suggested that the use of catalogues have a positive impact on the quality of the developed software [4, 6]. However, catalogues do not scale efficiently [5, 6] hence, the larger the amount of knowledge we gather the more difficult it is to reuse it.

On a different path, Sancho et al. [11] suggested to use an ontological database and exemplified its use with the Software Performance Engineering Body of Knowledge. Their proposal consists of two ontologies both written in OWL [12]: the NFR ontology and the SIG ontology. The NFR Ontology describes the NFRs concept and relationships among them. The SIG Ontology depicts SIG constructs and their relationships. We have identified two shortcomings of this ontology. First, the SIG ontology does not define any class to describe the correlation interdependency between softgoals. Second, it does not enforce the use of the proper kind of softgoals as parent and offspring of each refinement. A few other semantic-based approaches have been proposed in the NFR context. ElicitO [13] is an ontology-based tool that supports NFRs elicitation; it provides a knowledge base about NFR requirements and associated metrics. Hu [14] proposed an ontology model as a baseline for modeling NFRs.

However, these works do not facilitate the required reasoning for a RE to analyze the tradeoffs involving different NFRs. Neither have they tackled the important aspect of creating methods to retrieve knowledge using searches with varying levels of granularity and filters to help visualize the retrieved knowledge. The QR Framework aims at mitigating some of these gaps. The following sections will illustrate how the QR Framework proposes to tackle these issues. More specifically we will focus on the first part of the framework targeting the elicitation and reasoning of NFRs solutions for a given project.

3 The Core of the QR Framework

In this work, we present an initial approach to providing the QR Framework with basic capabilities that will form the core of the framework, i.e., instruments to facilitate the elicitation of NFRs. In order to facilitate NFR elicitation, we departure from the idea that using Catalogues indeed help eliciting NFRs but do present a challenge in dealing with the acquisition of knowledge to create these catalogues and moreover, how to store and represent its knowledge in such a way that retrieving all pertinent information can be done efficiently [4,5,6].

The QR Framework was introduced in [8] and allows storage and retrieval of as many as possible options for satisficing NFRs, together with the consequences of choosing one option over another and its effects in other NFRs. The QR Framework is centered on the use of ontologies and semantic web techniques to facilitate reuse. The use of techniques such as RDF schemas [15] and SPARQL [16] aims at exploring the possibility of having rules that can help to identify interdependencies among operationalizations that could easily go unnoticed otherwise. It may also facilitate the ability to explore different levels of granularity in queries to retrieve knowledge [8].

Ontologies are frequently associated with description logic and represented with languages like OWL [12] which provides a vocabulary to describe classes, their properties, relations among them and cardinalities. The NDR Ontology [7] maps the NFR Framework [2] allowing us to describe well-formed SIGs in a machine-readable format allowing software engineers to explore the knowledge embedded in SIGs together with any rationale associated with possible design decisions.

However, we need to provide an environment that can hide all the details of handling ontologies concepts and querying it. A prototype of the QR tool is presented in this paper to provide such environment handling domain-independent catalogues of NFRs together with the interdependencies among these catalogues. As a future work, it will be extended to allow these catalogues to be instantiated into domain-specific alternatives. It is important to note that the QR Framework does not intend to propose one single solution (operationalization) to satisfice one or more NFRs. It aims at recovering as many as possible alternatives so developers can choose the one that fits better to their project. For example, the use of contextual metadata may hurt privacy expectations in the healthcare domain but may play an essential role in self-driving cars where the expectation of privacy would be much lower and less important.

3.1 The QR Framework Architecture

The QR Framework is first mentioned in [8]. In this present work, we introduce the first version of the QR tool that includes the acquisition of data from several SIGs and demonstrate the many possibilities from retrieving possible operationalizations together with their eventual correlations with other NFRs. We designed the tool to be deployed in a cloud environment. In other words, our approach implements a RESTful API, promoting a Web Service behavior. We believe that this design choice supports our application’s extensibility. Furthermore, the framework will be more capable of integrating with multiple third-party modeling applications. Additionally, by following Web Service standards, the tool may provide NFR knowledge on demand as a resource for smart applications and self-adaptive systems. Perhaps more importantly, in the future, the knowledge base will be able to be used by anyone, researchers or practitioners, interested in reusing knowledge on NFR satisficing. It may even be possible to open the knowledge base to be updated in a controlled manner by other researchers.

3.2 The QR Tool

The QR tool contains two repositories: Knowledge and Ontology. The knowledge repository stores the NFR information extracted from SIG catalogues, maintaining the knowledge evolution associated with a particular ontology model. On the other hand, the ontology repository holds different ontology models. Consequently, within this approach, we envision to provide support to multiple ontologies that are also capable of representing NFRs and design rationale knowledge, other than limit the tool operation solely to the NDR ontology. The meta-ontology used here is described in [7]. To fulfill its functioning and achieve needed requirements, the QR tool incorporates multiple technologies. We mainly adopted alternatives that have an open-source implementation and active community support. Further details can be found in [8].

NFR Knowledge Acquisition.

The QR tool acquires NFR information through a three-fold process: (i) Knowledge Extraction (ii) Knowledge Conversion and (iii) Ontology Update.

To better emphasize and describe each knowledge acquisition steps, we propose the following scenario: a SIG catalogue containing information about Transparency being imported into our platform.

Once the SIG catalogue is uploaded into the platform, the QR tool performs a series of actions to interpret and extract NFR information provided within the model. Firstly, it parses each XML element and transforms them into memory-based candidates for new NDR ontology individuals.

Subsequently, the platform verifies if each recently created candidate already exists in the knowledge repository. If a candidate is already persisted as an individual of NFR knowledge, the tool disregards it and utilizes the already existent instance from the knowledge repository. This candidate verification occurs iteratively throughout the knowledge extraction step.

After verifying the status of each candidate, the tool converts the information extracted from the original representation to OWL format, complying with the model proposed by the NDR ontology [7]. In other words, at this point, the platform converts the verified new candidates into fresh ontology individuals and references linked to the already existent ones within the NDR ontology during the knowledge conversion step. For each new reference for already existent information, the tool relies on inference techniques provided using ontologies to identify and suggest possible interdependencies among the ontology individuals. Therefore, the knowledge extraction and conversion steps adequately express how the QR tool practices knowledge evolution through the reutilization of NFR information already existent in the knowledge repository. The reason for applying such knowledge evolution originates from the constraints of the NFR Framework [2]. Since the NDR ontology follows the NFR Framework definitions, our approach enforces its restrictions during the knowledge acquisition phase.

Within the Semantic Web, Inferences are meant to discover new relationships among diverse resources [17]. Therefore, the tool leverages this Semantic Web-native feature during the knowledge extraction process to identify interdependencies between several SIG catalogues automatically. As a result, when new NFR information is imported into the knowledge repository, the QR tool would assume that when a decomposition or operationalization is described in more than one catalogue, a potential correlation between the primary NFRs described in each catalogue should be defined.

Identifying these interdependencies is one of the most significant challenges REs face when evaluating how one solution for one specific NFR might affect other NFRs. For example, in a catalog for Privacy that is already stored in the knowledge base, some of the softgoals in Privacy have operationalizations that are also used in the Transparency catalog. For example, while Anonymity may help to operationalize Privacy, it would hurt Completeness, Clarity, and Operability regarding Transparency. By the same token, while operationalizations such as Data Collection, Cloud and Exposure to Personal Information could help to operationalize Transparency, it would hurt Privacy. In that way, when the RE is searching for solutions to implement Privacy concerns he/she will be pointed out to look at Transparency issues and vice versa.

It is true that linking NFRs during the importation process still relies on the matching of terms used to naming softgoals and operationalizations. We have started to investigate how to overcome this problem using synonymies and dictionaries among other techniques, but this is future work. However, even as it stands right now, we were able to merge catalogs developed by at least four different people (two associate professors and two master students from different universities that have worked independently for generating these catalogs) and yet finding many matches linking the SIGs, which suggests the mechanism is useful even with the current limitation. A near-match process to suggest possible missed correlation is being developed to start tackling this issue.

Finally, after handling the knowledge extraction and conversion, the QR tool updates the target ontology with the new information. This step occurs directly at the SPARQL server and is defined in an OWL level.

Searching and Visualizing NFR Knowledge.

The QR tool provides two main possibilities for searching specific NFR Knowledge across the repository: (i) Selection of one specific NFR and (ii) Input of a custom search term.

Suppose our knowledge base has acquired the knowledge for the NFRs of Security, Privacy, Usability, Traceability, and Transparency. One could select one specific NFR by using the dropdown menu option “Choose an NFR” as it can be seen in Fig. 2. After selecting one of them, the tool will then display the SIG that represents the chosen NFR together with a list of correlated catalogues. Based on the constraints of the NFR Framework, the tool uses two distinct colors to reason about the nature of an association between two elements. Therefore, it applies the Green color when an association has a positive nature and employs the Red color when an association has a negative nature. For instance, a decomposition that may help to satisfice a given softgoal is highlighted in green color. On the other hand, a correlation that may hurt satisficing a particular softgoal is represented using red color.

As a consequence of demonstrating the occurrences of a queried element, the QR Tool may infer existent correlations when a parent NFR is utilized as a query term. After searching for Usability, the tool graphically outputs the complete existent knowledge associated with Usability showing subgoals such as Conduct Usability Testing, Provide Appropriate Level of Information and Provide Appropriate amount of Information. Figure 1 illustrates the resulting partial SIG after using the “zoom in” capability provided by the + and − buttons in the upper left part of the Figure (Print Screen). Correlations with other SIGs can be seen on the upper part of the screen where one ca0n read “See its usage also in Privacy, Security, and Transparency”.

Fig. 1.
figure 1

QR tool: partial SIG for usability and some correlations

Full size image

The partial SIG illustrated in Fig. 1 shows for example that Conduct Usability Testing helps to satisfice the subgoals Provide Appropriate Level of Information and Provide Appropriate amount of Information but hurts Performance concerns as well as Cost. On the other Hand, while Provide Appropriate Level of Information and Provide Appropriate amount of Information helps to satisfice Usability it also presents a possible conflict with the Security SIG. Performance does not appear together with the other three NFRs on the list of correlations (“See its usage also in”) because at this point it is not a SIG in our knowledge base yet.

To explore different levels of granularity regarding how solutions are developed and modeled, the tool provides the capability to use a custom search term. Typing one or more words in a search box will trigger the tool to look for any element that contains the term provided. The QR tool performs a non-restrictive search for custom terms. Therefore, it will produce a list with all the occurrences of this term in different SIGs.

Figure 2 illustrates this mentioned scenario. Let us suppose one wants to investigate where disclosure of data may be an issue. Filling up the search box with the term “disclosure” will trigger the tool to produce a series of SPARQL queries. These queries are then executed on the SPARQL server side in a recursive manner. Hence, the system can retrieve every interdependency associated with each element related to the search. Figure 3 depicts an example of two SPARQL queries: the former is produced and applied by the tool during the search term process, and the latter is employed just before the tool begins with the drawing process. As a result, the system outputs a partial graphic visualization starting from the chosen search possibility, in this case, five possible operationalizations as demonstrated in Fig. 2. If necessary, the user can still select among the subgoals/operationalizations associated with the term “disclosure” one of the five possible operationalizations to be further explored. Figure 4 illustrates the partial graphic output originated from clicking on the Reduce Need For Personal Data Disclosure.

Fig. 2.
figure 2

QR tool: non-restrictive search – disclosure

Full size image
Fig. 3.
figure 3

QR tool: non-restrictive search – SPARQL queries

Full size image
Fig. 4.
figure 4

QR tool: partial graphic output resulted from search – disclosure

Full size image

The search capability provided by the QR tool works on different levels of granularity. In other words, a RE can search for any NFR related capabilities (subgoals or operationalizations), ranging from early refinements to a particular refinement level. This characteristic provides a versatile mechanism for scenarios where the granular level of a needed NFR related solution is unknown.

Any correlations are displayed using a type of hyperlink between solution patterns where the RE can navigate through those patterns. Put that together with the non-restricted search capability as well as the search for specific NFRs and REs have a powerful and flexible search mechanism at hand. It helps the scalability problem allowing one to search for one specific NFR and yet be directed to look where a specific solution will hurt or help another solution either for this NFR or another.

By the same token, the search by substring where the tool searches the complete knowledge base for all the occurrences of this substring helps to find information in a vast SIG as well as looking at many different SIGs at once. We believe that such kind of tool facilitates the reuse of solutions patterns previous elicited helping REs to evaluate many different possible alternatives to comply with stakeholder’s demands.

We understand that each project will face different challenges, eventually addressing different types of users and therefore stakeholders, in general, may need different solutions for different domains and even for different companies in the same domain. That is the main reason why our SIGs do not evaluate which alternative would be better to use nor do we weigh different solutions against each other. We aim at providing a general approach that can be used in various projects regardless of the type of development process being used or the domain in which it is being used.

4 Evaluation

To evaluate how the QR tool can aid REs to elicit NFRs better, we conducted a controlled experiment involving a software development scenario. Our primary goal was to establish a reasonable comparison between the results obtained by participants divided into two groups: Group using individual NFR Catalogues and Groups using the QR tool. Due to space constraint, we will limit our explanation regarding the evaluation to a minimum necessary to its understanding.

4.1 Hypotheses

The observed measures associated with the assessment of the experiment were the number of correctly identified Operationalizations and Correlations. Thus, our experimentation was based on the following formal hypotheses:

  • H0oper: There is no difference between the QR tool and NFR Catalogs regarding the identification of Operationalizations

  • H1oper: There is a significant difference between the QR tool and NFR Catalogues regarding the identification of Operationalizations

  • H0corr: There is no difference between the QR tool and NFR Catalogues regarding the identification of Correlations

  • H1corr: There is a significant difference between the QR tool and NFR Catalogues regarding the identification of Correlations

4.2 Methodology

To properly test our formal hypotheses, we designed an empirical evaluation with participants randomly assigned to a control group (Group using individual NFR Catalogues) and to an experimental group (Group using QR tool). Participants within the control group could only rely on pure printed NFR Catalogues as a knowledge-assistance method for eliciting and modeling NFRs. Pure NFR Catalogues were represented in a SIG format and defined as static images. On the other hand, members of the experimental group used the QR tool as a knowledge-assistance technique for eliciting and modeling NFRs. Moreover, both knowledge-assistance techniques covered the same amount of domain-free NFR Knowledge. Both had the same NFRs and the same set of operationalizations. The key differences were related to how each group could access and explore a needed NFR Knowledge. Additionally, both groups had to elicit and model NFRs following an identical software development context.

The participants targeted for this study were 4th-year undergraduate students attending a Requirements Management course from the Information Technology program at York University, Toronto - Canada. Their participation was on a voluntary basis, and each participant attended twelve non-consecutive hours of training provided in a workshop manner before performing the experiment. The training workshop mainly consisted of lessons on how to model appropriate SIGs representing solutions to expected NFRs for a particular software scenario.

The expected outcome from each participant was a set of SIGs catalogues representing a complete as possible set of NFRs and operationalizations that could be used in the proposed software development scenario. The main idea was to compare each developed SIG catalogue to an authoritative control, which was produced based on the literature on SIG catalogues and represented the expected NFRs for the proposed scenario. For comparing results, we mainly focused the number of correctly identified Operationalizations and Correlations for a particular NFR. We followed the notion that the higher the number of correctly identified Operationalizations, the higher the chance of satisficing an NFR. The same assumption applies to Correlations since it expresses possible synergies and conflicts among NFRs. We repeated this evaluation for every SIG developed by every participant. Additionally, by following the idea that different projects may lead to distinct solutions for the same NFR, our goal was to evaluate whether the QR tool could help a RE to recognize a more comprehensive set of possible alternatives for satisficing a particular NFR together with the possible implications for each possibility.

The set of SIGs developed for the authoritative controls followed the same idea and contained all possible Operationalizations and Correlations for each possible necessary NFR applicable to the problem. It was first developed by the first author and later validated by the second author.

To measure the similarity between a participant’s SIG catalogue and the authoritative set of SIG models, we counted the number of correctly identified Operationalizations and Correlations in the participant’s response model. An Operationalization or Correlation had to be explicitly expressed in the authoritative set of SIG catalogues to be considered accurate. Our comparison strategy also took into the account the taxonomy variations regarding the name of elements used for both Correlations and Operationalizations. We manually verified these taxonomy variations due to the subjective nature of NFRs. For instance, we manually validated that an operationalization involving “Use Desktop” in the authoritative model was correctly expressed in a participant’s SIG catalogue as “Use Computer”. Thus, similar terms can be applied to emphasize the same solution for satisficing an NFR. However, most of the time we were able to find exact matches quietly likely due to the use of the catalogues in both formats.

Additionally, students were randomly assigned to each of the two groups and our comparison was conducted in a blind evaluation manner. Hence, the comparison between the participant’s responses and the authoritative control was performed in a random approach, disregarding the group information about the target participant.

4.3 Proposed Scenario

To bring this study as close to reality as possible, we proposed a hypothetical software development scenario involving a system for an autonomous taxi service company. The idea originated from the current discussions regarding autonomous systems in our present society and intended to trigger the participant’s interest, demanding a significant number of NFRs to be elicited. At the same time, it intended to use a non-trivial problem in which we would find a large number of NFRs and possible Operationalizations and Correlations. Finally, it also intended to minimize the influence of participants using their own expertise instead of the guidance of the catalogues.

Participants were asked to model a set of SIGs representing the expected NFRs and their Operationalizations and Correlations. We provided full documentation regarding the demanded system to each participant, including a Requirements Table characterizing every needed Functional Requirement. We also provided documentation based on Use Case diagrams, Class diagrams, and Sequence diagrams.

The identification of NFRs by the participants was part of the experiment. In other words, our documentation did not provide information about NFRs. Each participant had to elicit as many NFRs as he/she thought was necessary, relying on their own interpretation. Our authoritative control highlighted Privacy, Security, Traceability, Transparency, Usability, and Performance NFRs since those were the catalogues students were given to work with. The whole set of expected NFRs within the authoritative control featured a total number of 52 Operationalizations and 28 Correlations. Figure 5 demonstrates an example of the authoritative SIG depicting solutions for Security. It is noteworthy to mention that the goal of the SIGs composing the authoritative control was to represent the largest set of alternatives that could be used under the target domain.

Fig. 5.
figure 5

Authoritative control: security SIG

Full size image

4.4 Analysis

As a result of our evaluation, we were able to collect a sample composed of 18 participants. Half of the participants used the QR tool as a knowledge-assistance technique, while the other half performed the experiment using pure NFR Catalogues as a knowledge-assistance method. The data collected from each participant’s response was essentially the percentage of Operationalizations and Correlations correctly identified. For example, if a particular participant found 8 correct Operationalizations out of the total of 52, we considered 15.38% as the quantified data for our analysis. The same logic was applied to the total of Correlations correctly identified.

Correspondingly, both groups define a population positively skewed. The central tendency measure indicates that the average of results from the Group with QR tool (Median = 36.54%) is significantly higher than the one from the Group with NFR Catalogues (Median = 13.46%). Additionally, the Group with QR tool demonstrated a higher variability among the obtained results in comparison to the Group with NFR Catalogues. Moreover, the distribution of results from the Group with NFR Catalogues concentrates towards the minimum possible value. This observation suggests that most of the participants in Group with NFR Catalogues identified a low percentage of operationalizations in comparison to Group with QR tool.

When analyzing both groups regarding the percentage of correctly identified correlations, both groups presented a positively skewed dataset. Also, the central tendency measure shows that the average for the Group with QR tool (Median = 25%) is significantly greater than the average presented by the Group with NFR Catalogue (Median = 3.57%), which seems to have the same value as the first quartile value within the dataset. Furthermore, the Group with QR tool demonstrated a higher variability among the obtained results. Additionally, in this comparison, both groups demonstrated a distribution more concentrated towards the minimum value from each dataset. However, it is noticeable that participants within the Group with QR tool could still elicit a high percentage of correlations since their minimum presented value (17.86%) can be considered as a medium-high percentage value in comparison to the results obtained from the Group with NFR Catalogues.

To test our formal hypotheses, we also conducted an inferential statistical analysis. We carried out a Mann-Whitney U test for both Hypotheses. The results indicate that for the first hypothesis (H1oper), the null hypothesis (H0oper) should be rejected (2-tailed sigma of 0.015). Consequently, we can confirm our first hypothesis: There is a significant difference between the QR tool and NFR Catalogues regarding the identification of operationalizations. For the second hypothesis (H1corr), the results also indicate that the null hypothesis (H0corr) should be rejected (2-tailed sigma of 0.002). Consequently, we assume the confirmation of our second hypothesis: There is a significant difference between the QR tool and NFR Catalogues regarding the identification of correlations. Also, in both comparisons, the Group with QR tool presented a significantly higher mean rank value than Group with NFR Catalogues.

4.5 Threats to Validity

From the perspective of Internal Validity, Maturation was not considered a threat to our study since each participant had to perform the tasks for the experiment only once. As for Instrumentation, we controlled this threat by providing identical conditions for every subject regarding tools and execution time. Also, we designed the experiment in a similar manner to an optional academic course assignment. Biased subject selection prevailed partially uncontrolled in our study since each target subject consisted of an undergraduate student attending a Requirements Management course. However, among this population, we randomly choose who was going to participate in each group from the available participants. To mitigate Experimental mortality, we provided the entire essential material through a web-portal, and we let each subject perform the experiment in their timely manner until a specific due date.

Regarding External Validity, there is the issue of interactions between selection biases and the independent variable: This threat remained uncontrolled due to our target population in this study. We acknowledge that using only students may raise doubts on whether results could be extrapolated to experienced REs. However, studies such as the one from Salman et al. [18] show that, as well as in other studies, when applying new technology students results do not statistically differ from professional results. Recently, editors of the Empirical Software Engineering engaged in discussion with other authors and the authors in Salman et al. discussed the validity of using students [19]. Several arguments are made both in favor of using students and the preference for using professionals. However, it is acknowledged the use of professionals can be prohibitive, and in many situations, the use of students can lead to equivalent results.

Our results suggest that the QR tool may facilitate the reuse of NFR Knowledge and therefore contribute to elicit a more comprehensive set of possible NFRs. Given the statistical outcome, we believe that the features provided by the QR tool offered convenience and helped participants to navigate through the available NFR Knowledge in a more efficient way. Among these features we emphasize: (i) The capability to navigate from one NFR to another, (ii) The ability to query one particular NFR and retrieve it operationalizations together with possible correlations, (iii) The non-restrictive search capability that covers different granularity levels regarding NFR operationalizations and interdependencies and (iv) The provided inference mechanism for identifying occurrences of a particular element across the knowledge repository, leading to correlations.

We also consider that these features assisted the participants in identifying more possible synergies and conflicts regarding an element involved in satisficing an NFR. For instance, a participant wondering whether Transparency correlates with another NFR could easily use the tool to search for “Transparency” and quickly see its occurrence across the Knowledge Repository. At this point, a participant would also be able to verify every correlation associated with Transparency according to the NFR Knowledge in the repository.

Regarding the sample size of the experiment, we understand that a larger dataset would produce more accurate results regarding both groups. We aim to reproduce this study in the future with a bigger population. However, we believe that the number of participants is enough to support our conclusions.

We also understand that as pointed out by Gramatica [6] the use of catalogues may differ for expert users when compared to novice users and in our experiment, we only used novice participants. Nevertheless, we believe that both would gain from using our framework.

Additionally, it is important to mention the current limitations of our approach. At this moment, we consider the tool’s Knowledge Acquisition feature as a semi-automated process. Currently, the platform assumes that every SIG provided as a knowledge input contains substantial NFR information. Therefore, the QR tool still relies on a manual knowledge validation as pre-step for the knowledge acquisition phase. We aim to address this challenge by applying the use of a custom and automated ontology reasoner during the importation of NFR Knowledge into the repository in the future to facilitate this semi-automated process.

As mentioned before, the use of catalogues brings the question of the expressions used to name both NFRs and operationalizations. We are currently researching mechanisms such as the use of lexicons, thesaurus ad non-restrictive searches to mitigate this problem. Nevertheless, we believe that despite the current limitation the use of our approach already presents relevant results.

Although we have initially tackled some issues exploring the ability to use different levels of granularity to search the knowledge base, we understand that we still must improve it.

Lastly, the current version of the QR tool is designed to handle domain-free NFR Knowledge only. However, the NDR Ontology already envisions domain-specific knowledge, and we believe that the current implementation of the NDR needs minimal improvement to support domain-specific NFR Knowledge. Nevertheless, extensive tests and validations are still required to be performed. Therefore, we believe that the next version of the QR tool will include the support of domain-specific NFR Knowledge.

In an overall manner, we consider that our approach can be highlighted as an alternative for dealing and facilitating the reuse of NFR Knowledge during the early phases of the software development lifecycle. We believe that our work arises as an important contribution to the relatively few alternatives for promoting the reuse of NFR Knowledge. On a longer perspective, we will investigate how to facilitate the integration of possible solutions into models depicting the whole solution for the problem.

5 Conclusions

This paper introduces a Framework to promote the elicitation and modelling of NFRs. At this point, we present the initial approach to this framework focusing on the elicitation part. We introduce the QR tool as a support mechanism based on a previously defined ontology [7] to support the acquisition of knowledge to satisfice NFRs while allowing queries to be made on this knowledge base. The tool supports a semi-automated process of knowledge acquisition linking related SIGs. It also provides a retrieval approach that allows the requirements engineering to query the existent knowledge base using different levels of granularity as well as different approaches to search and visualize results.

We carried out a controlled experiment to evaluate how the use of the tool would help to find solutions to address needed NFRs. The experiment suggests that not only we could identify more NFRs, but perhaps more importantly, we could identify a significantly higher number of possible conflicts and synergies among possible solutions.

Future work will involve replicating this experiment with other groups to further evaluate the use of the tool and to identify possible angles to amplify the ability to provide different levels of granularities for searching solutions. We also aim at investigating mechanisms to tackle the problem of expressing one NFR or operationalization using different names/expressions. On a longer perspective, we will investigate how to facilitate the integration of possible solutions into models depicting the whole solution for the problem.