1 Introduction

Agile methods have been used in Software Engineering as an alternative to tackle high rates of failure and project delays [1]. To improve software usability quality, new approaches have been studied to integrate User-Centered Design and Agile Development.

Previous studies [2–5] discuss approaches that aims to integrate usability evaluations on agile software development. In a first approach, Sy uses the parallel tracks of development technique to ensure that those responsible for implementation receive a prototype of interfaces with usability already evaluated.

In a second approach, Sy adopts the prototyping interfaces technique to refine interaction projects. Both approaches are also observed by Silva da Silva.

Crispin and Gregory use prototype evaluation methods to help teams to understand every system requirement, before starting source-code implementations and usability evaluation with users, to observe them in real use cases of the system.

Lars Nielsen uses heuristic evaluation to verify developed features implementation in each iteration.

This work presents the results obtained from a master degree research aiming to investigate, implement, and apply a method for usability evaluations on agile software development with Scrum framework [5], mainly to improve product quality and user satisfaction.

The proposed method, named AGILUS, was created based on the Agile Manifesto principles and values [7]. It uses usability evaluation methods and user-centered design techniques, both integrated to Scrum framework development process. In order to verify the AGILUS effectiveness, two case studies were performed, as well as some analyses and discussions.

Section 2 delimits the proposed method application scope on the agile development main stages. Section 3 describes the AGILUS mainly steps, pointing the objectives of each usability evaluation. Section 4 describes the use of the AGILUS method on agile development process. Section 5 describes two case studies where the method was used. Section 6 presents some analyses and discussions from method application results. Finally, conclusions are presented in Sect. 7.

2 AGILUS Method Application Scope

The AGILUS method scope starts after the Product Backlog definition and finishes when the Product Owner (PO) validates the software product increments, supported by the results of usability tests, involving representative system users.

Figure 1 highlights the AGILUS application method scope at the main agile development phases.

Fig. 1.
figure 1

The AGILUS method scope on agile development mainly phases

Full size image

Thereby, usability evaluations that may be applied at the initial phases of the Pre-Game, such as Product’s Vision, Requirements Elicitation, and Backlog’s Initial Definition are out of the proposed method scope.

The Post-Game phase activities of the agile development process, which involves usability evaluations on production environment, are also out of scope.

3 Main AGILUS Method Steps

The proposed method, named AGILUS, is composed of three main steps, with three distinct and complementary usability evaluations adapted to the agile development, as shown in Fig. 2.

Fig. 2.
figure 2

The AGILUS method steps, usability evaluations and their objectives

Full size image

The Pluralistic Walkthrough Evaluation in Prototype is responsible for the validation of Interaction Design in the initial step, where prototypes of interfaces are designed. We adopt this formative and empirical method to involve different stakeholders, including representative users in the evaluation. Thus, different project viewpoints contribute to the interaction design validation, that will be implemented in a parallel track of development.

The evaluation based on heuristics of usability, defined in the second step of the AGILUS method, is an analytical and exploratory testing to verify if implemented product increments meet usability principles.

As a final evaluation to support the validation of product increments, the Test with Users, defined in the third and final step of the proposed method, aims to measure the quality of usability as its efficiency, effectiveness, and user satisfaction.

4 AGILUS Method on Agile Development Process

This section aims to increase the understanding of the AGILUS method by identifying its steps on agile development with Scrum framework.

Figure 3 presents an overview of the agile development process where Scrum framework events were separated from the AGILUS method steps.

Fig. 3.
figure 3

The AGILUS method on agile development process with scrum

Full size image

The first step of the AGILUS Method is the application of the Pluralistic Walkthrough Evaluation in Prototype. This evaluation must occur in at least one iteration before the implementation of a priority requirement, using the parallel tracks development technique of interaction design and implementation.

At the beginning of the project, the Pluralistic Walkthrough Evaluation should occur even in the Pre-Game Phase, after the initial prioritization of the Product Backlog.

The User-Interface Prototyping and the Pluralistic Walkthrough Evaluation may occur in more than one cycle, refining the prototype until the Interaction Design is approved by the PO.

The approved prototype goes to the next Sprint Backlog, to be developed during the Sprint. As the developers begin coding the requirements, part of the team, responsible for the Interaction Design, starts in parallel, the prototyping of priority requirements interfaces, which are candidates to be implemented in the next Sprint, if any.

After code implementation, on the second step of the AGILUS method, the Implementation is verified by Heuristic Evaluation, before the end of the Sprint, as part of the definition of “Done”.

Usability problems identified from Heuristic Evaluation should be resolved in the remaining time of the Sprint and before the Sprint Review. If the time for solutions implementation is insufficient, within the Sprint time, new usability requirements should be added to the Product Backlog, to be prioritized by the PO and, if possible, implemented in the next Sprint.

Likewise, sequentially, the following Sprints receive interfaces approved prototype, developed in the previous Sprint and, after its implementation, performs a Heuristic Evaluation.

After some Sprints or the development of a critical requirement for the project success, the PO must request a usability evaluation by Test with Users, as the third method step. In this evaluation to assess whether any usability problem was identified, one should measure the efficiency, effectiveness, and user satisfaction to evaluate the quality of the software usability.

As shown in Sect. 3, the Post-Game Phase is out of the AGILUS application scope.

5 Case Studies

To verify the usability quality of the developed software with the application of the AGILUS method, two case studies were conducted: A Socio-pedagogical Monitoring System (SMS); and an Event Management System (EMS), both developed for an educational institution.

These projects have been developed as follows: two weeks for the initial phase, two iterations of three weeks for development, and other two weeks for Tests with Users, involving six developers as volunteers from each team and twenty users also as volunteers being tested from each project.

To measure the usability of the developed systems, the following metrics were used: tasks runtime to measure efficiency; error and completion rate to measure effectiveness; and the System Usability Scale (SUS) [8] to measure user satisfaction.

Systems development were performed, as illustrated in Fig. 3. In each system, six tasks were set to perform the Test with Users in the final step of AGILUS method. For each task, it was measured its task runtime, error and completion rate, and, at the end of the test, the users were invited to answer the SUS questionnaire.

6 Analyses and Discussion of Results

In the Pluralistic Walkthrough Evaluation in Prototype step, significant results on improving projects usability were achieved:

  • The diversity of viewpoints and opinions contributing to discussions;

  • The product’s vision maturation;

  • The improvements on interaction design; and

  • During discussions, the new user needs raising from those involved, being transformed into requirements. Therefore, the evaluation also has contributed to the new requirements elicitation.

The main detected difficulties on implementing the Pluralistic Walkthrough were:

  • To find available schedule among the participants to undertake evaluations;

  • The reduced number of tasks and the simplicity of user interfaces, in one evaluation have generated few improvements, becoming unproductive, considering the effort to bring stakeholders together;

  • Discussions often have dispersed the task scope and demanded the moderator intervention to focus on its objectives; and

  • Some users had to be encouraged by the moderator, during the test, in order to contribute with their opinions.

Considering the benefits and difficulties presented in this first step of the proposed method, it can be inferred that, whenever possible, the maximum of prioritized User Stories must be included in a Pluralistic Walkthrough Evaluation in Prototype without exceeding meeting time. Thus, a reduction in the number of meetings involving stakeholders has contributed to increase the AGILUS method efficiency.

The Heuristic Evaluation, the second AGILUS method step, was the evaluation that most allowed to identify usability problems with respect to: effort; number of involved persons; and evaluation time.

The main benefits noticed from this step highlighted:

  • The deep exploration of software functionalities;

  • The verification of several errors possibilities and system status feedbacks; and

  • Standards consistency.

The main difficulties found were dependencies of expertise, skill, and experience from evaluators.

As the main empirical method for usability evaluation of the AGILUS method, the Test with Users, applied in the third step of each project at the end of the Game phase, aimed to measure the quality of usability and to identify problems by observing real system users performing software tasks.

Figure 4 shows the obtained results from the two case studies with average values of: 85 % and 84 % of efficiency, based on task runtime; 92 % and 85 % of effectiveness, based on the error rate; 98 % and 99 % of effectiveness, based on the completion rate; and 86 and 88 of SUS scores, for user satisfaction. These SUS scores results were considered “excellent”, according to the grid interpretation from Sauro [9].

Fig. 4.
figure 4

The two case studies main results

Full size image

7 Conclusion

Regarding usability quality and user satisfaction, from the two case studies, where the AGILUS method was applied, some significant results were obtained.

The main difficulties found from those two case studies were related to the application of the Pluralistic Walkthrough in Prototype method considering: the availability of those involved participants to arrange meetings; and the moderator need, to keep the participants focused on discussions of task goals, and to encourage them to give opinions during the test.

Depending on the achieved results with the application of the AGILUS method, the authors of this research believe that the proposed method can be successfully applied also in other knowledge domains involving the agile software development.

The authors also believe that, with few adaptations, the AGILUS method can be adopted by other agile development methods, besides the Scrum framework, providing software usability quality and user satisfaction.