Eliciting user requirements using Appreciative inquiry

Abstract

For decades and still today, software development projects have failed because they do not meet the needs of users, are over-budget, and abandoned. To help address this problem, the user requirements elicitation process was modified based on principles of Appreciative Inquiry. Appreciative Inquiry, commonly used in organizational development, aims to build organizations, processes, or systems based on success stories using a hopeful vision for an ideal future. In four studies, Appreciative Inquiry was evaluated for its effectiveness with eliciting user requirements in different circumstances. In the first two studies, it was compared with brainstorming, a traditional approach, with end-users (study 1) and proxy-users (study 2). The third study was a quasi-experiment comparing the use of Appreciative Inquiry in different phases in the software development cycle (study 3). The final (fourth) study combined all lessons learned using Appreciative Inquiry in a cross-case comparison study of 3 cases to gain additional understanding of the requirements gathered during various project phases. In each of the four studies, the requirements gathered, developer and user attitudes, and the Appreciative Inquiry process itself were evaluated. Requirements were evaluated for their quantity and type regardless of whether they were implemented or not. Attitudes were evaluated for commitment to the requirements and project using process feedback. The Appreciative Inquiry process was evaluated with differing groups, projects, and project phases to determine how and when it is best applied. Potentially interceding factors were also evaluated including: team effectiveness, Emotional Intelligence, and perceived stress. Appreciative Inquiry produced positive results for the participants, the requirements obtained, and the general requirements eliciting-process. Appreciative Inquiry demonstrated benefits to the requirements gathered by increasing the number of unique requirements as well as identifying more quality-based (non-functional) and forward-looking requirements. It worked best when there was time for participants to reflect on the thought-provoking questions and when the facilitator was knowledgeable of the subject-matter and had extra time to extract and translate the requirements. The participants (end-users and developers) expressed improved project understanding. End-users participated consistently with immediate buy-in and enthusiasm, especially those users who were technically-inhibited. We conclude that Appreciative Inquiry can augment existing methods by presenting a positive and future aspect for a proposed system resulting in improved user requirements.

Appendix

1.1 Appendix A: Participant Information Survey

Participant Information

1. Project Role: Technical Team Member? User/Company Representative

2. Gender: Male? Female?

3. Highest degree completed: High school? Bachelors? Masters? Doctorate?

4. I have participated in a software development project (check all that apply): a developer? to provide user requirements? in another role (e.g., project manager)?

5. Personal computer ability and expertise. (A scake if 1 to 5 was used with 1 as “need a lot of help using a computer” and 5 being “use a computer with no help from others.”)

6. I have a professional technical certification in an IS-related discipline (e.g., GISP, ICCP, CCNP, CISSP): Yes? No?

1.2 Appendix B: Participant and Process Feedback Survey

TEAM FOCUS

1. My team was focused with more providing opportunities and possibilities with a positive outlook as opposed to fixing problems

PROJECT FEEDBACK1

2. I am satisfied with our current prototype and/or identified requirements.

3. I am confident that the current prototype and/or identified requirements will satisfy the client.

APPRECIATIVE SESSION

4. The Appreciative session was effective in identifying requirements.

5. The Appreciative session was effective in identifying additional/new requirements that the team/client would not have otherwise identified.

6. The Appreciative session was effective in identifying different types of requirements that the team/client would not have otherwise identified.

7. I feel confident that I can apply Appreciative techniques in a JAD session with a client.

8. I considered the future when identifying requirements

9. I considered ___________ months or years (select one) into the future when identifying requirements.

10. What is your overall feedback regarding the project’s current requirements and/or prototype?

11. What is your overall feedback regarding the use of Appreciative Inquiry for identifying requirements?

1.3 Appendix C: Brainstorming Questions

Sample Brainstorming Questions

1. What is the goal for our online community? What do we want to accomplish? What benefits do we want to provide?

2. Who are our target audience/users in our online community?

3. Are there any other audiences/users of our online community?

4. What useful features do we want available in our online community

5. What tasks do we want to provide in our online community?

6. What types of users do we want available in our online community?

7. What rules or protocols do we want in our online community?

8. What security do we want in our online community?

9. What expectations do we have for the availability & reliability of our online community? What would be the critical times for availability?

10. Do we have any back-up or contingency requirements?

11. Are there any compliance or regulatory requirements that we should be aware of?

12. Is there any related marketing or communication that we want related to our online community?

13. What training should we provide or is expected for our online community?

14. Do we have a tag-line for our online community? Do we want to declare an “identity” for our online community?

1.4 Appendix D: Project Team Report Samples

All project team reports contained similar content. Figure 9 shows a sample of a Table of Contents from a project team report. Figure 10 shows a sample of the requirements matrix required for all reports.

Fig. 9

Table of contents sample

Fig. 10

Project requirements matrix sample

1.5 Appendix E: End-User Case Study (Study 1)

Participants

Four teachers participated in three separate interviews.

Methodology and Procedure

The participants met with the researcher, either in-person or via phone, and were verbally given a simple description of an online user community in terms of how it can support the developed PDA communication software application (Pixtalk). The teachers were then interviewed in a traditional manner using 14 brainstorming questions to gather requirements (first interview). Each interview started with an explanation of the proposed online user community, possible features, functionality and resources, e.g., discussion boards to interact with others using traditional brainstorming questions used for eliciting user requirements as shown in Appendix C. Once the responses were received, the participants were introduced to the concept of Appreciative Inquiry, using the general procedure described above, and interviewed again by phone (second interview). The Appreciative Inquiry interview was conducted 1–2 weeks later to give them time to reflect on the Appreciative Inquiry questions. At the interview, Appreciative Inquiry was re-explained, questions re-presented and detailed responses recorded. Brainstorming questions were not repeated since different responses were not expected since the participants had an existing positive open relationship with the researchers. All interviews were transcribed and answers and attitudes compared with those from the traditional approach. A complete list of the brainstorming questions is provided in Appendix C.

Measurements

We measured the requirements gathered, the responses of the participants, and the effectiveness of the Appreciative Inquiry process itself. Requirements and participant attitudes were compared between Appreciative Inquiry and the direct questions. Measurements were obtained via transcription by the researcher.

1.6 Appendix F: Proxy User Controlled Experiment (Study 2)

Participants

25 students participated. They were university Information System (IS) Master Degree and Doctoral students. The participants were invited via the department listserv. The context of the experiment was described as developing requirements for “a ‘connected’ campus that integrates technology into course curriculum and campus life”. Participants were provided $20 for their participation.

Methodology and Procedures

There were two conditions: the Appreciative Inquiry and the brainstorming conditions. Students were randomly assigned to teams and each team to a condition. For each condition, there were three groups with the number of participants ranging from three to five participants in each group. Regardless of the requirements’ eliciting technique, participants were assigned team roles. There were three different roles: uses, business analysts, and developers. Based on their assigned role, the study participants represented different types of participants in system development projects. These roles were assigned in an effort to make the experiment realistic. ‘Users’ were described as the faculty and students at the university who would be considered users; ‘developers’ were described as those who would design and develop the systems based on the requirements captured; and ‘business analysts’ were described as the liaison among users and developers in order to elicit, analyze, communicate and validate requirements for the information system.

In both the brainstorming and Appreciative Inquiry conditions, we tested two group compositions. The first composition consisted of the users, business analysts and developers roles; the second group composition consisted of the users and business analysts roles. The two group compositions were chosen based on whether developers directly participated in requirements gathering. Group compositions were determined based on random assignment. An overview of the distribution of participants, their roles, and the respective elicitation methods is shown in Table 8.

Table 8 Distribution of participants for Study 2

Participants received a description of the information system development project to create a “connected” university at their campus. The participants were told that the outcomes of the experiment were the identified requirements and their feedback on the process.

As explained above, the Appreciative Inquiry session started with a definition of an affirmative topic for the session. The general theme was adjusted for the connected campus example:

When professors and students are successful, a student’s education allows them to realize their vision and goals; and professors are successful at research which improves practical applications and enhances the education they provide to students. Technology is a tool that facilitates the achievement of this vision and goals. It frees people from processes and methods. It greatly improves student success, supports successful relationships between the communication, practitioners, researchers and students. It supports creativity, enables quality and produces desired results.

Following the presentation of the theme and opportunity for the participants to ask questions, the researcher guided the participants through answering a basic set of Appreciative Inquiry questions:

• a. What were your hopes and dreams when you chose to attend or work at this University?

• b. Think over your educational career and all the work you have. Think about the greatest experience you have ever had in the classroom or in your university life. Think about when you felt most successful and most satisfied. Consider situations where you were helped by a colleague, fellow student, or professor. Consider situations where you helped a colleague, fellow student, or professor. Consider an important experience or event that provided unexpected opportunities or enabled you to face a difficult challenge.

• c. What was it like?

• d. What conditions contributed to that extraordinary level of success and satisfaction?

• e. What do you value most about yourself and your capabilities as a colleague, fellow student, or professor?

• f. What do you value most about yourself as a member of an organization and/or member of a team/community?

• g. What are the most important attributes that support your highest levels of success and satisfaction?

• h. What results would you want from technology in your campus life?

• i. What do you envision as an ideal technological solution in the future several years in the future for this University’s students? (When your children or your children’s children attend this University?)

The brainstorming session started with an explanation of brainstorming and an invitation to suggest requirements for the information system. As necessary to address any lull in the conversation, the researcher stimulated the conversation with traditional questions used for gathering system requirements (Appendix C).

Measurements

As with the prior study, we counted the requirements gathered, the responses of the participants, and the effectiveness of the Appreciative Inquiry process itself as transcribed by the researcher during the session and collected via survey (Appendix B). Requirements and participant attitudes were compared between Appreciative Inquiry and the brainstorming sessions.

1.7 Appendix G: Project Team Field Experiment (Study 3)

Participants

The participants were members of two student project teams within a Computer Information Systems capstone course at a California State Polytechnic University, Pomona. As part of this course, students develop software for a client and project which they have chosen from a list of projects solicited by the course professor. The course uses an “Evolutionary Prototyping” methodology that assumes that the requirements are not known at the beginning of the project and evolve as the project progresses. “Rapid Development: Taming Wild Software Schedules” (McConnell 1996) is used as the reference for this course as well as for another course in the curriculum. The course dictates the use of multiple Joint Application Development (JAD) sessions for the student teams to meet with the project clients to enhance client participation and collaboration with the development teams in developing requirements and reviewing prototypes.

There were seven participants with four in one project team and three in the other team. The first team developed a website application for a plumbing supply company and the second team developed a website for a race car import company.

Methodology and Procedures

User requirements for different phases of the system development life cycle were collected to evaluate the types of requirements and how that changes over time with and without the intervention of Appreciative Inquiry. Requirements were collected initially from the client’s original list, initial Appreciative Inquiry session (Team 1 only), their first joint application design (JAD) session, final prototype session, and the final Appreciative Inquiry session. Team 1 participated in an Appreciative Inquiry session twice: once after the client submitted their initial requirements and the second after their prototype session. Since differences in results for Team 1 can be attributed both to the timing as well as a learning effect, we organized the study so that Team 2 participated only once in an Appreciative Inquiry session after their prototype session.

For Team 1, an Appreciative Inquiry theme and questions were provided as in the previous case studies but were adjusted for the individual project. An example is shown below for the race car import company:

When customers and employees are successful, the company realizes their vision and goals for local, global and efficient import of race car parts; employees are successful at producing desired results and achieving their goals, and customers obtain their goals through meeting current and future needs. Technology is a tool that facilitates the achievement of this vision and goals. It frees people from processes and methods. It greatly improves company, customer and employee success, as well as supports successful relationships between the company, employees, and customers. It supports creativity, enables quality and produces desired results.

When the prototype was presented, the Appreciative Inquiry session was adjusted for the different time in the project cycle (initial class meeting as opposed to following the prototype session, Report 3). In the last phase, we asked the participants to not only reflect on what has worked well with this project but the group as a team as was consistent with other uses of Appreciative Inquiry with teams (Bushe and Coetzer 1995). The additional questions included some reference to the project team’s relationship as well as their role in helping the client realize the vision and goals. For example, the additional Discovery phase questions were:

Appreciate what worked well. Describe what worked well with this project. Features? Team dynamics? Client relationship? Learning Opportunities? What would you want to carry over to other projects? What was valued and worked well?

Measurements

As with the prior study, the following measures were collected: the number and type of solicited requirements, the responses of the participants, and the effectiveness of the Appreciative Inquiry process itself. Requirements and participant attitudes were compared between the different project phases (initial and final prototype): For Team #1, the Appreciative Inquiry process was conducted after the initial requirements from the client at the first class meeting and after their first prototype presentation, Report 3; For Team #2, the Appreciative Inquiry process was only conducted after their prototype presentation, Report 3. Results for each team were also evaluated to determine if additional requirements could be identified at the final Appreciative Inquiry session regardless of whether the team previously experienced an Appreciative Inquiry session.

1.8 Appendix H: Development Team—Multiple Case Study (Study 4)

Participants

Similar to the preceding study, the participants of each case study were members of student project teams within a CIS capstone course at California State Polytechnic University, Pomona. As with the previous study, the students select a development project submitted by external clients to the course professor.

This study consisted of three individual case studies based on student development teams assigned to the following projects:

• Team 1 A website for marketing and previewing acoustical panel and wall covering business for an existing business that caters to graphical designers and construction contractors.

• Team 2 An e-commerce website that provides information regarding a specific make of classic cars.

• Team 3 A website for a university violence prevention outreach program.

Each case study consisted of a team with five to six developer members and their corresponding client. All developer team-members were undergraduate male students within 3 to 6 months of receiving their degree. Each project had one client who was responsible for defining the requirements. All the clients have undergraduate degrees.

Project types varied in complexity and definition. Team #1’s and Team #2’s projects were concise and more defined by their client. Conversely, Team #3 project was presented by the client with broad, vague and unconstrained requirements. Table 9 provides an overview of the structure of the project teams.

Table 9 Project and team overview (Study 4)

Methodology and Procedures

Multiple case studies were conducted and compared with respect to evolving requirements for new additions, participant feedback, and team effectiveness as the projects’ phases evolved through the development process. For each case study, there were four points in time when requirements were evaluated by the researcher.

• 1) The initial set of requirements (Initial) presented by the client to their selected development team at their initial meeting.

• 2) The first developer-documented set of requirements (Report 1) that is submitted to the professor after the developer’s first joint application design session (JAD1) between the development team and their client.

• 3) The second developer-documented set of requirements (Report 2) that is submitted to the professor after the development team’s second joint application design session (JAD2) between the development team and their client. Report 2 is an opportunity for the development team to clarify their initial set of requirements as well as add more.

• 4) The third developer-documented set of requirements (Report 3) that is submitted to the professor after the developer’s first prototype (Proto1) is demonstrated by the developers to their client. Report 3 provides another opportunity for the development team to clarify requirements by demonstrating a system prototype. The client and development team often suggest additional requirements at this stage once the clients view this first working model of their project and the development team gains more confidence in their abilities to meet the client’s requirements.

All reports are submitted with a standardized format provided by the course professor who then provides feedback to the team prior to each report’s final submission. The reports are cumulative with additional information being added as the project continues, such as requirement changes. A sample of the report content and format, including the requirements matrix, is provided in Appendix D. The reports are submitted to the course professor and to the client at biweekly intervals throughout the progression of the course.

Appreciative Inquiry sessions were performed individually by the researcher alone with each development team after Report 2 and after Report 3. Report 1 was used as a baseline of user requirements for comparison with the other reports. The data collection for this study concluded with Report 3, the first prototype session even though the course and projects continue through to one more prototype session and a final presentation. (The teams continue their work with a second prototype session, Report 4, and the final application presentation and turnover to the user with a final report to the professor.)

The Appreciative Inquiry sessions were performed consistent with the prior studies. However, the researcher worked alone with the development teams individually to apply Appreciative Inquiry to their current project and encourage their use of the Appreciative Inquiry method with their client. The first Appreciative Inquiry session, between the research and the team, was themed for “effective teams” as a means of introducing Appreciative Inquiry to the team without direct discussion of project requirements. The second Appreciative Inquiry session was themed to address the requirements of their project. Table 10 presents an overview of the timing of the team activities, team documentation, researcher activity, and data collected.

Table 10 Overview of Study 4 activities and measures conducted for each case study