This section reports three evaluations of the VBRE method using case study and action research approaches. The first two evaluations report use of the method first on development of collaborative decision-support tools for epidemiological research, and secondly on development of tools to analyse early onset of dementia from logs of computer use. The third evaluation reports use of the method in industry, supported by the authors in an action research investigation into how the method might be adopted and adapted by others in practice. All three evaluations addressed the research questions about the usability of the VBRE method in practice, the impact of applying the method on requirements outcomes, and how the method was adapted during practice. Since performing a field experiment testing practice with and without the method was impossible given resource limitations, we concentrated on qualitative analysis of method use.
ADVISES case study
The following experiences emerged during application of the initial (paper-based) method to requirements analysis in the ADVISES project . ADVISES is a decision-support system for academic researchers and National Health Service public health analysts who investigate epidemiology problems. The two distinct stakeholder communities had different goals; for academic researchers, understanding the generic causes of childhood obesity by statistical analysis of health records was a high-level goal. In contrast, the goal of public health analysts was local health management; for example, identifying where best to target interventions, such as promotion of local sports facilities, healthy-eating campaigns. Two academic researchers (both male, age 31, 52) and seven public health analysts (4 male, 3 female, age range 27–41) were recruited through the authors’ personal contacts.
VBRE was introduced in the early stages of project, after terms of references were agreed, through to prototype development. The method was applied by the first author during interviews, scenario-storyboard requirements exploration sessions, and requirements analysis workshops. The stakeholders were encouraged to think aloud and explain their approaches during sessions observing their work practices, which were particularly useful in eliciting values and attitudes to their work. Most meetings and interviews were 60–90 min. Recordings of the various meetings and interviews were transcribed and analysed using the taxonomy and key issues.
The initial list of hunches based on values directly expressed by our stakeholders included being methodical, precise, systematic in their research, while taking a creative approach. The VBRE taxonomies were applied to the hunch list to produce preliminary requirements linked to values and stakeholders’ motivations with cross-references to individual identities and the requirements elicitation documents, e.g. interview transcripts. Requirements and values were refined during further analysis interviews when the requirements and their potential VME implications were discussed with stakeholders. This produced further insights; for example, stakeholders were concerned about their public profile and need to collaborate, though they found sharing of resources difficult, perhaps because of lack of trust.
Following several iterations of VBRE analysis making use of both observation and interview data, we developed a deeper understanding of stakeholders’ values and emotions. Innovation and creativity are key to their research work, but there is a conflict between a strong technical focus and willingness to adopt new software and concerns about control. Collaboration with external groups/people is important for the stakeholders’ research but they rarely share details of their analyses and expressed little trust and confidence in others’ abilities, revealing tension between trust and data security. VMEs elicited by the method informed development of mock-ups and early prototypes, in particular addition of security features to customise data access to particular stakeholder roles. The method also contributed requirements for the socio-technical system design which informed training material and user manuals for data access, and data sharing procedures.
The key issues identified at the beginning of the project were confirmed, and the apparent contradiction between expected and actual external collaboration suggested requirements for better collaborative tools with trust-building measures, e.g. visualisation of work flows and research activities. Security and privacy of data emerged as an important value which was added to the key issues. Sensitivity to the stakeholders’ emotional responses guided the RE process. Stakeholders’ frustrations and anxieties about different aspects of the project were recognised and explored, particularly with the pace of prototype development keeping in step with the requirements elicitation processes. The stakeholders expressed pleasure and engagement when involved in requirements workshops, responding enthusiastically to software prototypes which confirmed the RE approach was appropriate.
Values and motivations tables were more useful than the emotions since the system did not provoke any strong emotions, although the stakeholders did express frustration at delays in the prototype evaluation cycles, caused by resource problems. The value map resulting from the analysis is illustrated in Fig. 5.
The academic researchers’ values and motivations emerged from reviewing interview recordings about their work practices. They talked about working in isolation, being responsible for their own success and productivity, the methodical ways they organise and check their work and the care taken to keep their research data and practices secure. Observation of meetings between academic researchers showed them generating and reviewing ideas for new analyses or research. Similarly, interviews and meeting observation provided a rich source of evidence for values and motivations of the public health analysts. They had a strong sense of altruism, both in working with and sharing resources with other analysts, and in a desire to see positive change in the health of children in their local areas.
Collaboration, security and trust were shared values, but there were differences between the stakeholder in design exploration in prototypes, customisation, adaptability and security non-functional requirements. These were addressed by functional requirements for data security on servers, configurable workflows to match systematic or more opportunistic processes, while creative values were supported by interactive visualisation for data analysis. Collaboration and trust were fostered by an iterative user-centred RE process to build up trust within the team, and in the system implementation as a collaborative application. Sociability, altruism and achievement motivations informed decomposition of stakeholder goals. For example achievement, altruism and systematic values led to a subgoal to record analytic procedures, enabling academic researchers to track their own work, while also supporting public health analysts in sharing analysis techniques and results with colleagues.
The major functional requirements (goals) of the systems were for research and analysis support, namely database searches ranging from simple queries to complex associations between variables, leading to display of detailed epidemiological data set in a context with map and graph overviews and functions to compare trends over time and different areas on maps. The VME analysis added further non-functional requirements for privacy and security, while the trust, sociability and collaboration values had implications for CSCW (computer-supported collaborative work) functionality. Systematic, achievement and creativity values for the academic researchers suggested new functional requirements for flexible, extensible analysis tools (creativity support), with a clear process of investigation and audit trail (systematic).
SAMS case study
The SAMS project aims to increase the proportion of dementia sufferers receiving an early diagnosis by detecting changes in their pattern of computer use. At its core is a set of passive monitors that collect data as the user interacts routinely with their computer. These data are analysed to infer the stakeholders’ cognitive health against a set of clinical indicators representing memory, motor control, use of language, etc. For example, loss of vocabulary is a common symptom of dementia . The VBRE method was applied by the second author during interviews, scenario-storyboard requirements exploration sessions, and requirements analysis workshops. The method was applied from the initial start-up phase of the project, through early requirements analysis with mock-ups and early prototypes to the design phase with fully functional prototypes.
Requirements analysis was initiated with five workshops, conducted with a total of 24 participants (14 male, 10 female, age range 60–75, median 66). Participants were recruited through the Alzheimer’s Society, U3A (University of the Third Age), and local advertising in care homes, GP surgeries and elsewhere. All workshops were structured in two sessions lasting approximately 1 h. In the first session, the SAMS system aims, major components and operation were explained followed by presentation of PowerPoint storyboards illustrating design options for the alert-feedback user interface, such as choice of media (video, text, computer avatars), content (level of detail, social network) and monitoring (periodic feedback, alert-only, explicit tests). The second session focused on discussion of privacy issues in monitoring computer use, data sharing and security, ethical considerations, emotional impact of alert messages, stakeholders’ motivations and likelihood of taking follow-up tests.
Requirements issues raised in the workshops were explored further in 13 interviews presenting scenarios to illustrate similar design options with discussion on privacy, security and ethical issues. Questions in the interviews also probed stakeholders’ reactions to different levels of monitoring (e.g. actions, text) and their perceived trade-off between benefits/motivations versus fears/barriers for adopting the system and taking follow-up action after an alert message. Respondents (4 male, 9 female) ranged in age from 67 to 89 (median 72). The VBRE ‘hunches’ analysis was applied to interview notes and audio recordings of interviews and workshops. The scenarios used in both sessions were designed to test different design approaches that tacitly explored values, such as human-like presence in exploration, social networks (trust, sociability values) and explicit probing issues of security and privacy. The scenarios were designed using the value and motivation taxonomies, e.g. with probe questions to investigate self-efficacy (ability to monitor and control own health), altruism and curiosity. The list of emotions informed analysis of both sessions, with anxiety (invasion of privacy), fear (of diagnosis) and frustration (with computer design and medical professionals) constituting some of the observed reactions.
Results: values and emotions
In the workshops, scenarios and prototype designs stimulated only mild emotional reactions with some anxiety over privacy intrusion and accuracy of the computer analysis. No particular design was favoured; however, the avatar and social media options were the least favoured, indicating sociability was not a prominent motivation. All participants expressed concerns over privacy and security arising from monitoring their computer use. Although they were reluctantly willing to share their data with the researchers for analysis, most participants insisted they should have control over their own data. Sharing data with their close kin/friends had to be under their control and the majority would not share information or the alert with their doctor. The majority in all workshops were willing to allow monitoring of their computer use and e-mail text content, made suitably anonymous to protect the identities of other parties to conversations. Most participants expected to experience anxiety and fear if they received an alert message. Contact with a human expert or carer was cited as an important support, with connections to support groups (e.g. the Alzheimer’s Society) for reassurance (empathy) and as additional sources of information to motivate people to take follow-up tests.
As with the workshops, no consensus emerged with the prototypes from the interviews, although more information and sympathetic messages were preferred to the avatar/social media designs, suggesting that self-control (efficacy) was an important value. The respondents were even more concerned about privacy and security, possibly because three participants had recently experienced phishing attacks on the Internet. However, only two individuals were not willing to have their e-mail content monitored. Opinions on minimal data sharing and the need to maintain control over their own data were similar to the workshop participants. The majority of the respondents (11/13) expressed anxiety about being monitored, and expected to experience discomfort, fear and worry when they received an alert message, although ten respondents reported that they could not realistically imagine how they would react in a real-life situation. Five individuals noted that further explanation after the alert message would be vital and all reported that their main motivation for using the system was self-efficacy: a feeling of being in control by self-management of their health.
Conclusions: method and requirements implications
Given the diversity of opinion in both the workshops and interviews, prioritising requirements from this analysis was not an easy task. Several values, motivations and possible emotional reactions appeared to have an important bearing on the requirements and design options.
Privacy and security values these were the most common values, with implications for requirements concerning controls over any data sharing, encryption and secure transmission of data to university site, encryption on own PC to mitigate hacking attacks, depersonalised data only for wider research sharing. These values prioritised requirements for secure data transmission, data encryption on client (user) PCs as well as servers, and selective data recording so sensitive data such as home banking and all passwords were not recorded.
Trust in the SAMS system was closely related to the security, but it also involved accuracy of system information and diagnosis as well organisational trust in the universities (system authors), and healthcare professionals. Trust-building requirements included a user control to turn off SAMS recording, and consultation processes for involving users in the research and its results.
Motivations efficacy, desire for self-control, altruism: participation might help research on dementia. Self-control prioritised the user ‘stop recording’ control described above, and for visualisation requirements so users could view summaries of their own recorded activity. Altruism was reflected in publicity material for recruiting participants.
Emotion anxiety and fear of negative alert messages, uncertainty over personal reaction. These emotions led to further analysis of the wording of alert messages, use of graphics and images to explain the rationale behind the message, and options for social configuration, so carers and family members could have access to alerts and their explanation.
The VBRE method helped frame the scenarios and interview questioning strategy, as well as informing the analysis using the taxonomies to review interview notes and hunch lists. VMEs influenced design of data security, including encryption, secure transmission protocols, depersonalisation of data for research purposes, as well as empathetic design of the feedback-alert user interface. Emotional reactions encountered limitations since many participants remarked that they could not anticipate how they would feel when presented in a future reality with emotionally disturbing news (i.e. possible diagnosis of dementia). The scenarios did elicit possible reactions, but doubts were expressed about how accurate this information might be.
Case study: introducing VBRE into industrial practice
Diary-based case studies were used to validate the effectiveness and utility of the method with practising analysts who used the VBRE method and website in live projects. Diary studies provide a means of studying the effectiveness of the method over time in a real-world setting . The studies aimed to investigate:
Four requirement analyst volunteers trialled the method in their workplaces. Three were recruited by advertising within local requirements and usability groups. The fourth volunteer was then recruited on the recommendation of one of the existing volunteers who found the method sufficiently useful that he encouraged his colleague to join the evaluation.
An initial interview captured information about analysts’ backgrounds, their past project experience and forthcoming project work. This introductory meeting included a tutorial explaining the VBRE method, the taxonomies and associated guidelines/advice, and how to use the website. The analysts were encouraged to use the method and website as they felt appropriate to their particular project. They kept a record of their experiences of using the method, using a short-structured diary whenever they made use of any component of the method or completed requirements analysis activity on their project. The diary prompts requested information on the RE activity being undertaken, method components used (e.g. hunch list, taxonomies, website), implications (list values, etc., identified), and impact (new and changes to requirements). Interviews were carried out at the end of the study, during which the content of the diary was reviewed. All interviews and meetings were audio-recorded and transcribed.
It was anticipated that the analysts would apply the method to pieces of intermittent requirements work spread over several weeks or months (many of the analysts interviewed worked across several projects at a time, while others carried out multiple roles on their projects, e.g. project manager, user experience or software developer). It was not practical for the researcher to interview the analysts after each application of the method and participants were unlikely to be willing to alert the researcher to every use of the method, therefore diaries were used to collect data about the use of the method. A semi-structured diary format was adopted to facilitate consistent reporting between case studies, while providing some flexibility for participants to document issues or occurrences. Semi-structured approaches are also recommended as they provide participants with some guidance as to how to complete the diary . However, it is inevitable that the degree to which respondents complete the diary will vary and diaries may be incomplete; it is impossible to know how much information has not been reported. Therefore, a final interview reviewed the diary content to clarify any ambiguous entries.
Case study results
Table 6 provides an overview of the four studies, describing the requirements elicitation techniques adopted by each analyst, the number of diary entries, components of the method they opted to use and the length of time of the projects.
Analysts 3 made four formal diary entries but also reported regularly using the front page of the website as an aide memoire before meetings with stakeholders (approximately 1–3 times a week).
The participants were given the freedom to apply the method as they felt most appropriate to their project, and consequently the diary structure was designed to capture information about the ways the method was used, as well as its impact, with the following prompts:
The frequency of use of the VBRE method
The circumstances of use: format of the requirements elicitation activity and the situation
Method components used
Analyst’s assessment of the impact of the method
Problems applying the method.
The analysts kept a record of their experiences of using the method, using a short-structured diary whenever they made use of any component of the method or completed requirements analysis activity on their project. They were also given written guidance on the completion of the diary. Given that each analyst’s work situation was different and that some participants worked across multiple projects, analysts were asked to complete the diary for what they considered to be one phase of their requirements process on one project, covering initial elicitation to requirements definition, rather than to use the method for a set time period. Analysts were asked to contact the researcher to indicate when they felt they had completed an iteration of their requirements work; however, given the length of the evaluations for two of the analysts (6 months, 1 year) the researcher intermittently e-mailed them approximately every eight weeks for progress updates. Interviews were carried out at the end of each study, during which the content of the diary was reviewed and analysts were asked about their perceptions of the utility and usefulness of the method. They were also asked whether they would use the VBRE approach in future. All interviews and meetings were audio-recorded and transcribed.
A1 had a background in nursing and health informatics, but little formal requirements analysis training and had only worked as an analyst on one previous software project, which was badly received by some stakeholders. A1 believed this was due to late and poor stakeholder engagement, and consequently felt anxious about forthcoming stakeholder interviews. He tested the VBRE method in a project developing web-based software to enable the sharing of information about scientific research projects. The initial requirements elicitation consisted of a series of interviews with five research scientists.
A1 followed the VBRE method closely, making an initial hunch list which he updated after reviewing findings from each of his interviews. He used the VBRE website interview questions and scenarios when preparing for interviews. The key VME identified were privacy, ownership, collaboration and altruism.
When drawing up his preliminary hunch list A1 identified potential issues around the sharing of research data, in particular that researchers did not wish to share information with their rivals, that there might be a lack of interest in collaboration, and that the website might increase speculative contacts requesting data sharing which could be intrusive (privacy, ownership values). He extended his interview plan to include questions to explore whether these concerns were correct. He explored the question of data sharing and developed a more nuanced understanding about what the scientists were and were not willing to share. In contrast to his original expectations, he was surprised to discover that most of the interviewees were happy to share details of their research, viewing it as a way to promote collaboration and an altruistic act which would help novice researchers trying to design their first studies.
A1 also identified a clash in goals between the project funders and the researchers around the sharing of details about biological samples: the funders considered samples collected during projects they had financed to be publicly owned, whereas the researchers were concerned that sharing information might reveal confidential information about their research in advance of publication. This issue was consequently flagged to the project steering committee to develop a policy around sample ownership over the course of a research project. A1 commented that he felt the VBRE method was most useful in helping him prepare for interviews, in particular using the hunch list to prompt him to consider asking questions about stakeholders’ values, while the website interview questions advised on how to broach these subjects.
A2 has a background in bioinformatics and has worked as a requirements analyst for 5 years on a series of large European projects. He tested the VBRE method within one of these projects, developing data management software for an EU consortium. The project faced substantial problems related to trust when it was initiated. Over the course of this case study, some of the original stakeholders left, and new people joined. A2 was concerned about handling this transition and keeping the stakeholder group running smoothly.
A2 used the method to help prepare for the large stakeholder meetings held every 6 months, and then to review the outputs of these meetings. The main components of the method used were the hunch list, the values review, website front page and the scenarios. The key VME identified were openness, tolerance and trust.
A2 found that drawing up the hunch list and reviewing meeting notes was useful: “a way to tie down things that had been buzzing about in my head for a while” and “it’s a reminder to look at things from a different point of view”. He felt it enabled him to plan the workshops with greater consideration about how to integrate new stakeholders into the existing group. In particular, he highlighted openness, trust and collaboration as positive values that were shared by the existing stakeholders and described wanting to ensure these values were promoted and shared with new members. After the first meeting involving the new members of the group, he identified a need to change how the meetings were run, realising that although the established members were comfortable with each other and used to being direct, and sometimes confrontational, the newer members were quite anxious and shy. To this end, the second workshop was planned with a series of smaller breakout sessions to allow the new and old members to mingle, rather than a large ‘town-hall’ gathering (openness, tolerance, anxiety and trust).
He made some use of the website, commenting that he liked the front page as a reminder of VME issues when planning the analysis sessions and that the scenarios were a valuable way of thinking about the consequences of change. He remarked that “the scenarios helped the interviews go better because I anticipated the issues”.
A3 has a degree in computer science and has worked as a requirements analyst for over 10 years. He is a contractor working in a wide variety of industries and tested the method within a project developing clinical trial software, clinicians, academic researchers and the pharmaceutical industry—he described this project as “technically straightforward but intensely political”. All of the requirements for this project were gathered via workshops and telephone conferences. A3 was confident in running workshops and felt he had good people skills.
A3 started using the VBRE method when he was several months into the project and drew up a hunch list identifying values and emotions covering all the parties involved in the project, which was regularly reviewed and updated in advance of meetings. He made little use of the VBRE website beyond the values list on the homepage. The key VME identified were innovation, ambition, security, and ownership.
Many of the emotions A3 identified within his project were negative: resistance to change due to concerns about the consequences of failure, anxiety about loss of status. A3 tried to address these concerns and to reinforce positive outcomes of the project. He identified shared, cross-organisation values: an enthusiasm for innovation, as well as ambitions around career progression if the project is a success; speaking to these shared values provided a way to encourage disagreeing stakeholders to find ways to reach consensus on requirements for the sake of the bigger picture.
A3 also made use of his understanding of stakeholders’ values (security, control) in defining system requirements, e.g. building in mechanisms to allow data owners to track and control requests for access to their data, to view reasons for requests and refuse them if considered inappropriate.
A3 felt that the most useful part of the method was drawing up a hunch list and reflecting on the values related to what was happening on the project. He noted that the home page of the website was the most important page for an experienced analyst: “Once I’ve triggered something I can cascade most of what I need but without the front page it slips into the background. There was comparatively little on the website that was new to me, though it was well expressed—I would use it when talking to newer analysts—the main value to me is the reminder list and the intent to leave some time to reflect on this stuff”.
A4 has worked as a commercial software developer for nine years but his computer science degree did not include any RE training, and until his most recent project, he has done little requirements analysis. He is responsible for requirements analysis in support of a tool to collect research data. This involves collecting requirements from nurses and from an external research organisation.
Creation of hunch list, taxonomies, lists of interview questions and scenarios, reflection on stakeholders’ values and, particularly, emotions. The key VME identified were ambition, trust, responsibility and peer-esteem.
A4 felt the method was helpful in preparing for difficult meetings. He drew up a hunch list and reflected on the different stakeholders’ involvement and felt this enabled him to unpick why the meetings were tense and to think about the participants’ motivations. He identified: ambition, lack of trust, anxiety, extremely high value placed on accuracy, concerns about being perceived as competent and inequality among stakeholders as issues for the project. Beyond this, he was then able to pre-prepare a set of requirements that addressed some of the stakeholder conflicts. A4 commented that he felt more in control of the requirements process and less likely to be surprised. In particular, he realised that key sticking points for getting agreement on the requirements were issues around the nurses’ dependencies on external staff during data collection. A4 did not make use of the scenarios or the design and project advice, commenting that he was very focused on his own project and did not want to think about anything else. He reported feeling better prepared for subsequent meetings due to having anticipated likely problems, and consequently more confident and less anxious.
All the participants reported that their use of the method had prompted them to consider stakeholders’ VME. All the analysts were able to give examples in which they attributed changes either to their requirements process, or to project requirements as a consequence of applying the method, including:
Interview and workshop planning
Challenging analysts’ assumptions about stakeholders’ VMEs
Reflection on outcomes of meetings
The addition of new functionality to a project in order to encourage stakeholders to adopt the software.
When asked whether they felt VME were relevant to RE, the analysts primarily focused on stakeholder anxiety, and it is noticeable that anxiety was listed as a relevant VME in three of these case studies; however, a wide range of VME were identified. It is also noticeable that in all the case studies, the analysts identified positive values they wished to promote either within the requirements analysis process, or as qualities of the new system.
The method and website were used extensively in preparing for requirements analysis interviews and workshops. Scenarios, taxonomies and implications guidelines helped anticipate problems, and shaping interview questions to explore problems and find solutions. Less use was made by more experienced analysts of the interview questions and requirements advice, but all found the website provided a useful prompt list to help widen their thinking about social issues and stakeholder values.
Participants were given the freedom to apply the method as they felt appropriate. All drew up hunch lists at the start of their requirements analysis, and felt this was a worthwhile exercise. Both of the novice analysts, A1 and A4, described reviewing some of their interview notes alongside their hunch list. While all the analysts commented on using the website extensively in preparing for meetings, other use of the website varied, with some referring simply to the home page and the taxonomies, while others made use of the detailed guidance. Three of the analysts felt that the full method was time-consuming to apply, although two stated that this would not be a barrier to future use. Analyst A3 had chosen not to apply the ‘cookbook’ version of the method, instead picking and choosing when to visit the website, and had not found this particularly time-consuming. The analysts used a mixture of interviews and workshops as their main requirements elicitation techniques. None reported any issues with integrating the VBRE method into their existing RE approach, understanding the website content or the application of the method.
During the final interview, participants were asked whether they would make use of the method again. Two of the analysts said they would, and A3 commented that he continues to visit the website regularly. A2 does not intend to use VBRE again within his current project but would consider using it when starting a new project. A4 thought that he was unlikely to continue to use the method as it was not part of his company’s standard processes.
Clearly, it is not possible to know whether design decisions and activities undertaken by the analysts are a direct consequence of using VBRE; however, all described using the method to influence their requirements process or understanding of the requirements. For example, A1 raised an issue related to clashing motivations underlying data sharing versus career development, while A2 described changing his approach to workshops to improve the integration of new team members. A3 noted requirements for data access controls, and tracking updates as a consequence of security value concerns, while A4’s analysis of trust values suggested requirements for visibility and checks on data accuracy. A4 felt drawing up the VBRE hunch list enabled him to better understand the causes of conflict within his project, and addressed some of the issues with changes to the system requirements. It is also apparent from the analysts’ hunch lists and diary entries that they paid close attention to their stakeholders’ values, motivations and emotions and acted upon this information.
The generalisability of this evaluation is limited by the small sample size used, and the self-selecting nature of our analyst volunteers. The diary-based approach suffered from the inconsistent reporting at regular intervals and partial conformance to completion instructions; however, it did collect sufficient data from working analysts that allowed us to explore the utility and usability of the method in real life, providing good ecological validity to our evaluation. Gaps in the diary records were followed up in interviews to produce a more complete record of analysts’ activity.