Abstract
Comparing software systems for a purchase decision in a highly regulated public procurement process is a new domain for usability testing. We performed a comparative summative usability evaluation of Current Research Information Systems (CRIS) as a part of a public procurement process in Finland, EU. The evaluation method had to provide objective and unbiased results for comparison and it had to be defined in detail already in the invitation to tender. In this paper, we report the details of our successful procedure that enabled straightforward and quick decision making in selecting the winning system. Additionally, we present calculations to legitimate the potentially higher investment costs through estimated savings and increased income.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Bangor, A., Kortum, P.T., Miller, J.T.: An empirical evaluation of the system usability scale. Int. J. Hum-Comput. Int. 6, 574–594 (2008)
Bevan, N. Claridge, N., Maguire, M., Athousaki, M.: Specifying and evaluating usability requirements using the common industry format: four case studies. In: Proceedings of IFIP 17th World Computer Congress, pp. 149–159. Kluwer, B.V., Deventer (2002)
Bias, R., Mayhew, D.: Cost-Justifying Usability. Academic Press, Boston (1994)
Booth, P.A.: An Introduction to Human-Computer Interaction. Lawrence Erlbaum, Hove (1989)
Bradley, M.M., Lang, P.J.: Measuring emotion: the self-assessment manikin and the semantic differential. J. Behav. Ther. Exp. Psy. 1, 49–59 (1994)
Buie, E., Murray, D. (eds.): Usability in Government Systems: User Experience Design for Citizens and Public Servants. Morgan Kaufmann, Waltham (2012)
Bødker, S., Madsen, K.: Methods & tools: context: an active choice in usability work. Interactions 4, 17–25 (1998)
Carey, T.T.: A usability requirements model for procurement life cycles. In: Carey, J.M. (ed.) Human Factors in Information Systems: An Organizational Perspective, pp. 89–104. Ablex, Norwood (1991)
Directive 2004/18/EC of the European Parliament and of the Council of 31 March 2004 on the coordination of procedures for the award of public works contracts, public supply contracts and public service contracts. http://eur-lex.europa.eu/legal-content/EN/TXT/?qid=1434726761459&uri=CELEX:32004L0018
Dumas, J.S., Redish, J.C.: A Practical Guide to Usability Testing. Ablex, Norwood (1993)
Finnish Act on Public Contracts (348/2007): http://www.finlex.fi/en/laki/kaannokset/2007/en20070348.pdf
Hartson, H.R., Andre, T.S., Williges, R.C.: Criteria For evaluating usability evaluation methods. Int. J. Hum-Comput. Int. 1, 145–181 (2003)
ISO 9241–11: Ergonomic requirements for office work with visual display terminals (VDTs) Part 11: Guidance on Usability. ISO (1998)
ISO 9241–210: Ergonomics of human-system interaction. Part 210: Human-centred design for interactive systems. ISO (2010)
ISO/IEC 25062: Software engineering – Software product Quality Requirements and Evaluation (SQuaRE) – Common Industry Format (CIF) for usability test reports. ISO (2006)
ISO/TR 16982: Ergonomics of human-system interaction – Usability methods supporting human-centred design. ISO (2002)
Jokela, T., Laine, J., Nieminen, M.: Usability in RFP’s: the current practice and outline for the future. In: Kurosu, M. (ed.) HCII/HCI 2013, Part II. LNCS, vol. 8005, pp. 101–106. Springer, Heidelberg (2013)
Lazar, J., Heidi, J., Hochheiser, H.: Research Methods in Human-Computer Interaction. Wiley, Glasgow, 426 p. ISBN: 978–0-470-72337-1. (2010)
Lewis, J.R.: Sample sizes for usability studies: Additional considerations. Hum. Factors 2, 368–378 (1994)
Molich, R., Dumas, J.S.: Comparative usability evaluation (CUE-4). Behaviour and Information Technology, vol. 27, No. 3, pp. 263–281. ISSN: 0144929X, DOI: 10.1080/01449290600959062. (2008)
Nielsen, J.: Estimating the number of subjects needed for a thinking aloud test. Int. J. Hum-Comput. Stud. 3, 385–397 (1994)
Nielsen, J., Molich, R.: Heuristic evaluation of user interfaces. In: Chew, JC., Whiteside, J. (eds.). Proceedings of the ACM CHI’1990 Conference on Human Factors in Computing Systems (CHI1990). ACM, New York, pp. 249–256. ISBN: 0-201-50932-6, DOI: 10.1145/97243.97281. (1990)
Sauro, J., Kindlund, E.: A method to standardize usability metrics into a single score. In: Proceedings of CHI 2005, pp. 401–409, ACM Press, New York (2005)
Sauro, J., Lewis, J.R.: When designing usability questionnaires, does it hurt to be positive? In: Proceedings of CHI 2011, pp. 2215–2224, ACM Press, New York (2011)
Virzi, R.A.: Refining the test phase of usability evaluation: how many subjects is enough? Hum. Factors 4, 457–468 (1992)
Whiteside, J., Bennett, J., Holzbatt, K.: Usability engineering: our experience and evolution. In: Helander, M. (ed.) Handbook of Human-Computer Interaction. Elsevier BV, Amsterdam (1988)
Acknowledgements
We thank all the volunteers who took part in the test sessions as well as the moderators of the tests, and Sampo Teräs for his comments on the first draft of this article.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2015 Springer International Publishing Switzerland
About this paper
Cite this paper
Riihiaho, S., Nieminen, M., Westman, S., Addams-Moring, R., Katainen, J. (2015). Procuring Usability: Experiences of Usability Testing in Tender Evaluation. In: Oinas-Kukkonen, H., Iivari, N., Kuutti, K., Öörni, A., Rajanen, M. (eds) Nordic Contributions in IS Research. SCIS 2015. Lecture Notes in Business Information Processing, vol 223. Springer, Cham. https://doi.org/10.1007/978-3-319-21783-3_8
Download citation
DOI: https://doi.org/10.1007/978-3-319-21783-3_8
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-21782-6
Online ISBN: 978-3-319-21783-3
eBook Packages: Computer ScienceComputer Science (R0)