Procuring Usability: Experiences of Usability Testing in Tender Evaluation

  • Sirpa RiihiahoEmail author
  • Marko Nieminen
  • Stina Westman
  • Ronja Addams-Moring
  • Jukka Katainen
Conference paper
Part of the Lecture Notes in Business Information Processing book series (LNBIP, volume 223)


Comparing software systems for a purchase decision in a highly regulated public procurement process is a new domain for usability testing. We performed a comparative summative usability evaluation of Current Research Information Systems (CRIS) as a part of a public procurement process in Finland, EU. The evaluation method had to provide objective and unbiased results for comparison and it had to be defined in detail already in the invitation to tender. In this paper, we report the details of our successful procedure that enabled straightforward and quick decision making in selecting the winning system. Additionally, we present calculations to legitimate the potentially higher investment costs through estimated savings and increased income.


Usability evaluation Public procurement Research information system Test user Test task System usability scale 



We thank all the volunteers who took part in the test sessions as well as the moderators of the tests, and Sampo Teräs for his comments on the first draft of this article.


  1. 1.
    Bangor, A., Kortum, P.T., Miller, J.T.: An empirical evaluation of the system usability scale. Int. J. Hum-Comput. Int. 6, 574–594 (2008)CrossRefGoogle Scholar
  2. 2.
    Bevan, N. Claridge, N., Maguire, M., Athousaki, M.: Specifying and evaluating usability requirements using the common industry format: four case studies. In: Proceedings of IFIP 17th World Computer Congress, pp. 149–159. Kluwer, B.V., Deventer (2002)Google Scholar
  3. 3.
    Bias, R., Mayhew, D.: Cost-Justifying Usability. Academic Press, Boston (1994)Google Scholar
  4. 4.
    Booth, P.A.: An Introduction to Human-Computer Interaction. Lawrence Erlbaum, Hove (1989)Google Scholar
  5. 5.
    Bradley, M.M., Lang, P.J.: Measuring emotion: the self-assessment manikin and the semantic differential. J. Behav. Ther. Exp. Psy. 1, 49–59 (1994)CrossRefGoogle Scholar
  6. 6.
    Buie, E., Murray, D. (eds.): Usability in Government Systems: User Experience Design for Citizens and Public Servants. Morgan Kaufmann, Waltham (2012)Google Scholar
  7. 7.
    Bødker, S., Madsen, K.: Methods & tools: context: an active choice in usability work. Interactions 4, 17–25 (1998)CrossRefGoogle Scholar
  8. 8.
    Carey, T.T.: A usability requirements model for procurement life cycles. In: Carey, J.M. (ed.) Human Factors in Information Systems: An Organizational Perspective, pp. 89–104. Ablex, Norwood (1991)Google Scholar
  9. 9.
    Directive 2004/18/EC of the European Parliament and of the Council of 31 March 2004 on the coordination of procedures for the award of public works contracts, public supply contracts and public service contracts.
  10. 10.
    Dumas, J.S., Redish, J.C.: A Practical Guide to Usability Testing. Ablex, Norwood (1993)Google Scholar
  11. 11.
    Finnish Act on Public Contracts (348/2007):
  12. 12.
    Hartson, H.R., Andre, T.S., Williges, R.C.: Criteria For evaluating usability evaluation methods. Int. J. Hum-Comput. Int. 1, 145–181 (2003)CrossRefGoogle Scholar
  13. 13.
    ISO 9241–11: Ergonomic requirements for office work with visual display terminals (VDTs) Part 11: Guidance on Usability. ISO (1998)Google Scholar
  14. 14.
    ISO 9241–210: Ergonomics of human-system interaction. Part 210: Human-centred design for interactive systems. ISO (2010)Google Scholar
  15. 15.
    ISO/IEC 25062: Software engineering – Software product Quality Requirements and Evaluation (SQuaRE) – Common Industry Format (CIF) for usability test reports. ISO (2006)Google Scholar
  16. 16.
    ISO/TR 16982: Ergonomics of human-system interaction – Usability methods supporting human-centred design. ISO (2002)Google Scholar
  17. 17.
    Jokela, T., Laine, J., Nieminen, M.: Usability in RFP’s: the current practice and outline for the future. In: Kurosu, M. (ed.) HCII/HCI 2013, Part II. LNCS, vol. 8005, pp. 101–106. Springer, Heidelberg (2013)Google Scholar
  18. 18.
    Lazar, J., Heidi, J., Hochheiser, H.: Research Methods in Human-Computer Interaction. Wiley, Glasgow, 426 p. ISBN: 978–0-470-72337-1. (2010)Google Scholar
  19. 19.
    Lewis, J.R.: Sample sizes for usability studies: Additional considerations. Hum. Factors 2, 368–378 (1994)zbMATHGoogle Scholar
  20. 20.
    Molich, R., Dumas, J.S.: Comparative usability evaluation (CUE-4). Behaviour and Information Technology, vol. 27, No. 3, pp. 263–281. ISSN: 0144929X, DOI:  10.1080/01449290600959062. (2008)
  21. 21.
    Nielsen, J.: Estimating the number of subjects needed for a thinking aloud test. Int. J. Hum-Comput. Stud. 3, 385–397 (1994)CrossRefGoogle Scholar
  22. 22.
    Nielsen, J., Molich, R.: Heuristic evaluation of user interfaces. In: Chew, JC., Whiteside, J. (eds.). Proceedings of the ACM CHI’1990 Conference on Human Factors in Computing Systems (CHI1990). ACM, New York, pp. 249–256. ISBN: 0-201-50932-6, DOI:  10.1145/97243.97281. (1990)
  23. 23.
    Sauro, J., Kindlund, E.: A method to standardize usability metrics into a single score. In: Proceedings of CHI 2005, pp. 401–409, ACM Press, New York (2005)Google Scholar
  24. 24.
    Sauro, J., Lewis, J.R.: When designing usability questionnaires, does it hurt to be positive? In: Proceedings of CHI 2011, pp. 2215–2224, ACM Press, New York (2011)Google Scholar
  25. 25.
    Virzi, R.A.: Refining the test phase of usability evaluation: how many subjects is enough? Hum. Factors 4, 457–468 (1992)Google Scholar
  26. 26.
    Whiteside, J., Bennett, J., Holzbatt, K.: Usability engineering: our experience and evolution. In: Helander, M. (ed.) Handbook of Human-Computer Interaction. Elsevier BV, Amsterdam (1988)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Sirpa Riihiaho
    • 1
    Email author
  • Marko Nieminen
    • 1
  • Stina Westman
    • 2
  • Ronja Addams-Moring
    • 1
  • Jukka Katainen
    • 3
  1. 1.Department of Computer ScienceAalto UniversityAaltoFinland
  2. 2.Department of Media TechnologyAalto UniversityAaltoFinland
  3. 3.IT ServicesAalto UniversityAaltoFinland

Personalised recommendations