Skip to main content
SpringerLink
Log in

Systems usability framework for evaluating tools in safety–critical work

  • Original Article
  • Published:
Cognition, Technology & Work Aims and scope Submit manuscript

Abstract

This paper presents a methodology with which evaluation of control rooms in safety–critical industry has been carried out. The results of two cases of evaluation in nuclear power plants (NPP) are presented to elaborate the method thoroughly. The evaluation methodology draws from conventions of human factors and usability evaluation methods but adds an activity theoretical viewpoint by considering different control room functions in a purposeful activity system. A control room has three functions in an activity: instrument, psychological tool, and a communicative tool. The introduced evaluation approach allows understanding of quality of the control room on the level of outcomes it produces, work practices that it consolidates, and the user experience it creates. The evaluation approach is demonstrated by presenting baseline evaluations of two hybrid NPP control rooms that were carried out before major upgrade projects were implemented. Hybrid control room refers to a state in which modern and original hardwired technological solutions are in use simultaneously.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Notes

  1. Probably the most commonly used term is human computer interaction (HCI) but by replacing the “computer” with a more general “technology” the scope of the research field widens slightly and also technologies not involving computers can be addressed.

  2. Developing the methodology to describe situations in a way that makes explicit the connections of the events in the situation to higher level safety and efficiency functions, has been on our research agenda for many years. An overview of the earlier joint work in this issue is provided by Norros (2004). The use of FSM is explained also by Salo et al. (2009).

References

  • Bannon L (1991) From human factors to human actors. In: Greebaum J, Kyng M (eds) Design at work: cooperative design of computer systems. LEA, Mahwah, pp 27–44

    Google Scholar 

  • Béguin P, Rabardel P (2000) Designing for instrument-mediated activity. Scand J Information Systems 12:173–190

    Google Scholar 

  • Bødker S (2006) When second wave HCI meets third wave challenges. NordiCHI. doi:10.1145/1182475.1182476

  • Braarud P-O, Skraaning Jr G (2006) Insights from a benchmark integrated validation of a modernized NPP control room: performance measurement and the comparison to benchmark system. American Nuclear Society, NPIC&HMIT, Albuquerque

  • Burkhardt J, Détienne F, Hébert A, Perron L, Safin S, Leclercq P (2009) An approach to assess the quality of collaboration in technology-mediated design situations. ECCE 2009, Helsinki, pp 30:1–30:9. Retrieved from http://portal.acm.org/citation.cfm?id=1690508.1690551

  • Carroll JM (1997) Human–computer interaction: psychology as a science of design. Int J Human-Computer Studies 46(4):501–522. doi:10.1006/ijhc.1996.0101

    Article  Google Scholar 

  • Dahl Y, Alsos OA, Svanæs D (2010) Fidelity considerations for simulation-based usability assessments of mobile ICT for hospitals. Int J Human-Computer Interaction 26(5):445–476. doi:10.1080/10447311003719938

    Article  Google Scholar 

  • Daniellou F (2005) The french-speaking ergonomists’ approach to work activity: Cross-influences of field intervention and conceptual models. Theoretical Issues in Ergonomics Science 6(5):409–427. doi:10.1080/14639220500078252

    Article  Google Scholar 

  • Dekker S (2011) Drift into failure—from hunting broken components to understanding complex systems. Ashgate, Burlington

    Google Scholar 

  • Dewey J (1999) The quest for certainty. A study of the relation of knowledge and action (finnish translation). Gaudeamus, Helsinki

  • Dourish P (2001) Where the action is—the foundations of embodied interaction. MIT Press, Cambridge

    Google Scholar 

  • Engeström Y (1987) Learning by expanding: an activity-theoretical approach to developmental research. Orienta-Konsultit, Helsinki

    Google Scholar 

  • Gay G, Hembrooke H (2004) Activity-centered design: an ecological approach to designing smart tools and usable systems. MIT Press, Cambridge

    Google Scholar 

  • Harrison S, Sengers P, Tatar D (in press) Making epistemological trouble: third-paradigm HCI as successor science. Interact Comput. doi:10.1016/j.intcom.2011.03.005

  • Hart SG, Staveland LE (1988) Development of a multi-dimensional workload rating scale: results of empirical and theoretical research. In: Hancock PA, Meshkati N (eds) Human mental workload. Elsevier, Amsterdam, pp 139–183

    Chapter  Google Scholar 

  • Huang E, Mynatt E, Trimble J (2006) Displays in the wild: understanding the dynamics and evolution of a display ecology. In: Fishkin K, Schiele B, Nixon P, Quigley A (eds) Pervasive computing. Springer, Berlin, pp 321–336

    Chapter  Google Scholar 

  • Hyysalo S (2002) Transforming the object in product design. Outlines, Critical Social Studies 4(1):59–83

    Google Scholar 

  • Kaptelinin V, Nardi B (2006) Acting with technology: activity theory and interaction design. MIT Press, Cambridge

    Google Scholar 

  • Kestenbaum V (1977) The phenomenological sense of John Dewey: habit and meaning. Humanities Press, Atlantic Highlands

    Google Scholar 

  • Klemola U-M, Norros L (2001) Practice-based criteria for assessing anaesthetists’ habits of action: outline for a reflexive turn in practice. Med Educ 35(5):455–464. doi:10.1046/j.1365-2923.2001.00894.x

    Article  Google Scholar 

  • Klemola U-M, Norros L (2002). Activity-based analysis of information characteristics of monitors and their use in anaesthetic practice. ECCE 2002, Catania

  • Koski-Jännes A (1999) From addiction to self governance. In: Engeström Y, Miettinen R, Punamäki R-L (eds) Perspectives on activity theory. Cambridge University Press, Cambridge

    Google Scholar 

  • Kuutti K (1991) The concept of activity as a basic unit of analysis for CSCW research. ECSCW 1991, Amsterdam, pp 249–264

  • Laarni J, Norros L, Rinttilä E (2006) Control room modernisation from an evolutionary point of view—experiences gained at the loviisa NPP. Workshop on Future Control Station Designs and Human Performance Issues in Nuclear Power Plants, Halden

    Google Scholar 

  • Leontjev AN (1978) Activity, consciousness, and personality (finnish translation) Kansankulttuuri Oy, Helsinki

  • Lepreux S, Abed M, Kolski C (2003) A human-centred methodology applied to decision support system design and evaluation in a railway network context. Cogn Technol Work 5(4):248–271. doi:10.1007/s10111-003-0128-9

    Article  Google Scholar 

  • Lützhöft M (2004) “The technology is great when it works”: maritime technology and human integration on the ship’s bridge. Dissertation, Linköping University

  • Määttänen P (2009) Toiminta ja kokemus (action and experince). Gaudeamus, Helsinki

    Google Scholar 

  • Mattsson S (2011) Measures of collaboration in CSCW: usability and collective measures in remote and co-located problem-solving. Master’s thesis, Chalmers University of Technology

  • McCarthy J, Wright P (2004) Technology as experience. MIT Press, Cambridge

    Google Scholar 

  • Merleau-Ponty M (1986) The phenomenology of perception. Routledge & Kegan Paul, London

    Google Scholar 

  • Nachreiner F, Nickel P, Meyer I (2006) Human factors in process control systems: the design of human–machine interfaces. Saf Sci 44(1):5–26

    Article  Google Scholar 

  • Neale DC, Carroll JM, Rosson MB (2004) Evaluating computer-supported cooperative work: Models and frameworks. In: Proceedings of CSCW 2004. Chicago, IL, pp 112–121. doi:10.1145/1031607.1031626

  • Nickel P, Nachreiner F (2008) Evaluation of presentation of information for process control operations. Cogn Technol Work 10(1):23–30. doi:10.1007/s10111-007-0074-z

    Article  Google Scholar 

  • Norman DA (2006) Logic versus usage: the case for activity-centered design. Interactions 13(6):45-ff. doi:10.1145/1167948.1167978

    Google Scholar 

  • Norros L (2004) Acting under uncertainty. The core-task analysis in ecological study of work. VTT, Espoo

    Google Scholar 

  • Norros L, Klemola U (1999) Methodological considerations in analysing anaesthetists’ habits of action in clinical situations. Ergonomics 42(11):1521–1530

    Article  Google Scholar 

  • Norros L, Nuutinen M (2005) Performance-based usability evaluation of a safety information and alarm system. Int J Hum-Comput Stud 63(3):328–361. doi:10.1016/j.ijhcs.2005.03.004

    Google Scholar 

  • Norros L, Salo L (2009) Design of joint systems: a theoretical challenge for cognitive systems engineering. Cogn Technol Work 11(1):43–56

    Article  Google Scholar 

  • Norros L, Savioja P (2007) Towards a theory and method for usability evaluation of complex human-technology systems. Activités 4(2):143–150

    Google Scholar 

  • Norros L, Liinasuo M, Hutton R (2011a) Evaluating the potential of new technological tools for safety critical work. Interact Comput 23(4):299–307. doi:10.1016/j.intcom.2011.05.003

    Article  Google Scholar 

  • Norros L, Liinasuo M, Savioja P (2011b) Operators’ conceptions of procedure guidance in NPP process control. In: Proceedings of EHPG. Halden, Norway

  • O’Hara J (1999) A quasi-experimental model of complex human-machine system validation. Cogn Technol Work 1(1):37

    Article  Google Scholar 

  • O’Hara J, Pirus D, Beltratcchi L (2003) Information display: considerations for designing modern computer-based display systems (no. EPRI-1002830). EPRI, Palo Alto

  • Park J (2009) The complexity of proceduralized tasks. Springer, London

    Book  Google Scholar 

  • Peirce CS (1991) Peirce on signs. In: Hooper J (ed) Writings on semiotic by Charles Sanders Peirce. University of North Carolina Press, Chapel Hill

    Google Scholar 

  • Peirce CS (1998a) The Harvard lectures on pragmatism. In: T.P.E. Project (ed) The essential Peirce. Selected philosophical writings. Indiana University Press, Bloomington, pp 133–241

  • Peirce CS (1998b) The Peirce edition project. Introduction. In: T.P.E. Project (ed) The essential Peirce. Selected philosophical writings. Indiana University Press, Bloomington, pp 133–241 (pp XXX)

  • Raeithel A (1983) Tätigkeit, arbeit und praxis. Campus, Frankfurt am Mein

    Google Scholar 

  • Randell R, Wilson S, Fitzpatrick G (2010) Editorial—evaluating new interactions in health care: challenges and approaches. International Journal of Human-Computer Interaction 26(5):407–413. doi:10.1080/10447311003719847

    Article  Google Scholar 

  • Roto V, Law E, Vermereen A, Hoonhout J (2011) User experience white paper - bringing clarity to the concept of user experience. http://www.allaboutux.org/files/UX-WhitePaper.pdf. Accessed 1 December 2011

  • Rückriem G (2003) Tool or medium? The meaning of information and telecommunication technology to human practice. A quest for systemic understanding of activity theory. Paper Presented in FISCAR ‘03. Espoo

  • Rückriem G (2009) Digital technology and mediation: a challenge to activity theory. In: Sannino A, Daniels H, Gutiérrez KD (eds) Learning and expanding with activity theory. Cambridge University Press, New York, pp 88–111

    Chapter  Google Scholar 

  • Salo L, Norros L, Savioja P (2009) Using operating procedures in NPP process control. In: Proceedings of ECCE 2009, pp 258–266

  • Savioja P, Norros L (2008) Systems usability—promoting core-task oriented work practices. In: Law E, Hvannberg E, Cockton G (ed) Maturing usability. Quality in software, interaction and value. Springer, London, p 123

  • Savioja P, Norros L, Salo L (2008) Evaluation of systems usability. Proceedings of ECCE 2008, pp 110–117

  • Schraagen J, Militello L, Ormerod T, Lipshitz R (2008) Naturalistic decision making and macrocognition. Ashgate, Burlington

    Google Scholar 

  • Stanton N, Salmon P, Jenkins D, Walker G (2010) Human factors in the design and evaluation of central control room operations. CRC Press, Boca Raton

    Google Scholar 

  • Stevenson D, Hutchins M, Smith J (2010) Human-centered evaluation for broadband tertiary outpatient telehealth: a case study. Int J Hum Comput Interact 26(5):506–536. doi:10.1080/10447311003719979

    Article  Google Scholar 

  • Twidale M, Randall D, Bentley R (1994) Situated evaluation for cooperative systems. In: Proceedings of CSCW 1994. Chapel Hill, North Carolina, pp 441–452. doi:10.1145/192844.193066

  • Upton C, Doherty G, Gleeson F, Sheridan C (2010) Designing decision support in an evolving sociotechnical enterprise. Cogn Technol Work 12(1):13–30. doi:10.1007/s10111-008-0124-1

    Article  Google Scholar 

  • Vicente KJ (1999) Cognitive work analysis. Toward a safe, productive, and healthy computer-based work. Lawrence Erlbaum Publishers, Mahwah

  • Vygotsky LS (1978) Mind in society. The development of higher psychological processes. Harvard University Press, Cambridge

    Google Scholar 

  • Ylirisku S, Halttunen V, Nuojua J, Juustila A (2009) Framing design in the third paradigm. In: Proceedings of the 27th international conference on human factors in computing systems, Boston, MA, USA, pp 1131–1140. doi:10.1145/1518701.1518874

Download references

Acknowledgments

The authors have been fortunate to work with an excellent research team. We wish to thank project manager Jari Laarni and the whole team: Leena Salo, Marja Liinasuo, Iina Aaltonen, Hannu Karvonen and Hanna Koskinen for inspiring collaboration in conducting the work. The work presented here has been funded by the Finnish National Nuclear Safety Program SAFIR, and the Fortum Foundation. We thank the funding organizations for the possibility to conduct the studies. The authors thank the collaborating NPPs and their personnel especially the trainers and the operators for their contributions in the evaluation studies.

Author information

Authors and Affiliations

  1. VTT Technical Research Centre of Finland, PL 1000, 02044, VTT, Finland

    Paula Savioja & Leena Norros

Authors
  1. Paula Savioja
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Leena Norros
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Paula Savioja.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Savioja, P., Norros, L. Systems usability framework for evaluating tools in safety–critical work. Cogn Tech Work 15, 255–275 (2013). https://doi.org/10.1007/s10111-012-0224-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10111-012-0224-9

Keywords

Search

Navigation