Abstract
One way to investigate and account for the role of experience in dynamic decision-making tasks is to use a knowledge elicitation method, for example verbal protocols. Recently, methods for verbalisation by other subjects have been suggested as a way to elicit more information on thinking. One of these methods is here investigated in a quasi-experimental study on highly experienced and skilled train dispatchers. Collegial verbalisation is based on the procedure of videotaping practitioners while they perform their normal work tasks in their normal work setting. This is followed up by having a close colleague of the practitioner watch the video recordings and verbalise. The general hypothesis that verbal reports based on collegial verbalisations can provide protocol data that are close to the structure and content of verbal reports based on retrospective verbalisations was investigated. From this systematic comparison it is concluded that collegial verbalisations produce verbal protocols that are close to the retrospective verbal protocols on protocol and topic levels, but not on statement level. From this study, we conclude that the collegial protocols can be used as an independent source of data. It seems possible for a colleague to report verbally on a practitioners’ observable behaviour in the same way as when the practitioner is doing a retrospective verbalisation and it may also be possible for a colleague to explain some of the non-observable behaviour of the practitioner.
Similar content being viewed by others
References
Arunachalam V, Sasso W (1996) Cognitive processes in program comprehension: an empirical analysis in the context of software reengineering. J Syst Softw 34:177–189
Bainbridge L (1979/1999) Verbal reports as evidence of the process operator’s knowledge. Int J Human Comput Stud 51:213–238
Bartl C, Dörner D (1998) Speechless when thinking: about the role of speech for problem solving. Sprache und Kognition 17:224–238
Bielaczyc K, Pirolli PL, Brown AL (1995) Training in self-explanation and self-regulation strategies: investigating the effects of knowledge acquisition activities on problem solving. Cogn Instr 13(2):221–252
Boren MT, Ramey J (2000) Thinking aloud: reconciling theory and practice. IEEE Trans Prof Commun 43(3):261–278
Brehmer B (1992) Dynamic decision making: human control of complex systems. Acta Psychol 81:211–241
Chi MTH, Bassok M, Lewis MW, Reimann P, Glaser R (1989) Self-explanations: how students study and use examples in learning to solve problems. Cogn Sci 13:145–182
Cook RI, Woods DD (1994) Operating at the sharp end: the complexity of human error. In: Bogner MS (ed) Human errors in medicine. Erlbaum, Hillsdale
Cooke NJ (1999) Knowledge elicitation. In: Durso F, Nickerson RS, Schvaneveldt RW, Dumais ST, Lindsay DS, Chi MTH (eds) Handbook of applied cognition. Wiley, New York, pp 479–509
D’astous P, Detienne F, Visser W, Robillard PN (2000) On the use of functional and interactional approaches for the analysis of technical review meetings. In: Blackwll AF, Bilotta E (eds) Proceedings of the twelfth annual conference of the Psychology of Programming Interest Group (PPIG 12), pp 155–170
Dickson J, McLennan J, Omodei MM (2000) Effects of concurrent verbalisation on a time-critical, dynamic decision making task. J Gen Psychol 127:217
Ericsson KA (2001) Protocol analysis in psychology. In: Smelser N, Baltes P (eds) International encyclopedia of the social & behavioral sciences. Elsevier, Oxford, pp 12256–12262
Ericsson KA, Crutcher RJ (1991) Introspection and verbal reports on cognitive processes—two approaches to the study of thought processes: a response to Howe. New Ideas Psychol 9:57–71
Ericsson KA, Simon HA (1980) Verbal reports as data. Psychol Rev 87:215–251
Ericsson KA, Simon HA (1984) Protocol analysis: verbal reports as data. MIT Press, Cambridge
Erlandsson M, Jansson A (2007) Collegial verbalisation—a case study on a new method on information acquisition. Behav Inf Technol 26(6):535–543
Gibbons FX (1983) Self-attention and self-report: the “veridicality” hypothesis. J Pers Assess 51:517–554
Hammond KR (1993) Naturalistic decision making from a Brunswikian viewpoint: its past, present, future. In: Klein GA, Orasanu J, Calderwood R, Zsambok CE (eds) Decision making in action: models and methods. Ablex Publishing, Norwood
Hammond KR (2000) Judgments under stress. Oxford University Press, New York
Hughes J, Parkes S (2003) Trends in the use of verbal protocol analysis in software engineering research. Behav Inf Technol 22(2):127–140
Jansson A, Olsson E, Erlandsson M (2006) Bridging the gap between analysis and design: improving existing driver interfaces with tools from the framework of cognitive work analysis. Special Issue Int J Cogn Technol Work 8:41–49
Kauppi A, Wikström J, Sandblad B, Andersson AW (2006a) Future train traffic control: control be re-planning. Special issue Int J Cogn Technol Work 8:50–56
Kauppi A, Wikström J, Sandblad B, Andersson AW (2006b) Control strategies for managing train traffic—difficulties today and a vision for the future. Technical report 2006-24. Department of Information technology, Uppsala University, ISSN 1404-3203
Klein GA (1993) A recognition-primed decision (RPD) model of rapid decision making. In: Klein GA, Orasanu J, Calderwood R, Zsambok CE (eds) Decision making in action: models and methods. Ablex Publishing, Norwood
Kuipers BJ, Kassirer JP (1983) How to discover a knowledge representation for causal reasoning by studying an expert physician. In: Proceedings of the American Association of Artificial Intelligence. Morgan Kauffmann Publishers, Inc., Los Altos
Kuusela H, Paul P (2000) A comparison of concurrent and retrospective verbal protocol analysis. Am J Psychol 113(3):387–404
Letovsky S, Pinto J, Lampert R, Soloway E (1987) A cognitive analysis of a code inspection. In: Olson G, Sheppard S, Soloway E (eds) Empirical studies of programmers. Second workshop. Ablex, Norwood, pp 231–247
McIlroy RC, Stanton NA (2010) Observing the observer: non-intrusive verbalisations using the concurrent observer narrative technique. Cogn Technol Work. doi:10.1007/s10111-010-0160-5
McNeese MD (2004) How video informs cognitive systems engineering: making experience count. Cogn Technol Work 6:186–196
Miller JE, Patterson ES, Woods DD (2006) Elicitation by critiquing as a cognitive task analysis methodology. Cogn Technol Work 8:90–102
Narayanan NH, Suwa M, Motoda H (1995) Diagram-based problem solving: the case of an impossible problem. In: Proceedings of the 17th annual conference of the Cognitive Science Society, Lawrence Erlbaum Associates, Hillsdale, pp 206–211
Nisbett RE, Wilson TD (1977) Telling more than we can know: verbal reports on mental processes. Psychol Rev 84:231–259
Pennington N (1987) Comprehension strategies in programming. In: Olson GM, Sheppard S, Soloway E (eds) Empirical studies of programmers, second workshop. Ablex, Norwood, pp 100–113
Pennington N, Nicolich R, Rahm J (1995) Transfer of training between cognitive subskills: is knowledge use specific? Cogn Psychol 28:175–224
Pew RW, Miller DC, Feehrer CE (1981) Evaluation of proposed control room improvements through analysis of critical operator decisions, NP-1982. Electric Power Research Institute, Palo Alto
Polanyi M (1974) Personal knowledge: towards a post-critical philosophy. University of Chicago Press, Chicago
Rasmussen J (1983) Skills, rules, and knowledge; signals, signs, and symbols, and other distinctions in human performance models. IEEE Trans Syst Man Cybern, SMC-13, 257–267
Rasmussen J, Pejtersen AM (1995) Virtual ecology of work. In: Flack J, Hancock P, Caird J, Vicente KJ (eds) Global perspectives on the ecology of human–machine systems. Lawrence Erlbaum Associates, Hillsdale, pp 121–156
Reason J (1990) Human error. Cambridge University Press, Cambridge, pp 53–54
Sarter NB, Woods DD (1995) How in the world did we ever get into that mode? Mode error and awareness in supervisory control. Hum Factors 37(1):5–19
Schenk KD, Vitalari NP, Davis KS (1998) Differences between novice and expert systems analysts: what do we know and what do we do? J Manage Inf Syst 15(1):9–50
Sonnentag S (1998) Expertise in professional software design: a process study. J Appl Psychol 83(5):703–715
Swann WB, Griffin JJ, Predmore SC, Gaines B (1987) The cognitive affective crossfire: when self-consistency confronts self-enhancements. J Pers Soc Psychol 52:881–889
van Der Veer GC (1993) Mental representations of computer languages—a lesson from practice. In: Lemut E, du Boulay B, Dettori G (eds) Cognitive models and intelligent environments for learning programming. Springer, Berlin, pp 21–33
van Someren MW, Barnard YF, Sandberg JAC (1994) The think aloud method: a practical guide to modeling cognitive process. Academic Press, London
Vessey I (1985) Expertise in debugging computer programs: a process analysis. Int J Man Mach Stud 23:459–494
Vicente KJ (1999) Cognitive work analysis: toward safe, productive, and healthy computer-based work. Lawrence Erlbaum Associates, Mahwah
Whyte J IV, Cormier E, Pickett-Hauber R (2010) Cognitions associated with nurse performance: a comparison of concurrent and retrospective verbal reports of nurse performance in a simulated task environment. Int J Nurs Stud 47(4):446–451
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Erlandsson, M., Jansson, A. Verbal reports and domain-specific knowledge: a comparison between collegial and retrospective verbalisation. Cogn Tech Work 15, 239–254 (2013). https://doi.org/10.1007/s10111-012-0223-x
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10111-012-0223-x