Cognition, Technology & Work

, Volume 15, Issue 3, pp 239–254 | Cite as

Verbal reports and domain-specific knowledge: a comparison between collegial and retrospective verbalisation

Original Article


One way to investigate and account for the role of experience in dynamic decision-making tasks is to use a knowledge elicitation method, for example verbal protocols. Recently, methods for verbalisation by other subjects have been suggested as a way to elicit more information on thinking. One of these methods is here investigated in a quasi-experimental study on highly experienced and skilled train dispatchers. Collegial verbalisation is based on the procedure of videotaping practitioners while they perform their normal work tasks in their normal work setting. This is followed up by having a close colleague of the practitioner watch the video recordings and verbalise. The general hypothesis that verbal reports based on collegial verbalisations can provide protocol data that are close to the structure and content of verbal reports based on retrospective verbalisations was investigated. From this systematic comparison it is concluded that collegial verbalisations produce verbal protocols that are close to the retrospective verbal protocols on protocol and topic levels, but not on statement level. From this study, we conclude that the collegial protocols can be used as an independent source of data. It seems possible for a colleague to report verbally on a practitioners’ observable behaviour in the same way as when the practitioner is doing a retrospective verbalisation and it may also be possible for a colleague to explain some of the non-observable behaviour of the practitioner.


Information acquisition Human computer interaction Cognitive work analysis Verbal protocols Verbalisation Knowledge elicitation Domain-specific knowledge 


  1. Arunachalam V, Sasso W (1996) Cognitive processes in program comprehension: an empirical analysis in the context of software reengineering. J Syst Softw 34:177–189CrossRefGoogle Scholar
  2. Bainbridge L (1979/1999) Verbal reports as evidence of the process operator’s knowledge. Int J Human Comput Stud 51:213–238Google Scholar
  3. Bartl C, Dörner D (1998) Speechless when thinking: about the role of speech for problem solving. Sprache und Kognition 17:224–238Google Scholar
  4. Bielaczyc K, Pirolli PL, Brown AL (1995) Training in self-explanation and self-regulation strategies: investigating the effects of knowledge acquisition activities on problem solving. Cogn Instr 13(2):221–252CrossRefGoogle Scholar
  5. Boren MT, Ramey J (2000) Thinking aloud: reconciling theory and practice. IEEE Trans Prof Commun 43(3):261–278CrossRefGoogle Scholar
  6. Brehmer B (1992) Dynamic decision making: human control of complex systems. Acta Psychol 81:211–241CrossRefGoogle Scholar
  7. Chi MTH, Bassok M, Lewis MW, Reimann P, Glaser R (1989) Self-explanations: how students study and use examples in learning to solve problems. Cogn Sci 13:145–182CrossRefGoogle Scholar
  8. Cook RI, Woods DD (1994) Operating at the sharp end: the complexity of human error. In: Bogner MS (ed) Human errors in medicine. Erlbaum, HillsdaleGoogle Scholar
  9. Cooke NJ (1999) Knowledge elicitation. In: Durso F, Nickerson RS, Schvaneveldt RW, Dumais ST, Lindsay DS, Chi MTH (eds) Handbook of applied cognition. Wiley, New York, pp 479–509Google Scholar
  10. D’astous P, Detienne F, Visser W, Robillard PN (2000) On the use of functional and interactional approaches for the analysis of technical review meetings. In: Blackwll AF, Bilotta E (eds) Proceedings of the twelfth annual conference of the Psychology of Programming Interest Group (PPIG 12), pp 155–170Google Scholar
  11. Dickson J, McLennan J, Omodei MM (2000) Effects of concurrent verbalisation on a time-critical, dynamic decision making task. J Gen Psychol 127:217Google Scholar
  12. Ericsson KA (2001) Protocol analysis in psychology. In: Smelser N, Baltes P (eds) International encyclopedia of the social & behavioral sciences. Elsevier, Oxford, pp 12256–12262CrossRefGoogle Scholar
  13. Ericsson KA, Crutcher RJ (1991) Introspection and verbal reports on cognitive processes—two approaches to the study of thought processes: a response to Howe. New Ideas Psychol 9:57–71CrossRefGoogle Scholar
  14. Ericsson KA, Simon HA (1980) Verbal reports as data. Psychol Rev 87:215–251CrossRefGoogle Scholar
  15. Ericsson KA, Simon HA (1984) Protocol analysis: verbal reports as data. MIT Press, CambridgeGoogle Scholar
  16. Erlandsson M, Jansson A (2007) Collegial verbalisation—a case study on a new method on information acquisition. Behav Inf Technol 26(6):535–543CrossRefGoogle Scholar
  17. Gibbons FX (1983) Self-attention and self-report: the “veridicality” hypothesis. J Pers Assess 51:517–554Google Scholar
  18. Hammond KR (1993) Naturalistic decision making from a Brunswikian viewpoint: its past, present, future. In: Klein GA, Orasanu J, Calderwood R, Zsambok CE (eds) Decision making in action: models and methods. Ablex Publishing, NorwoodGoogle Scholar
  19. Hammond KR (2000) Judgments under stress. Oxford University Press, New YorkGoogle Scholar
  20. Hughes J, Parkes S (2003) Trends in the use of verbal protocol analysis in software engineering research. Behav Inf Technol 22(2):127–140CrossRefGoogle Scholar
  21. Jansson A, Olsson E, Erlandsson M (2006) Bridging the gap between analysis and design: improving existing driver interfaces with tools from the framework of cognitive work analysis. Special Issue Int J Cogn Technol Work 8:41–49CrossRefGoogle Scholar
  22. Kauppi A, Wikström J, Sandblad B, Andersson AW (2006a) Future train traffic control: control be re-planning. Special issue Int J Cogn Technol Work 8:50–56CrossRefGoogle Scholar
  23. Kauppi A, Wikström J, Sandblad B, Andersson AW (2006b) Control strategies for managing train traffic—difficulties today and a vision for the future. Technical report 2006-24. Department of Information technology, Uppsala University, ISSN 1404-3203Google Scholar
  24. Klein GA (1993) A recognition-primed decision (RPD) model of rapid decision making. In: Klein GA, Orasanu J, Calderwood R, Zsambok CE (eds) Decision making in action: models and methods. Ablex Publishing, NorwoodGoogle Scholar
  25. Kuipers BJ, Kassirer JP (1983) How to discover a knowledge representation for causal reasoning by studying an expert physician. In: Proceedings of the American Association of Artificial Intelligence. Morgan Kauffmann Publishers, Inc., Los AltosGoogle Scholar
  26. Kuusela H, Paul P (2000) A comparison of concurrent and retrospective verbal protocol analysis. Am J Psychol 113(3):387–404CrossRefGoogle Scholar
  27. Letovsky S, Pinto J, Lampert R, Soloway E (1987) A cognitive analysis of a code inspection. In: Olson G, Sheppard S, Soloway E (eds) Empirical studies of programmers. Second workshop. Ablex, Norwood, pp 231–247Google Scholar
  28. McIlroy RC, Stanton NA (2010) Observing the observer: non-intrusive verbalisations using the concurrent observer narrative technique. Cogn Technol Work. doi: 10.1007/s10111-010-0160-5
  29. McNeese MD (2004) How video informs cognitive systems engineering: making experience count. Cogn Technol Work 6:186–196CrossRefGoogle Scholar
  30. Miller JE, Patterson ES, Woods DD (2006) Elicitation by critiquing as a cognitive task analysis methodology. Cogn Technol Work 8:90–102CrossRefGoogle Scholar
  31. Narayanan NH, Suwa M, Motoda H (1995) Diagram-based problem solving: the case of an impossible problem. In: Proceedings of the 17th annual conference of the Cognitive Science Society, Lawrence Erlbaum Associates, Hillsdale, pp 206–211Google Scholar
  32. Nisbett RE, Wilson TD (1977) Telling more than we can know: verbal reports on mental processes. Psychol Rev 84:231–259CrossRefGoogle Scholar
  33. Pennington N (1987) Comprehension strategies in programming. In: Olson GM, Sheppard S, Soloway E (eds) Empirical studies of programmers, second workshop. Ablex, Norwood, pp 100–113Google Scholar
  34. Pennington N, Nicolich R, Rahm J (1995) Transfer of training between cognitive subskills: is knowledge use specific? Cogn Psychol 28:175–224CrossRefGoogle Scholar
  35. Pew RW, Miller DC, Feehrer CE (1981) Evaluation of proposed control room improvements through analysis of critical operator decisions, NP-1982. Electric Power Research Institute, Palo AltoGoogle Scholar
  36. Polanyi M (1974) Personal knowledge: towards a post-critical philosophy. University of Chicago Press, ChicagoGoogle Scholar
  37. Rasmussen J (1983) Skills, rules, and knowledge; signals, signs, and symbols, and other distinctions in human performance models. IEEE Trans Syst Man Cybern, SMC-13, 257–267Google Scholar
  38. Rasmussen J, Pejtersen AM (1995) Virtual ecology of work. In: Flack J, Hancock P, Caird J, Vicente KJ (eds) Global perspectives on the ecology of human–machine systems. Lawrence Erlbaum Associates, Hillsdale, pp 121–156Google Scholar
  39. Reason J (1990) Human error. Cambridge University Press, Cambridge, pp 53–54Google Scholar
  40. Sarter NB, Woods DD (1995) How in the world did we ever get into that mode? Mode error and awareness in supervisory control. Hum Factors 37(1):5–19CrossRefGoogle Scholar
  41. Schenk KD, Vitalari NP, Davis KS (1998) Differences between novice and expert systems analysts: what do we know and what do we do? J Manage Inf Syst 15(1):9–50Google Scholar
  42. Sonnentag S (1998) Expertise in professional software design: a process study. J Appl Psychol 83(5):703–715CrossRefGoogle Scholar
  43. Swann WB, Griffin JJ, Predmore SC, Gaines B (1987) The cognitive affective crossfire: when self-consistency confronts self-enhancements. J Pers Soc Psychol 52:881–889CrossRefGoogle Scholar
  44. van Der Veer GC (1993) Mental representations of computer languages—a lesson from practice. In: Lemut E, du Boulay B, Dettori G (eds) Cognitive models and intelligent environments for learning programming. Springer, Berlin, pp 21–33Google Scholar
  45. van Someren MW, Barnard YF, Sandberg JAC (1994) The think aloud method: a practical guide to modeling cognitive process. Academic Press, LondonGoogle Scholar
  46. Vessey I (1985) Expertise in debugging computer programs: a process analysis. Int J Man Mach Stud 23:459–494CrossRefGoogle Scholar
  47. Vicente KJ (1999) Cognitive work analysis: toward safe, productive, and healthy computer-based work. Lawrence Erlbaum Associates, MahwahGoogle Scholar
  48. Whyte J IV, Cormier E, Pickett-Hauber R (2010) Cognitions associated with nurse performance: a comparison of concurrent and retrospective verbal reports of nurse performance in a simulated task environment. Int J Nurs Stud 47(4):446–451CrossRefGoogle Scholar

Copyright information

© Springer-Verlag London Limited 2012

Authors and Affiliations

  1. 1.Department of Information TechnologyUppsala UniversityUppsalaSweden

Personalised recommendations