1 Introduction

Accelerating learning by capturing, documenting, and utilising professional cognitive expertise where technological excellence is fundamental to success is a critical challenge for many high-reliability, global organizations. As Hoffman et al. (2014) highlight, Goffee and Jones (2007, p 1) state that if an organization was able to document embedded knowledge from their clever people’s minds—all it would need is an improved knowledge-management system. Failing to capture such tacit knowledge that they argue is one of the greatest disappointments of knowledge management initiatives.

The naturalistic decision-making (NDM) community has reported the advantages of applied cognitive task analysis (ACTA) and associated cognitive task analysis (CTA) techniques (Hoffman and Militello 2008; Roth 2008; Militello et al. 2010) for capturing and developing our insight of cognition. Notably, these techniques have also begun to steadily grow in other research areas of organizational behaviour and management practice (Gore and Ward 2017; Gore and McAndrew 2009; McAndrew and Gore 2012, 2013; Gore et al. 2015a, b, c; Osland 2010; Osland et al. 2013).

Tofel-Grehl and Feldon (2013) have also noted the growing popularity of cognitive task analysis (CTA) in both research and practice, and completed a meta-analysis of studies to examine the value of such training. They report that though their meta analysis is limited due to its small number of studies, the effect of CTA instruction is large (Hedges’s g = 0.871) Reports which focus upon the training of practitioners to adopt such techniques, however, are less well documented (Gore et al. 2015a, b, c; Potworowski and Green 2016). The work presented here,, therefore, continues to examine the importance of the role of academics translating methodological research developments for impact and exploration of and in knowledge management practice (Anderson 2007; Gore and Conway 2016). In addition, we aimed to ensure that aspects of cognitive expertise that are difficult to articulate are documented with clear application validity.

2 Unpacking expertise

Researchers have endeavored to uncover the concept of expertise for several decades through laboratory-based examination and naturalistic investigation (Ericsson and Smith 1991; Feltovich et al. 1997), exemplified by the naturalistic decision-making (NDM) framework, eliciting that experts learn in four ways (Koehler and Harvey 2004):

  1. 1.

    engaging in deliberate practice, often setting goals, and criteria for evaluation;

  2. 2.

    compiling extensive experience banks;

  3. 3.

    obtaining feedback that is accurate, and timely;

  4. 4.

    enriching their experiences by reflecting on their experience and lessons learnt from mistakes.

Several categories of knowledge discriminate experts from others, including novices, as both declarative (“what” static knowledge is known) and procedural knowledge (“how to” complete a task), are more apparent in experts who draw from a wide personal knowledge base. In addition, strong perceptual skills are purported as another essential component of expertise (Klein and Hoffman 2008); these include mental models; adept sense making of associations; the ability to run mental simulations. Such enriched mental models enable experts to spot anomalies and problems quickly and formulate information seeking tactics to manage uncertainty. NDM research further suggests that experts’ metacognitive processes ensure that they take into account their own individual strengths and limitations when processing information and generating knowledge (Ward et al. 2018).

Hence, the primary aim of this project was to document to what extent CTA training and participating with, in this instance, ACTA would enable professionals to share their expertise-based cognition. Notably, we offer suggestions for adapting and improving the CTA methods for management practitioners and highlight the importance of developing socio-cognitive competence.Footnote 1 This latter area, as yet, has been unexplored and echoes Hoffman et al. (2014) call for further explorations of the social aspects of CTA. We also note the importance of translating the findings from CTA for knowledge management, future scenario planning, and management learning development.

3 Organizational context

The participants’ workplace is a global organization in the formulation, manufacture, and marketing of petroleum additives for lubricants and fuels with a long history of innovation in decision-making. The organization has effectively used scenario planning for more than 45 years (see Wilkinson and Kupers 2013 for a recent review) initially through facilitated dialogue in which managers’ assumptions could safely be shared, questioned, and challenged and subsequent focus on the strategic. Scenario practice in part of the organization began by exposing and questioning the future and facilitated dialogue in which managers’ assumptions could safely be shared, questioned, and challenged. Many business units and different organizational functions besides strategy and finance went on to develop scenarios which focussed upon the big picture. In the 1980s, however, a refocus was required which concentrated on ‘deep listening’ to uncover uncertainties, probing the core concerns of leaders. At this time, more than 100 one-to-one interviews were completed with petroleum executives, each lasting 3–4 h. This approach has continued and is reported as being effective with the vast majority of the resultant scenarios being shared across the organization. Scenarios have continued to evolve and scenario developers aim to keep scenarios relevant and challenging learning tools which have impact upon organizational thinking and cognition.

This work has also had a renaissance outside the organization with many companies expecting to use scenario planning to add value via (1) an enhanced capacity to perceive change, and to (2) interpret and respond to change, (3) to influence other actors, and (4) enhance the capacity for organizational learning. Scenario planning continues to help shape-related global thinking and has had a huge influence on how businesses, governments, and organizations think about the future (Wilkinson and Kupers 2013, p 121).

Set within this innovative organizational culture, the authors were invited to explore within a much wider organizational project on knowledge management, how best expert cognition in engineering expertise could be elicited, documented, and shared, aiming to provide knowledge which would accelerate novice engineers’ complex cognitive decision-making processes. The organization examined here was concerned with capturing expertise at the level of the individual. A key challenge was to ensure the practitioners’ accurately captured cognition to maintain continuous knowledge transfer within this highly qualified workforce. This paper documents the process of training transfer and provides illustrative results of an engineering practitioner’s ACTA. The expert cognition associated with managing uncertainty is highlighted (Lipshitz and Strauss 1997) and aspects of hot/sensoryFootnote 2-based cognition explored.

4 Applied cognitive task analysis

The ACTA technique is intended to support the identification of the key cognitive elements required to perform a task proficiently, albeit in a way useful to practitioners (Militello and Hutton 1998). As previously discussed in a research methods note to the psychological academic and practitioner community (Gore and McAndrew 2009), pragmatic work has successfully used ACTA to understand expertise in a wide range of areas including: fire fighting (Klein et al. 1989) recruitment, (Gore and Riley 2004), weather forecasting (Hoffman et al. 2006), clinical medical contexts (Militello and Hutton 1998; Militello et al. 1997), aviation and command and control operating procedures within naval and military environments (Drury and Darling 2008), and financial decision-making (McAndrew et al. 2009; McAndrew and Gore 2010, 2012, 2013). Crandall, Klein, Hoffman, Militello, and other researchers in the NDM community have adapted these methods, assisting knowledge transfer and training amongst expert domains. Such rigorous development has provided professionals with domain contextualised knowledge based on extensive real-life scenarios (Hoffman 1992; Klein and Crandall 1995; Schraagen et al. 2000, 2008). This has meant that training for cognitively demanding tasks within domain-specific areas has more explicitly documented tacit knowledge, skills, and expertise-based professional practice. Gore (2013) has previously argued that the ACTA techniques are evidence-based insights (Briner and Rousseu 2011) into domain-specific knowledge which is unlikely to be documented in the conventional training programmes. Within the management literature, evidence-based management encourages practitioners to use scientific information and stakeholder expertise to make more informed judgments about decision-making. Hence, the type of knowledge elicited by the ACTA techniques is often undocumented experiential, evidence-based insights.

Whilst much of the work completed in judgment and decision-making has a history of examining the cognitive errors associated with decision-making (Kahneman et al. 1982), the theoretical origins of the decision-making perspective adopted in this study focus on real-world decision-making, i.e., naturalistic decision-making (NDM) (Klein et al. 1993). This perspective emphasises expertise, positive aspects of cognition and the study of expert intuition.

The cognitive areas ACTA aims to examine in depth are: (1) difficult judgments and decisions; (2) attentional demands; (3) critical cues and patterns; (4) problem solving strategies (2009). The ACTA methods are distinct from other techniques which explore tacit cognitive processes as they employ a range of knowledge elicitation and representation techniques that logically progress, combining the elicitation of task-specific, expertise-based knowledge. ACTA’s knowledge representation techniques deliver a systematic schematic for arranging and juxtaposing cognitive information (cognitive mapping), whilst the knowledge elicitation techniques comprise of in-depth interviews (and occasionally observations).

5 Method

After assessing organizational needs via a pilot study stage, one of this studies included first day (7 h) briefing about the use of ACTA techniques which was provided for a small group of professionals with different areas of engineering expertise. During a second day, one of the authors trained three engineers to use a selection of the ACTA techniques (7 h) (Militello and Hutton 1998).

Stage two: a 3 day longitudinal (21 h) training event completed over 3 months was provided by the authors/CTA instructors for engineering professionals (N = 22, 5 female, 17 male). The professionals had a range of engineering expertise in management, manufacturing technology, finance, human resources, information technology, product development, and operation management. Many of the participants were senior research scientists educated to doctoral level, all with 5–15 years of domain-specific experience (domain experts).

5.1 Procedure—training the participants

First, the researchers completed a task diagram and knowledge audit to illustrate the interview techniques associated with stage one and two of ACTA. This process was stopped and re-started in order for the engineers to ask questions and clarify the process. The first stage of ACTA, the production of a task diagram, provides the interviewer with a broad overview of the task. This interview helps to identify areas requiring complex cognitive skills which can be examined in depth in stage 2 of the process: the knowledge audit. Task diagrams were completed for key areas of engineering work involving cognitive complexity to identify elements seen as essential by the expert engineers. The researchers prior need analysis which had documented that such information was not currently documented meaningfully in training procedures, but that this was potentially vital to train and induct novice engineers. The next step was an in-depth interview process conducted between participants, the researchers acting as observers and facilitators as necessary.

The interviewers’ (practitioner engineers’) began by asking the interviewees (expert engineers) to break down a cognitive task related to their expert job role into three-to-six steps. These steps/stages were documented by a cognitive map showing three-to-six connecting nodes/circles which relate to the task. The interviewer then asks which step/stage of the task is most cognitively challenging and why, maybe, novices find this difficult. This first stage can take up to 30 min to complete. The interviewer is encouraged to check on understanding with the expert to ensure that she or he agrees that the task diagram accurately provides a broad overview of the task. Together, the interviewer and interviewee identify which element of the task is most cognitively complex and takes most thinking, judgment, and decision-making. This stage of the task is then explored and probed in great detail by completing stage two of ACTA, the Knowledge audit.

Second, the engineers practiced Knowledge Audit techniques with each other and documented their understanding of complex cognition. Again, a stop–start approach was adopted to facilitate the question technique and the documentation of knowledge elicited. The knowledge audit focuses upon a cognitive sub-task elicited from the task diagram and is well documented in the research literature in expert–novice differences (Crandall et al. 2006). A series of well-developed questions which are based on extensive research on expert thinking form the focus of the knowledge audit (Militello and Hutton 1998; see Table 1). This stage of the ACTA is iterative and can take up to 2 h to complete, eliciting lived stories and scenarios from the experts being interviewed.

Table 1 Knowledge audit probes.
Full size table

Finally, a Cognitive Demands table was completed by the engineers, providing an analytical summary of data elicited. The cognitive demand table is summary which provides an analysis of key aspects of expert cognition within the domain context and also clearly illustrates which aspects novices may find difficult. By documenting difficulties and capturing key cues and strategies for success, tacit knowledge is thus illustrated. In addition to providing training in the ACTA techniques, we also provided a briefing about theoretical issues in decision-making and an exercise to facilitate active listening and questioning skills, as most of the participants had not previously had experience of research-based interviewing and had a genuine interest in the theoretical roots of the CTA methods. All participants had no prior experience of intensive research-based interviewing and completed a questionnaire evaluation of their training experience. This questionnaire was developed aiming to evaluate cognitive, skill-based, and affective learning outcomes (Kraiger et al. 1993), providing construct-orientated evidence of validity. A peer evaluation of the application validity of the cognitive demands tables and training scenarios produced from the interviews was also completed in collaboration with experienced analysts. In addition, data were checked with other engineering experts to establish how far they agreed with the cognition elicited and most importantly how far they concurred that this tacit information was not currently available to novices.

6 Results

Our in situ training observations paired with the feedback evaluations gathered at the end of each training day show that the engineers found the process of interviewing and being interviewed using the ACTA techniques initially both cognitively and emotionally challenging. The socio-cognitive capabilities of the interviewer as facilitator of cognitive knowledge elicitation, and the interviewee, to take time to reflect in a thoughtful, reflexive, meaningful and organised way, were key to the success of the interviews. As an example, we observed how and then sensitively questioned why some of the ACTA outputs, including the task diagrams and knowledge audits, were more truncated and less nuanced for one sub-group of participants than for others having taken less time to complete during the training on day 2. During follow-up questioning at the end of the training day, the interviewee felt confident that they had participated, and given ‘robust’ information, the interviewee implied that answers had been monosyllabic and at points condescending, giving a little insight into actual expert cognitions. They attributed this to a perceived difference in status and seniority between the parties involved, as well as an (implicit) attitude on behalf of the interviewee that their mere presence had been enough. In contrast, we observed other dyads with richer outputs, where the interview process was a two-way and constructive. Questioned at the end of the first interview period, all participants found the training involved a great deal of focus requiring regular breaks to counteract fatigue, and that a key mechanism for counteracting such fatigue was the interviewer’s skill to vary questions to sensitively probe information, as well as summarising the information at key points which helped both parties to ensure that key information had been understood and accurately recorded. They also voiced that whilst they found the briefing on socio-cognitive factors as well as general interview and feedback guidance informative, they would have appreciated a ‘crib sheet’ for how exactly they could implement and embed this guidance in the ACTA technique as such.

As a result of these observations, the authors and engineers co-created innovative practice guidance, shown in Table 2, to maximise the task diagram and knowledge–elicitation phase of ACTA and recognise the importance of socio-cognitive competence/insight. This innovation assisted participants and added to the language and positive social context for knowledge transfer, providing assurance and reference for best practice. This guidance sheet openly acknowledged the perceived difficulty, but also occasional frustration of the ACTA process, which helped to normalise that these are inherent features of a challenging processes, rather than due to deficiencies of the parties involved in the process.

Table 2 Accessing expertise in the field: top tips for getting rich data/developing socio-cognitive competence/insight from ACTA
Full size table

The tasks covered by the managers/engineers varied according to their organizational role and included everything from plant trial management; complex decisions surrounded choice of experiments for fuel testing; running a new project; improving supply security; to preparing to meet a new customer.

The task illustrated here (see Table 3) is from an expert process engineers analysis of the key cognitive demands involved in the task of completing an “initial manufacturing plant trial start-up”. When new petroleum additives are developed, teams of research engineers with professional expertise in chemistry and physics run pilot or ‘start-up’ trials within the manufacturing plant. This process has a high degree of risk associated with it, and can be expensive and time-consuming, in terms of both health and safety and continued product innovation success. Process and product development are also subject to unique legal confidentiality agreements in this area of engineering. The task of setting up and monitoring a plant trial, therefore, involves a high degree of macro-cognitive complexity within an NDM-type framework (Orasanu and Connolly 1993). The macro-cognitive complexity, i.e., “the cognitive adaptation to complexity” (Klein and Hoffman 2008), can involve several experts managing risk and uncertainty, making sense of their dynamic environment, with high stakes and shifting, ill-defined goals under time-pressured situations.

Table 3 Illustrative cognitive demand table for plant trials/start-up
Full size table

Each of the engineers reported that the knowledge elicited, including key cues for improving situation awareness, and scenario planning had rarely been documented in such a pragmatic way previously. As illustrated, the results of an independently completed cognitive demand table completed by an experienced CTA researcher were comparable to the engineers’ interpretations (see Table 3) focussing upon the task of completing a petroleum development plant trial. Here, we see clear areas of cognitive complexity elicited by the ACTA method for the expert plant trial engineer including: effective communication; planning ahead (responding to action-feedback loops); noting key technical cues and strategies. This summary was completed after reviewing the outputs from the ACTA interviews with three of the expert participants. They used a content analysis-based approach to highlight the key features of the task discussed by the engineers. They checked their summary with other experts in the organization before presenting it for further refinement to be used within future training scenarios.

In addition to documenting task-specific mental models, detailed perceptions of cues, and strategies, an important feature which emerged to the surprise of the engineers was the importance of hot/sensory-based cognition. For example, several engineers described noticing peculiar smells in the mornings which resulted in adjusting the manufacturing process before the new petroleum additive was destroyed, making significant economic savings and avoiding potential hazards. The completed summary analysis/cognitive demand table was then later used as a base for developing computer-based training which captured the lived expert realities of successful organizational tasks. The scenarios developed included a wide range of operational decision-making procedures through to highly scientific and engineering tasks. Each scenario clearly documented representations of expert mental models and was also subject to peer review from other expert engineering practitioners which assisted knowledge transfer.

Feedback from within the organization has been positive with the practitioners wanting to utilise more CTA-based training to continue to support organizational learning. The evaluation of the training suggested that the majority of participants felt that the ACTA techniques were “a very effective and efficient framework for helping articulate how experienced colleagues do specific tasks, provided structured learning and clear training outcomes”. One participant, however, suggested that applying ACTA may be problematic in terms of “drilling down to the right level of granularity of a task to access the most specific tacit knowledge”.

7 Discussion

Undertaking ACTA, examining its underpinning principles and evaluating its effectiveness require a longitudinal approach which with this study that we begin to offer. We demonstrate that ACTA is a promising vehicle to support knowledge management and transfer, and that, even with relatively short instruction time, practitioners can avail themselves of the technique and apply it in a peer-to-peer context. We reflect that an action-research centric paradigm is essential, however, as the observations on this case study-based research elicit that the emotional and socio-cognitive aspects of ACTA are important, and need to be facilitated and legimitised through relevant and tailored support. As the in-depth interview techniques are intensive and access System 2 thinking/cognition [slow, controlled, and analytical processing (Kahneman and Klein 2009)] to reflect upon System 1 thought/cognition processes (characterized by fast, heuristic-based, and emotional processing), careful interpretation, and mentoring and training are required.

We note as a limitation of the present paper that it was beyond its scope to undertake a truly longitudinal evaluation to determine and apply an internal and/or external benchmark to verify to what extent knowledge management as well as training and induction for novices had been improved. We also wish to emphasise that findings of the substantive content of the knowledge elicited by the engineers were not the primary aim of this paper and that the results presented are limited illustrations. What is important here is the contribution to training and improving training in general for practitioners (and potentially students) who wish to use ACTA techniques to reflect upon and elicit their expertise.

8 Conclusion

This work is original in its application as a few studies document such applied inclusion of practitioners in the co-construction of knowledge. The study demonstrates the utility of applying ACTA to the domain of utilities management and engineering in a macrocognitive framework through the elicitation of scenarios to assist novice engineering professionals. In turn, these will serve to raise situational awareness in relation to specific tasks and clearly define cognitive complexity in an organizational-based repository of training scenarios.

Our contribution to the development of CTA methods and knowledge management impact here strongly highlight the importance of recognising, managing, and providing training which supports practitioners to develop their socio-cognitive competence and insight, alongside knowledge elicitation documentation and transformative knowledge management solutions. The utility of CTA is ultimately dependent on the human capacity to elicit and articulate expertise, it is our contention that increased emphasis needs to acknowledge the potential intellectual and interpersonal challenges involved in this process. The complexities for resulting knowledge transfer provide an interesting research agenda with a range of potential theoretical and pragmatic contributions. (Grossman et al 2014; Salas et at 2015). Theoretically, exploration of the links between developing the reflexive System 2 thinking that the ACTA techniques require to reflect upon System 1 thinking offers an exciting research agenda; future studies might address this by taking a comparative approach to distil which elements of ACTA are particularly effective in this context, and how practitioners can best be supported in the application of relevant techniques. We hope that other researchers adapt and adopt the process offered in this paper and thus contribute to the growing NDM literature, training, and understanding expertise.