1 Introduction

To understand the complexity of interactive technology use in practice, field studies are essential. However, entering an unfamiliar, complex environment can be overwhelming for an analyst without relevant background knowledge and experience (Wong and Blandford 2003). The analyst may experience difficulties in seeing a clearly defined structure in these systems (Norman 2011) and may direct their attention to unimportant aspects of the environment while neglecting essential ones.

Theories and frameworks have been proposed to provide a structure for data gathering and to offer scaffolding for the consolidation of results. Distributed Cognition (DCog: Hollan et al. 2000) is a theory that has been shown to deliver valuable insights into the design and use of technology (e.g. Hazlehurst et al. 2007, 2008; Xiao 2005). However, most accounts of the application of DCog omit details of the process by which data were gathered, structured, or analysed, making it difficult to follow or replicate the details. Halverson (2002) argues that the lack of explicitly named constructs or methods in DCog has impeded take-up and use of this theory; whether or not this is true, use has historically required substantial craft skill. Indeed, Benyon (2002, p. 191) explicitly omits DCog from his review of representations for human-centred system development “because real methods have not appeared based upon it”. In response to this, methods for applying DCog in practice have been developed, to facilitate learning and more clearly articulate principles of DCog. For example, Wright et al. (2000) propose the Resources Model for analysing an interface; Galliers et al. (2007) propose Determining Information flow Breakdown (DIB); and Furniss and Blandford (2006) propose Distributed Cognition for Teamwork (DiCoT) for analysing devices and small team working. All of these frameworks aim to make DCog more accessible to researchers and practitioners by adding structural support to the application of DCog. DiCoT presents concepts and principles of DCog linked with models proposed for Contextual Inquiry (CI: Beyer and Holtzblatt 1998).

The aim of the study reported here was to better understand how a novice analyst learns and applies CI and DiCoT. A complex healthcare setting (the work of anaesthetists in an operating theatre) was taken as a fieldwork setting for the study. Within that setting, the aim of the study was to understand the interactions of the anaesthetists, how the design of procedures and of the environment supports work, and particularly how they used infusion devices. The study had multiple aims: to better understand the interactions of anaesthetists with infusion devices and with each other; to identify requirements for improved learning resources for DiCoT; and to better understand and make visible the process of learning and using CI and DiCoT in a complex setting. The focus of this paper is on the last of these aims.

2 Background

Distributed Cognition is a theoretical perspective that views cognition as being distributed between people and artefacts within a work system. It focuses particularly on how information is represented, communicated, and transformed within the system (Hutchins 1995; Hollan et al. 2000). DCog analysis relies on ethnographically gathered data (Halverson 2002)—typically involving both observation and interviews. Halverson (2002, p. 249) argues that the utility of DCog lies in “its theoretical commitment to examine [the] broader socio-cultural-technical system, which is necessary for the collaboration between individuals mediated by artifacts”. This use of DCog in the operating theatre is exemplified by the work of Hazlehurst et al. (2007), who studied team coordination in the cardiac operating theatre, focusing particularly on how situation awareness is maintained between surgeons and perfusionist (but paying relatively little attention to the anaesthetists who are the focus in the study reported here).

As well as focusing attention on the broader system, DCog has also been applied to the details of individual user–device interactions. For example, Nemeth et al. (2004) analyse how the design of artefacts in the operating theatre supports or hinders communication and coordination between team members. Their focus is on artefacts, such as schedules and display boards, whose primary function is in coordinating work over an extended period of time, rather than those, such as patient monitors and infusion devices, whose primary purpose is for managing patient care within the context of an ongoing operation.

2.1 Learning a method: attributes and process

First, a word on terminology: we use the term “theory” to refer to DCog, recognising that the application of theory to any particular situation will require a process. Processes for analysis—whether or not grounded in theory—are widely referred to in the literature as “methods,” so where appropriate we refer to CI and DiCoT as “methods”. However, neither is highly proceduralised, so at other times we refer to them as “frameworks” (for structuring data gathering and analysis), to disambiguate them from the research method of this study.

The focus of this study is on learning to apply CI and DiCoT. There have been few structured studies of learning any methods in HCI, and we are not aware of any studies that present the details of how people have learned to apply CI or conduct a DCog analysis: the closest is the work of Sellberg and Lindblom (2014), described below. The previous work that most strongly influenced the design of this study was that of John and Packer (1995). They provide details of how one relative novice learned and used the cognitive walkthrough method using a case study approach. The study participant kept a detailed diary of actions, difficulties, insights, and discussions held while learning the method and applying it. Importantly, this details the process, which is absent from many studies. They argue that detailing process: (1) allows one to understand how a method was used and how insights were gained; (2) gives a better understanding of what to expect when setting out to use a method; and (3) gives meaningful feedback to method developers to improve their techniques. All of these considerations are of interest in this study, although the focus of reporting is on the first two.

2.2 Contextual Inquiry (CI)

In order to structure his learning, the novice analyst in this study went through an explicit stage of learning and applying CI (Beyer and Holtzblatt 1998).

Contextual Inquiry is part of the Contextual Design methodology proposed by Beyer and Holtzblatt (1998). CI uses contextual interviews: interviews with people within their working context. During an interview session, the analyst observes the interviewee and asks questions for clarification. Beyer and Holtzblatt (1998) assert that integration of observation and interview gives access to unarticulated knowledge and allows the analyst to develop a deeper understanding of the domain. Four interaction principles are proposed to guide the interviews: partnership, focus, interpretation, and context. The principle of “partnership” states that the interviewer should try to establish a master–apprenticeship relationship with the interviewee, making it clear that s/he has inferior knowledge about the domain but a high interest in understanding it. At the same time, s/he should try to maintain a constant “focus” on her/his unit of analysis, avoiding shifts. Following every contextual interview, the collected data are “interpreted” and a pictorial representation is drawn. Through this, the results are more accessible for discussions within the design team to build an understanding of “context”.

Five work models are created for every interviewee, each highlighting a specific aspect of work. These individual models are unified to five general models after all interviews have been conducted:

  • A flow model shows the overall organisation and coordination of the workflow;

  • A sequence model gives a description of tasks;

  • An artefact model represents objects used for work;

  • A cultural model provides a schematic overview of informal and broad influences on actors; and

  • A physical model illustrates the workplace’s physical structure.

Contextual Inquiry has been applied in various contexts, e.g. to study the introduction of ATMs in India (De Angeli et al. 2004) and the design of mobile exhibition services (Fouskas et al. 2002). In healthcare, CI has been used to inform the design of a clinical trials screening tool (Gennari and Reddy 2000) and in the area of telecardiology (Gil-Rodríguez et al. 2007), but no details of the modelling are presented in either of these cases. In contrast, Coble et al. (1995) report a detailed account of contextual interviews, model building, and consolidation for requirements capture for a clinical information system. They present a clear account of the value of gathering user requirements in this way, including identifying some unexpected clinician behaviours and the value of getting clinicians involved such that they felt like partners in the development programme; however, they present few details of how models were constructed, focusing more on the identification of user requirements.

Sellberg and Lindblom (2014) present a comparative study of CI (set within its broader context of Contextual Design) and two frameworks that are based on DCog (DIB, which focuses on information communication breakdowns, and CASADEMA, which focuses more on DCog theory and less on structured method). They first offer a review of the three frameworks based on prior literature on each, concluding that CI’s lack of theoretical basis may make it “uninformed and fraught with bias” (p. 471), but that it is better framed to have application power (i.e. relevance to design practice) than either of the DCog-based frameworks, which have better descriptive power. They subsequently offer a comparison based on an empirical study; in their study, all three frameworks were applied to the same dataset based on 30 h of observations in three dental clinics, focusing on work at the front desk. It appears that the main analysts were the authors, although this is not clearly stated. The CI analysis is one of the clearest available accounts of the experience and outcomes of applying this framework. The authors note that learning CI was much more difficult than the literature had led them to expect, e.g. in determining the appropriate level of detail for a flow model and in translating abstract guidance into details of the practice of modelling.

2.3 Distributed Cognition for Teamwork (DiCoT)

Analyses using DCog have traditionally been performed using cognitive ethnography (e.g. Hutchins 1995). DiCoT (Furniss and Blandford 2006) was developed as a semi-structured framework to help apply DCog, focusing on small team interactions. The original version of DiCoT, which formed the basis for this study, did not specify the data gathering technique, but assumed a mix of observation and situated interviews very similar to CI. A DiCoT analysis involves constructing five models (Blandford and Furniss 2006) to describe activities as observed:

  • The information flow model shows how information flows in tasks and between actors.

  • Artefact models focus on the structure of tools and representations, and how they affect cognitive work in practice.

  • Physical models focus on the layout of the environment, e.g. a desk or a room, and how this impacts the propagation of information.

  • The social model focuses on the social relationships, responsibilities, knowledge, and goal sharing between individuals, and how this influences the computation of the system.

  • The evolutionary model shows how the system has changed over time.

Each model is associated with a set of DCog concepts and principles, which serve as a checklist or vocabulary for analysing the model in terms of DCog theory. These enable the analyst to describe how the system “works” in DCog terms and to identify points of vulnerability, ambiguity, and strength in system performance. The full list of principles is included as “Appendix 1” to this paper.

Principles associated with information flow are concerned with how information moves and is transformed in the system (and any obstacles to effective information flow)—both formally and informally; whether any agent acts as an information hub (and if so, how that role is facilitated, and consequences of the hub being unavailable for any reason); and how interruptions are managed (in terms of triggering new activity or disrupting existing activity).

Principles associated with the physical model are concerned with how space, layout, and movements support cognition and action and how the design of the system supports situation awareness (Fioratou et al. 2010; Hazlehurst et al. 2007).

Similar principles are associated with artefact models; artefacts can also be moved around and used in various ways, so further principles are concerned with how they mediate communication and can be appropriated to support cognition in other ways.

The social structures and evolutionary models are less well developed, but associated principles are concerned with understanding how cognition is distributed across social groups and how both artefacts and expertise have evolved into their current forms.

Distributed Cognition for Teamwork has been applied to analyse various complex settings, including: an ambulance dispatch control room (Blandford and Furniss 2006; Furniss and Blandford 2006); the practices of an agile software development team (Sharp et al. 2006); mobile healthcare work (McKnight and Doherty 2008); a medical equipment library (Werth and Furniss 2012); infusion pump use in an Oncology Day Care Unit (Furniss et al. 2011); infusion practice in an ICU (Rajkomar and Blandford 2012); and the management of home haemodialysis (Rajkomar et al. 2013). These case studies were all resources used by the novice analyst to learn DiCoT. They exemplify DiCoT’s use, but there is, as yet, no single authoritative source for its practice.

2.4 Summary comparison of CI and DiCoT

Table 1 summarises the key features of CI and DiCoT.

Table 1 Comparison of CI and DiCoT
Full size table

In the study reported here, the CI analysis stopped at the point where the models had been used to describe work as performed, rather than progressing to any new design generation.

3 Study method

As noted above, the aim of this study was to better understand the process of learning to apply CI and DiCoT in a complex setting. This is a single case study. As Gerring (2007, p. 1) argues, “Sometimes, in-depth knowledge of an individual example is more helpful than fleeting knowledge about a larger number of examples”. Similarly, discussing idiographic research (which focuses on the details of particular cases rather than generalising across populations), Tsoukas (1989, p. 551) notes that “Idiographic research conceptualizes the causal capability of structures, while at the same time it sheds light on the contingent manner through which a set of postulated causal powers interact and gives rise to the flux of the phenomena under study”. The work reported here is in this idiographic tradition, seeking to gain understanding through an in-depth case study rather than by generalising across cases. As noted above, it replicates as closely as possible the study design of John and Packer (1995), which was also a single in-depth case study.

This work was made possible by the arrival of the first author, EB, as an intern in the research laboratory of the other authors. EB had a background in Psychology, and 1 year of education in HCI, including in ethnography, but no prior exposure to either CI or DCog. He also had an interest in health technology, but no prior experience of observational studies in operating theatres or knowledge of anaesthesia practice. As a consequence, he was well placed to conduct an in-depth diary study of learning and using DCog in an unfamiliar, complex setting.

The three authors of this paper, who we will refer to as EB, DF, and AB, took different roles in this work. EB’s role was to systematically learn and apply CI and DiCoT in the chosen context of study, maintaining a detailed diary of activities, difficulties, insights, and findings as he did so. DF took a mentoring role, advising as needed on the application of both CI and DiCoT. Example entries in EB’s diary illustrate the relationship:

In the talk with [DF] I discussed these questions and it helped me a lot to talk about it. Furthermore [DF] had some really good ideas. We concluded that CI is an orientation, and should not be used dogmatically.

Diary Entry, Friday, week 2 of part 1

Afterwards I discussed with [DF] about the difficulties I had during interviewing and modeling. I am very unhappy with some of the models and think that they do not map the context properly. However, [DF] argued that a lot of the context information is in my head, and that the models are only a representation of it.

Diary Entry, Wednesday, week 5 of part 1

The substance of discussions between EB and DF was recorded and subsequently included in the analysis for reflection. Our strategy was that EB should work as far as possible from published sources, so as to make the learning process as inspectable as possible. He received guidance from DF when needed, but much of this concerned gaining access to, and working in, the study setting, rather than details of how to gather and analyse data. AB oversaw the project, negotiated access to the study setting, provided strategic direction, and conducted a retrospective independent analysis of all the data generated by EB.

The “object of study” was the process and outcome of completing CI and DiCoT analyses. To complete the analyses, it was necessary to gather data in the chosen work context; this was operating theatres of a busy hospital in London. The focus of data gathering was on how anaesthetists coordinate their work and particularly how infusion devices were used by anaesthetists, and how their design influenced their use. Eleven observations were made: ten elective liver operations and one eye surgery, totalling 40 h. Sessions were conducted during the morning (the normal time for operations), and each took 3–5 h. In total, 11 different anaesthetists were interviewed and observed. All observations were conducted by EB, but meetings with DF and AB took place regularly.

Diary entries and the CI and DiCoT outcomes were analysed using Thematic Analysis (Braun and Clarke 2006) based around questions about the challenges of learning each framework, and what insights were achieved at different points in the study. The initial reflexive analysis was completed by EB, in discussion with DF; this was subsequently reviewed and extended by AB in discussion with both EB and DF.

3.1 Stages of analysis

The study took place in three parts: Part 1 applied CI, Part 2 applied DiCoT, and Part 3 was a reflexive analysis of the processes and outcomes of learning and applying CI and DiCoT to the work of anaesthetists. CI was learnt and applied first because DiCoT builds on CI’s techniques and models. The two phases of learning were separated so as to make it possible to identify insights that were derived directly from the initial modelling and those that were facilitated by the application of the DCog principles embodied in DiCoT. In the early phases of the study, we were regarding it as a comparison study, to identify what kinds of issues each framework helped to identify [in a similar style to the study of Sellberg and Lindblom (2014)]. During Part 3 (the reflexive analysis), we realised that this was an inappropriate way to think of the frameworks, since DiCoT makes use of some of the CI models, so our analysis shifted focus from how the frameworks compare with each other to better understand their complementarity and the overall process of learning CI and DiCoT as steps of conducting a DCog analysis.

The study working with each framework involved three sub-parts: learning the framework; data gathering and analysis using that framework (and maintaining a diary of learning); and reflection on what had been learned about the framework. In each case, EB’s learning was largely self-directed, based on reading existing resources presenting the framework and seeking clarification where needed from DF. Since data gathering and analysis were “owned” by EB, with mentoring guidance from DF and general from AB, we focus in the following sections on EB’s experience.

3.1.1 Part 1—5 weeks on CI

Contextual Inquiry was learnt over the first week, followed by 3 weeks of data gathering. Contrary to prior planning, the first week of data gathering was used for general context orientation, rather than the application of CI, because it proved to be too challenging to both make sense of the new setting and apply CI in that context.

until now I have mainly unstructured, chaotic data

Diary Entry, Wednesday, week 2 of part 1

I think because I was so overwhelmed by all the different information I sometimes took some information as granted, didn’t write it down. Maybe it is a good idea to create some kind of standardized observation sheet to steer the observation

Diary Entry, Friday, week 2 of part 1

The difficulties experienced were not attributed to CI itself, but to the complexity of the context of study. During the two subsequent weeks, CI was applied in six sessions. Extensive notes were taken during each session. Consolidation of the gathered data took place directly after the session. Two days each week were taken to create models, identify gaps in the data and prepare for subsequent observations. After data gathering was completed, results were consolidated and final adjustments to the models were made. The diary was evaluated in the last week of part 1 to reflect on what had been learned about CI.

3.1.2 Part 2—5 weeks on DiCoT

After the CI analysis was completed, attention switched to DiCoT. The learning took 2 weeks, a week longer than planned.

The learning of DiCoT is really cumbersome. So many different sources and the abstract principles make it really hard to learn.

Diary Entry, Wednesday, week 2 of part 2

In the following 2 weeks, five data gathering sessions were conducted. As before, consolidation of results took place immediately after the session. Two days a week were taken to create models, identify gaps in the data, and prepare for future sessions. Final consolidation of results, adjustments to the models and evaluation of the DiCoT diary, took place in the last week of part 2.

3.1.3 Part 3—3 weeks intensive, followed by further cycles of reflection and iteration

All data were reanalysed to review the process and outcomes of EB learning and applying CI and DiCoT, as phases of learning to do a DCog analysis.

4 Results

Here, we present EB’s experiences of learning and applying the frameworks sequentially. The first section reviews EB’s experiences of learning CI and DiCoT; in the following section, we reflect on his use of the frameworks and the insights they afforded.

4.1 Learning CI and DiCoT

When reviewing EB’s diary of learning CI, a theme that emerges repeatedly is the challenges that he faced in representing and reasoning about the complexity of the study setting, and in expressing that complexity in rather rigid CI models. When he moved on to learning DiCoT, his perception was that it was clearer how to use the models and principles to reason about the complexity of the setting. For DiCoT, the biggest challenge lay in the loose definition of the framework and the fact that the underpinning DCog theory was not covered in depth in the reference resources. Three main themes emerged in the analysis: how to represent and reason about complexity; the role of resources in supporting learning; and the process of learning to work with the models (and let them guide data gathering as well as analysis).

4.1.1 Representing and reasoning about complexity

Beyer and Holtzblatt (1998, p. 21) describe Contextual Design (of which CI is a key component) as being “optimized for large, complex projects” and yet, as noted also by Sellberg and Lindblom (2014), the text book did not offer strategies for managing the variety and complexity of activities observed in the anaesthetist’s work space.

When I firstly used CI I felt very stupid during the first observation days, because reading the book implied it to be a very easy straightforward task.

Diary entry, Wednesday, Week 1 of part 2.

Early on, the complexity of the work being studied felt overwhelming. Everything is interrelated, and EB wanted to capture that complexity:

It is just too much information for creating models. I need a focus. Everything is interrelated (if I only focus on drugs that are given for anaesthesia, I still have to focus on ventilation and monitoring).

Diary entry, Tuesday, Week 3 of part 1.

It was difficult to identify an appropriate level of abstraction to avoid being overwhelmed by details:

Might it maybe good idea to put some processing just in a black box, to be able to focus on the relevant things. However, […] tasks are so much interrelated it might steal a lot of the complexity of the actual task.

Diary entry, Wednesday, Week 3 of part 1.

The issue of level of abstraction and managing complexity emerged repeatedly. For example, when trying to create a first sequence model, EB resorted to creating an overview model showing all relevant tasks, which he then broke down into three sequence models. This was an improvised approach because the CI text does not provide any guidelines on the process of model creation. The assumption appears to be that complexity can be handled through decomposition, which was not readily achieved in the anaesthesia context.

Other questions emerged, for which no clear answer was found, such as whether there was an optimal order for developing models; how to model interruptions to work; where to make clear distinctions between different situations or merge models; and how to model contingent aspects of situations, given that there is no time component to some key models: they provide a static perspective on a continuously evolving context.

EB felt that DiCoT gave better guidance and resources for managing the complexity of the situation being studied:

[DiCoT papers] give better indications how to handle all the complexity one is facing at the beginning. Furthermore they […] propose a sequence of models […] What I also really appreciated was the mentioning of likely problems someone might face (being overwhelmed, don’t know where to start)

Diary entry, Wednesday, Week 1 of part 2.

One particular strategy for managing complexity that had been developed by Rajkomar and Blandford (2012) was the introduction of a system activities “meta-model”:

it isn’t really part of distributed cognition, but important to support the decisions one has made on the scope of one’s analysis. […] the ICU is an area with a lot of independent activities. You can never observe all of them, so you have to take decisions which activities to observe and which to let aside. Meta models help you to justify your decision. They show that there ARE other activities, but only describes them very broadly and their influence on the primary task.

Diary entry, Wednesday, Week 2 of part 2.

The greater support that the DiCoT resources provided for dealing with complexity did not, however, make the DiCoT phase of learning any more straightforward overall.

4.1.2 Working with inadequate information about DiCoT

As noted above, there is no single source of DiCoT guidance, so learning it involved reading several research papers. EB discerned inconsistencies between reports and yet felt that they reinforced each other.

The papers were mostly revision and did not add substantial information. I have the feeling that there is no consistent view on DiCoT yet, which makes it very flexible on the one hand, but also more difficult to learn and understand on the other.

Diary entry, Tuesday, Week 1 of part 2.

Reflecting the relative lack of maturity (and arguably greater ambition to model interactions) of DiCoT compared with CI, EB quickly became aware of some of the tensions and challenges of creating, in particular, the DiCoT artefact model, and the ways that it had been shaped by need and circumstance:

It is interesting to see how the artefact model evolved to a very complex model in DiCoT. In CI it was relatively constrained and easy to create. In DiCoT it gets a complete different meaning […]. Models aren’t as rigid as they are in CI and could be adjusted to one’s own requirements.

Diary entry, Thursday, Week 1 of part 2.

This view of CI as being rigid is probably a consequence of relying on one (well written, clear) reference source for the framework. One issue that is obvious with hindsight, but was not at the time, is that EB focused on reading DiCoT literature rather than the broader DCog literature. All the DiCoT models were consistent with the broader DCog literature, simply emphasising or de-emphasising different elements of DCog theory while also responding to the demands of the particular situation being represented. To the developers of different DiCoT models, the differences were not perceived as significant (because all were consistent with DCog), but to EB, the differences were perplexing and hard to deal with:

Today I tried to prepare for the first observation tomorrow and figured out, that it is much more difficult than I thought. The DiCoTs applied by [different authors] are quite different. They interpret the models in very different ways, which makes it hard to identify one single DiCoT framework.

Diary entry, Tuesday, Week 2 of part 2.

Distributed Cognition for Teamwork is based on the DCog literature and was developed as a guide to DCog analysis. At its heart are a set of abstract principles that are intended to guide the analyst: in the construction of DiCoT models; in drawing inferences from those models to inform future design; and in identifying the strengths and weaknesses of existing interactive (team based) systems (Furniss and Blandford 2006). However, the relationships between principles and models that were self-evident to DF and AB were not so to EB:

I still do not completely understand the abstract principles, and their links to the models.

Diary entry, Thursday, Week 2 of part 2.

Further, the DiCoT concepts, which are drawn from the DCog literature, were not adequately explained in the DiCoT literature.

it is really hard to apply the abstract concepts of the DiCoT to the concrete situation. What is an information transformation, who is a buffer?

Diary entry, Thursday, Week 3 of part 2.

Although not devised as a study of the existing DiCoT literature, the study did, in practice, serve as an enlightening formative evaluation of that literature, highlighting tacit assumptions, unclear explanations and, in particular, the extent to which an understanding of DiCoT relied on established understanding of DCog. This will inform future work on clarifying and testing DiCoT, but is not a focus for this paper.

4.1.3 Learning to work with models (any models)

An additional theme that emerged through the diary was the challenge of learning any model-based technique, and the way that it initially seems to present obstacles to overcome, but gradually becomes a powerful tool to aid thinking.

Any model-based technique focuses on some aspects while downplaying others. The novice analyst found this focus frustrating at times (even though, to take the example he uses here, one might argue that the analyst’s prior assumptions are not relevant to analysis):

I miss a possibility to fill in really broad observations, like “The work of an anaesthetist is less stressful than assumed”.

Diary entry, Tuesday, Week 3 of part 1.

In the early days, EB clearly had not internalised the semantics (purpose or meaning) of the models he was working with:

I wrote down all my observation again, categorized it (which still isn’t easy, because nearly every model fits every observation). I also find it very hard to be consistent. I think sometimes it happens that I mark something as “out of scope” whereas in other occasions it is still part of the “relevant” information.

Diary entry, Wednesday, Week 3 of part 1.

Initially, EB was trying to follow Beyer and Holtzblatt (1998) “to the letter”. When presenting a method, it is always difficult for authors to articulate how it can be legitimately adapted. Methods are, essentially, tools and are only useful insofar as they help with achieving the desired outcome. Expert analysts develop a sense of where methods and models can be adapted (Blandford et al. 1998), and where it is appropriate to take short cuts, but this was not apparent to a novice:

I read two papers on research that also applied the CI in their studies, in order to see how strictly they stuck to the “rules”. It became clear that they do not either. Both studies left out some of the five models and adjusted the approach in a manner fitting to their focus.

Diary entry, Thursday, Week 3 of part 1.

Reflecting back over the first 3 weeks of conducting observations and constructing models, EB could discern the influence of the modelling on his data gathering, showing that the modelling was shaping his later data gathering:

I could observe a pattern in my observations, because I have used colour coding for irrelevant observations. […] I sometimes focused on things that weren’t relevant after inspection afterwards. When I first started to create models, the information content of my observation increased significantly, probably because I now had a new focus and feeling what might be important.

Diary entry, Thursday, Week 5 of part 1.

Moving on to the DiCoT part of the study, as with the CI models, it gradually became evident that the requirements of constructing models were shaping the data gathering:

DiCoT highlights some aspects I haven’t looked at before—hierarchy, evolution, representation of resources, which I now try to implement in my models.

Diary entry, Monday, Week 4 of part 2.

By the latter stages of the study, EB was becoming masterful at using models to generate questions to focus further data gathering, so that data gathering was becoming increasingly shaped by the analysis activity:

Yesterday evening I reviewed my observations, the templates I created and the DCog principles and created questions for today. That works great. Every day I get a lot out of only a few hours.

Diary entry, Tuesday, Week 4 of part 2.

Models focus attention on particular aspects of the system. Their value gradually emerged over the time of the study, and EB gradually felt empowered by them. For example:

the social structure model made the organisation of work clearer to me.

Diary entry, Wednesday, Week 4 of part 2.

I have gradually built up a real understanding for the models. I often think ‘Oh that fits into the… model’. Furthermore I can ask much better questions, because I know what information I want to have. Last but not least, I come to a point where I can ask the same questions to different people and reveal contradictions.

Diary entry, Thursday, Week 4 of part 2.

It would be wrong to conclude that modelling became straightforward in the final few days of the project. Further difficulties were still encountered. Reference to worked examples helped; for example, it showed how flexibly models could be used. However, it did not resolve all issues:

I started with the artefact model of the [infusion device]. […] First I thought it was very clear, but after working longer on it I got unsurer and unsurer […] I read the paper of [DF], wherein he applied some kind of information flow model to the sequence of administration. We […] agreed that it would make more sense for me to put this part in the artefact model and then discuss the sequence in terms of resources.

Diary entry, Monday, Week 5 of part 2.

In particular, there were issues that emerged through observations that were not readily amenable to being described in the available models. As discussed below, model construction helped to expose many issues with the system, but it did not expose all the issues (such as the challenges of setting up one of the infusion devices) that emerged through observations and interviews:

I had some problems in using the existing models to show what I have observed. Most difficult was the modelling of the setup and adjustment process of the [named infusion device].

Diary entry, Sunday, Week 5 of part 2.

4.1.4 Summary

In summary, the approach of structuring learning into two stages helped to highlight contrasts between two elements of the DiCoT framework (modelling and principles) that are needed to completing a DCog analysis. It exposed many tacit assumptions in the extant DiCoT materials, e.g. that the link between DiCoT and DCog was self-evident; that the relationship between models and principles was self-evident; that the existing materials were clear enough to support a novice analyst; and that the variations in modelling approaches between different case studies were negligible. While we were aware that the DiCoT framework had been shaped by the case studies through which it had been developed, the fact that it had been applied in three complex work settings previously gave us reasonable confidence that it could be applied in anaesthesia—a confidence that was eventually borne out in practice, but only after overcoming significant conceptual hurdles.

4.2 The outcomes of applying CI and DiCoT

Complete sets of models for each framework were developed over the course of the study. In the early stages, the CI cultural model helped EB gain an overview of very broad influences on the unit of analysis. For example, it became apparent that there were issues with the purchase of equipment across the whole hospital, which resulted in there being different makes and models of devices, with inconsistent interfaces, being available in the department. Although this theme was not developed in the DCog analysis, early exposure to such broader contextual factors was helpful in better understanding the broad work system and outside influences that shape it.

The overall work of anaesthesia can be understood as involving three phases: inducing, maintaining, and ending anaesthesia. The focus for data gathering and analysis in this study was on maintaining anaesthesia. This, in turn, comprises three primary aspects: hypnosis (keeping the patient asleep), analgesia (keeping the patient free from pain), and keeping muscles relaxed for surgery. The principal technologies used by anaesthetists are infusion devices for administering drugs and monitors to maintain awareness of the state of the patient (these are shown in Fig. 3 below). EB’s analyses covered the flow of work and of information; the role of the physical layout in supporting or disrupting work; the social structures; and the detailed design and use of an epidural pump and a syringe driver (for pain management). The details of these analyses are beyond the scope of this paper.

For illustrative purposes, we focus on the flow models developed in the two stages of the study. We have chosen to focus on this model because DCog expands the information processing metaphor of the mind to broader sociotechnical systems (Hollan et al. 2000), which makes the information flow model central to DiCoT. Flow is also a key model in CI.

EB interpreted the use of the flow model subtly differently in the two frameworks. In CI, the flow model is concerned with the flow of work and the broad kinds of communication between agents, whereas DiCoT talks about the flow of information. Both are concerned with how interactions between actors and artefacts take place in the system, but they have differences in their level of abstraction and vocabulary. This had not been apparent in earlier work, where DCog had been learned before CI, and so the CI models had been implicitly interpreted in DCog terms.

The elements of workflow within CI can be people (ovals), groups (dotted rectangle), places, and artefacts (shaded ovals), as illustrated in Fig. 1. Connections between elements show how the elements interact to build up the workflow. For every person, their most important responsibilities are described. Small lightning bolts represent breakdowns within the flow.

Fig. 1
figure 1

Flow model (Contextual Inquiry), focusing on the anaesthetist team. Numbered issues are discussed below

Full size image

Based on CI, EB’s flow model description was as follows:

Two anaesthetists form the anaesthetic team. Both are responsible for maintaining the homeostasis of the patient and for supervision of the three subsequent steps of anaesthesia (inducement, maintenance and ending of anaesthesia). Both anaesthetists assist each other during their work and communicate frequently. Sometimes only one anaesthetist is present. Informal shift changes, in which one anaesthetist brings the other up to date about the current status of work, are therefore also part of their interaction. In rare cases, both anaesthetists may leave the room at the same time, which might lead to omitted information (Issue 1). In most cases one anaesthetist is a younger trainee, who is trained by a more experienced consultant. A further important part of the anaesthetic team is the array of devices, including the anaesthetic machine and infusion devices. The devices show important patient parameters as well as the current status of drug administration and assist the anaesthetist in delivering drugs to the patient. Because both anaesthetists have access to the devices, they act as an information vehicle, supporting the flow of information between anaesthetists and helping them develop a shared understanding of the current situation. The anaesthetic team monitors the patient, and delivers drugs as needed. The Operating Department Practitioner (ODP) facilitates the work of the anaesthetic team by preparing equipment and taking over other tasks. The surgical team is the second major group of actors within the anaesthetic workflow. It consists of at least two surgeons, who together operate on the patient and continuously communicate with the anaesthetists. Communication can have different content, ranging from patient-related topics (‘We have a severe blood loss’) to work-related (‘Could you please adjust the operating table’) and could flow in both directions. For example, one anaesthetist asked the surgeons to pause the operation because the patient status deteriorated severely. External people may enter the workflow by asking for information that is not related to the current operation. These might disrupt the anaesthetic workflow (Issue 2).

The flow model of CI helped EB to understand the workflow of the operating theatre, as it enabled him to structure his observations using the graphical language provided. For example, the distinction between two basic groups of actors (surgeons and anaesthetists) helped him to clearly align observations to one of the groups. Given the focus of the study on anaesthetists, this distinction helped in focusing subsequent observations to gather the most relevant information. The flow model shaped EB’s thinking; for example, it highlighted how artefacts can act as information vehicles between people. The advice from Beyer and Holtzblatt (1998) to take an abstract perspective to model the flow led him to concentrate on the most relevant flow elements, putting aside some other people in the operating theatre who had no direct influence on the anaesthetic workflow. Another insight was that according to definitions, the patient is regarded as an artefact during the operation and served as an information vehicle between surgeons and anaesthetists.

Distributed Cognition for Teamwork modelling built on the preceding CI modelling. The models developed included information flow models at different levels of abstraction. The overview information flow in anaesthesia is shown in Fig. 2. This is based on the CI flow model, but includes concepts that are key to the DCog literature, including information movement, transformation, hubs, and buffers (see “Appendix 1”).

Fig. 2
figure 2

Information flow model (DiCoT), based around the anaesthetist team

Full size image

Furniss and Blandford (2006) present principles to analyse the flow of information between important actors in a system, as listed in “Appendix 1”. The principles formed a basis for the model description, such as the following extract; in this, key principles and DiCoT concepts are underlined. It is likely that some issues (most notably issue 4) emerged because EB had spent more time working with the anaesthetists, rather than directly from the DiCoT analysis, although the focus on communication channels and information flow will have made these issues more salient:

The centre of analysis is the triad between patient, devices and the anaesthetists. Information enters this basic unit from different directions. The ODP’s communication with the anaesthetists flows over a channel with high bandwidth, as s/he stands right next to them, communicating face-to-face most of the time. When work is calm, anaesthetists and the ODP are talking informally about work-related and work-unrelated topics. Another important interaction can be observed between surgeons and anaesthetists. The information movement between these two groups is restricted as it takes place across a barrier that separates the anaesthetists’ area from the surgical space. Furthermore a paper mask covers the surgeon’s face, which decreases nonverbal communication through facial cues (Issue 1). As communication between the two groups is essential for the patient’s well-being it happens regularly nonetheless. However, situation awareness may be decreased, as surgeons and anaesthetists do not entirely know what the other team is facing (Issue 2). Besides the obligatory communication with ODP and surgeons, external persons might enter the theatre and interrupt the work (Issue 3). Most often the ODP or a nurse (not depicted in Fig. 2) reacts to these outsiders. As well as external people interrupting work, devices such as phones and laptops are sometimes used and may distract anaesthetists from the actual work (Issue 4). […] While in principle the patient’s face might give indications about the state of anaesthesia, this channel is blocked, because the face is routinely covered during the operation (Issue 5). […]

Figure 3 shows an illustrative example of the layout of the anaesthetists’ workspace, showing the patient (left) with face covered, infusion devices for delivering drugs, and monitors for tracking the state of the patient.

Fig. 3
figure 3

Anaesthetists’ workspace

Full size image

Distributed Cognition for Teamwork encourages analysts to look at the same situation at different levels of granularity, so a more fine-grained model was also developed. In Fig. 4, the interaction between anaesthetists and the most important devices is shown in more detail.

Fig. 4
figure 4

Information flow model (DiCoT)—focus on anaesthesia

Full size image

This more detailed analysis gave additional insights:

Anaesthetists derive information about the patient nearly exclusively from the patient status monitor. The monitor collects various parameters of the patient, thus acting as an information hub, and transforms them into a graphical representation on the screen. […] Communication with the monitors and infusion devices is not optimal, because the manual input is cumbersome and difficult, which reduces their value as mediating artefacts (Issue 6). The channel bandwidth between the infusion devices and the anaesthetists may be further impacted, because syringe drivers are all attached to one pump stand, which may lead to the higher pumps obscuring pumps lower down, affecting the user’s horizon of observation (Issue 7). Looking at the devices gives both anaesthetists a shared understanding of the situation and increases their situation awareness. For example, monitoring and infusion devices might act as behavioural triggers, informing the team about future actions. […] Anaesthetists act as information hubs within the system, integrating information from the surgeons, the monitors, the other anaesthetist and other sources to build one coherent picture of the patient state. […] Often, only one anaesthetist is present, making the use of a phone necessary; this reduces the bandwidth within the anaesthetic team significantly (Issue 8). This also takes away a hub and a buffer from the system, making it less stable (Issue 9). During informal shift changes, following a swap of anaesthetists, information might be omitted (Issue 10). Further, this prevents double checking, normally an additional control mechanism in critical settings whereby two people supervise important actions, jointly ensuring that errors are less likely (Issue 11).

These additional insights were derived from the DiCoT principles and concepts embodied therein (see “Appendix 1”). They illustrate the added value that EB found in applying the DiCoT concepts and principles to the observational data, moving him from a CI to a DCog analysis. This supports the assertion of Sellberg and Lindblom (2014) that methods based on DCog deliver more theoretical depth than CI.

4.2.1 Summary

We have presented an example of the findings from our DCog analysis, focusing on information flow, because information flow and transformation are at the heart of DCog. Other models focused on the design and use of particular artefacts within the work setting and on the physical layout of the space. Starting with CI supported the analyst in making sense of the situation and acquiring focus without being immediately required to think about the situation in terms of DCog principles. Adding in the DCog principles, and also analysing the situation at different levels of abstraction (as illustrated by the two levels of flow model presented in Figs. 2 and 4), enabled the analyst to identify issues that had not emerged through the CI analysis, which focuses on simpler concepts. DiCoT provoked the analyst to think beyond the direct observations in a way that CI did not.

5 Discussion

The goal of this paper has been to reflect on the process of learning and applying CI and DiCoT as methods for understanding complex work systems. The study design was adapted from that of John and Packer (1995). The key features were that the novice analyst maintained a reflective diary throughout the 10 weeks of learning CI and DiCoT, that learning was largely self-directed, based on available resources, and that he conducted data gathering and analysis in a complex healthcare setting.

Learning CI before DiCoT appeared, on the whole, to be beneficial, even though it became apparent that the models are less similar than we had previously believed (e.g. the CI flow model focuses on work rather than information). Learning CI first enabled EB to get to grips with constructing models and with the complex healthcare setting before learning and applying DCog principles. CI has a single authoritative source (Beyer and Holtzblatt 1998) with clear, simple examples for a novice to digest. However, the simplicity of the examples made it difficult initially to know how to construct models in a much more complex context. Moving on to learning DiCoT was more difficult than expected, mainly because information was distributed across several sources and was in places inconsistent. Also, DiCoT had been applied in complex settings; on the one hand, this meant that it was clearer how to apply it in a different complex setting; on the other hand, understanding the examples required understanding those complex settings. The DCog principles were found to be much harder to comprehend than the concepts (such as triggers and barriers) of CI. While a background in psychology helped with understanding DCog concepts, a structured introduction to the broader DCog literature before focusing on the (comparatively limited) DiCoT literature would almost certainly have helped too, particularly for understanding what was foundational to the method and what were incidental quirks introduced by different individuals as they developed and extended the framework.

We have summarised one example of applying CI and DiCoT, focusing on the flow models. CI proved very valuable in structuring data gathering and analysis, while the DCog concepts and principles enriched EB’s vocabulary and challenged him to see and analyse information processing properties of the system that were not readily apparent before. For example, consequences of being interrupted and of equipment being obscured behind other equipment became more apparent. Similar themes emerged through the other models that were developed.

One area in which CI extends the unit of analysis beyond that naturally covered by DCog is in the cultural factors that influence the work place. As noted above, the cultural model helped EB in getting an overview of very broad influences on the unit of analysis. While it would, in principle, be possible to develop information flow and social and evolutionary models at an organisational level, that is not a level at which DCog has been commonly applied, so such an analysis would be unusual. This highlights both a focus on finer-grained detail of interaction (at an individual and team level) and limitation in scope (in accounting for factors at an organisational level) of a DCog analysis. Different frameworks support different aspects of DCog analysis. For example, the focus of the Resources Model (Wright, Fields and Harrison 2000) is on the individual user–system interaction, while the focus for DIB (Galliers et al. 2007) is on communication within an organisation, particularly in the context of adverse events in healthcare. As illustrated above (Sect. 4), the focus of DiCoT is naturally on the small team level, where a team is typically co-located within a physical space and has a shared high-level objective. In recent work (post-dating the resources available to EB), Furniss et al. (submitted) have proposed an extension to DiCoT that situates the user–device interaction within a broader sociotechnical system context (e.g. user–device interaction; the immediate environment around that interaction involving the patient and other artefacts; the broader ward context; out to the hospital context, and potentially even wider); however, this development still puts user–technology interaction at the heart of the analysis, in contrast to CI, which puts work at the centre.

Distributed Cognition for Teamwork includes evolutionary and social structures models and associated DCog principles, which do not build on any CI models. For example, EB developed an evolutionary model that summarised the introduction, assimilation, and uses of different technologies in the system. This helped to highlight interdependencies between technologies. These included interdependencies around a syringe driver that had an advanced mode that was rarely used despite it having advantages for the patient and anaesthetist. Initially, we thought the documentation of this advanced mode needed to be improved to encourage anaesthetists to use it. However, constructing the model drew attention to the fact that this mode requires additional monitoring technology that was not implemented in the hospital.

Both CI and DiCoT had to be adapted to better analyse the context. Because this was an exploratory study, CI could not be applied as per the textbook: finding an appropriate focus was iterative. Also, data gathering and analysis had to accommodate the fragmented process that the context demanded: EB rarely received sustained attention from his interviewees, and it was often impossible to ask questions during work because that work is characterised by a shortage of time and many interruptions. DiCoT is less prescriptive in terms of the data gathering approach but suffered from lack of advice. For example, EB developed the notation in Figs. 2 and 4 by drawing inspiration from CI models (e.g. Fig. 1).

5.1 Limitations

Idiographic studies such as this are rife with traps of validity and generalisability. What if EB had had a different background or abilities? What if the three co-authors had developed a different working relationship (e.g. with more direct input into EB’s learning process)? What if the study had been conducted in a different hospital, or a different area of the hospital, or a different kind of complex working environment? What if we had adopted a different procedure (e.g. placing more emphasis on learning DCog from the broader DCog literature, or omitting the phase of learning CI) or used a different semi-structured framework (e.g. DIB instead of DiCoT)? We cannot be fully aware of the consequences of every contributing factor or incidental decision made, and the study is not reproducible. We have aimed, in our analysis and our reporting, to be open, honest, and reflexive. Our interest is in better understanding how a novice analyst may learn and apply DCog in practice and the role of frameworks in supporting analysis of complex work settings. We recognise that every individual is different, and likely to learn in a different way. Despite its limitations, the study has yielded valuable insights about the costs, benefits and scope of DiCoT and CI, and requirements for future refinements of DiCoT. Our findings on the costs, benefits, and challenges of CI and DCog-based modelling (using DiCoT) are consistent with, and complementary to, those of Sellberg and Lindblom (2014): whereas they focused on questions of descriptive power and application power, this study has focused on the challenges of learning, the role of frameworks in supporting the study of a very complex work environment, and descriptive power.

5.2 Conclusion

Distributed Cognition does not give a full picture of the work of anaesthesia: any model-based framework prioritises some features of the setting while downplaying others. Perhaps the most important difference between a model-based approach to data gathering and analysis and an unstructured approach is that it is clear which features are being particularly attended to and which are being downplayed, rather than that being an incidental consequence of the analyst’s prior experience or of the circumstances of data gathering.

The decision to start with a CI analysis before moving on to the DiCoT analysis made the role of each framework more apparent than it might otherwise have been. This was particularly clear in constructing the cultural model of CI and the evolutionary model of DiCoT, neither of which has an obvious counterpart in the other framework, but it was also apparent in the other models.

For a rapid analysis of a context, CI was found to be easy to learn, efficient, and delivered good results, although it was difficult to find a suitable level of abstraction for capturing the complexity of anaesthesia. The benefits clearly outweigh the relatively small costs in learning. However, after applying CI, we assumed that workflow and device interaction in the operating theatre was highly optimised; it was the DiCoT analysis that revealed areas for improvement. DiCoT was found to be harder to learn, partly because there is no comprehensive, consistent source of information on it, and also because the principles demand mature understanding of DCog. Taking the two-stage process of learning CI first and then building on that with the DiCoT models and DCog principles proved to be an effective way to learn, both about DCog and about the work of anaesthesia. This was because CI was much simpler and better described, making it better suited for building initial understanding of the complex domain of anaesthesia; then, the DCog principles embodied in DiCoT enabled the analyst to see beyond the surface of the work environment, particularly in terms of resources for communication.