Keywords

1 Introduction

1.1 Background

Attempts to augment human abilities can be traced back through much of human history, when they included functional extensions of the human body through a physical medium [1]. Contemporary technological innovations allow more forms of Human Augmentation (HA), such as the extension of our senses via sensory technologies (e.g., night vision goggles), the improvement of physical abilities by hardware means (e.g., exoskeletons) or the enhancement of cognitive capabilities through a human-computer ‘closed-loop’, which characterize the field of Augmented Cognition (AC) [2].

Despite long-standing practices in HA, the field still lacks a generalized ‘theory of augmentation’ which guides the selection of such augmentations with respect to the types of tasks humans need to perform. In other words, we do not yet have a taxonomy which could help us understand which augmentation to use to resolve which type of cognitive problem. This lack of structure affects the access to existing knowledge and the integration of new contributions. The gap is more critical now that digital tools are becoming prevalent means of delivering augmentations [3], multiplying the augmentation possibilities. This paper focuses on the field of AC. It aims to provide a solid framework that helps to navigate the growing knowledge and guides the selection of cognitive-enhancing augmentations to address cognitive problems.

1.2 Previous Works

The recent classification of De Boeck, et al. [4] (see Table 1) summarizes several recent contributions that have attempted a categorization for the broad field of HA [5,6,7,8,9]. Their work identifies (a) four categories of augmentations (the type of aid: sensory, physical, cognitive, and social) and (b) three dimensions of augmentation (the ‘amount’ of aid relative to the human innate capabilities: replicating or replacing, supplementing, and exceeding).

The taxonomy introduced by de Boeck et al. has been useful for the development of the field, the four categories proposed are broad enough to cover previous HA applications. However, they are weak in describing the specificities of each case. The diagram helps to position a single HA within the dimensions, but it does not explain how an HA application could be generalized and linked to other cases. The absence of any correlation between augmentations restricts its potential to generate insights and to guide designers in their choices.

Table 1. HA domain adapted from De Boeck, et al. [4] with examples of HA applications. Highlighted is the domain of the taxonomy.
Full size table

1.3 Objective

Given the mentioned gaps, this work aims to answer the following questions:

  • For each type of cognitive problem, what type of AC has been tested?

  • How was each cognitive augmentation applied?

  • What other forms of AC have been tested for that problem?

This paper answers the questions by proposing a taxonomy for AC which has four dimensions:

  1. i.

    Field of Application (e.g., medical, military, education)

  2. ii.

    Limitation (the human condition which justifies an augmentation of capabilities, e.g., incorrect focus, memory fault)

  3. iii.

    Augmentation (the aid provided to the user, e.g., knowledge provision, task load reduction)

  4. iv.

    Implementation (the form through which the augmentation is delivered e.g., instructions, visual cues, alerts).

These dimensions are combined in linking grids and can be used to map previous steps in the field, to navigate the current state of the art, and to orient future research directions, like a compass. The user of the compass can start from any of the four dimensions and explore the others following the prompts shown in the compass dial (Fig. 1).

Fig. 1.
figure 1

The AC compass: the cardinal directions correspond to the four dimensions of the taxonomy. Each quadrant of the compass (FL, AL, AI, FI) is a linking grid that intersects two dimensions and provides insights about the AC field (see Tables 4, 5, 6 and 7 in the appendix).

Full size image

As in Fuchs et al. [10], the taxonomy separates the cognitive augmentations from the implementation strategies. In fact, the same augmentation can be implemented in multiple ways (e.g., knowledge provision via instructions or analytics). Likewise, the same implementation method can be used to provide different augmentations (e.g., visual cues for action correction or attentional deployment).

In this paper the compass is applied in the field of AC, however its dimensions are applicable to any type of augmentation, making it a robust tool to classify the whole HA field.

1.4 Definition of Augmentation

Several definitions of augmentation have been proposed in recent years [5, 6, 9, 11,12,13,14,15,16]. This paper adopts the robust definition by Moore [14] who defines human augmentation as:

“[…] any attempt to temporarily or permanently overcome the current limitations of the human body through natural or artificial means. It is the use of technological means to select or alter human characteristics and capacities, whether or not the alteration results in characteristics and capacities that lie beyond the existing human range.”

In this work, ‘technological means’ doesn’t necessarily indicate digital equipment, but any “artifact […] to extend human capabilities” [17]. That is to say: augmentation as an extension of our faculties and capabilities, regardless of the tool.

This is rather important given that the same technology used in different ways can provide different augmentations (e.g., haptic technology can be used for controllers’ feedback or as vibration alert for smartphones). Similarly, the same augmentation can be provided using different tools (e.g., wayfinding through signposting or by using GPS navigation instructions). Moreover, a taxonomy where the augmentations are independent from the tools will be more robust and resilient, especially in a fast-paced context where new technologies rapidly replace obsolete ones. Consider for instance sundials, mechanical watches, digital watches and now smartwatches. They are all tools made from different technologies. Over time, they replaced the functions of the previous one, but they all offer the same augmentation: providing the user with information.

In light of these considerations, technologies are not used as criteria in the definition of the taxonomy.

2 Method

2.1 Domain of the Taxonomy

The taxonomy is obtained from a review of articles in the area of AC, which is a sub-field of HA that seeks to extend cognitive abilities by addressing the humans’ intrinsic limitations in attention, memory, learning, comprehension, visualization abilities, and decision-making [18]. As per de Boeck and Vaes’ classification, case studies of cognitive augmentation that supplement or exceed human abilities have been categorized. The domain of the taxonomy is highlighted in Table 1.

2.2 Search Design

The selection of eligible articles for the taxonomy was based on the following criteria:

  1. 1.

    Search for published journal articles, conference papers and reviews only, written in English language, in Scopus (Elsevier) and Web of Science (Clarivate) databases. No timespan was considered, and the latest articles analyzed were published by December 2022.

  2. 2.

    Identify relevant articles by looking for one of the following terms in title, abstract or paper keywords: "augmented cognition", "augcog”, "human augment*", "human enanc*", "cognitive augment*".

  3. 3.

    Filter by subject areas and paper keywords related to AC.

  4. 4.

    Ensure relevance of the articles by reading all titles and abstracts, excluding duplicates, and checking that they fall within the domain of the taxonomy

  5. 5.

    The remaining articles have to be read completely to make sure that the discussion is related to AC and that all the four types of attributes of the compass (field of application, limitation, augmentation, implementation) are explicitly stated.

Description of tools, lists of hypothetical applications, references to other papers proposed as generic augmentations and papers without the explicit four attributes have not been considered. The final sample consisted of 77 articles. Table 2 gives an overview of the search process.

Table 2. Database search results.
Full size table

2.3 Categorization of the Dimensions

From each of the shortlisted articles, four attributes corresponding to each of the taxonomy’s dimensions (field of application, limitation, augmentation, implementation) were identified and listed as in the original text. If a case study presented more than an augmentation for the same situation or context, they were listed as separate entries (e.g., Dorneich, et al. [19] in Table 3). A total of 137 quartets of attributes were extracted from the 77 articles.

Each attribute was categorized through an inductive process [20], where categories are tentatively assigned. While progressing with the categorization, those categories are revised, eventually reduced to main categories and checked in respect to their reliability. The categories were finally aggregated in super-categories to build a hierarchical structure of the taxonomy. Three examples of categorization of augmentation attributes in quartets are shown in Table 3.

Table 3. Examples of attributes categorization.
Full size table

2.4 Validation of the Taxonomy

The taxonomy’s adequacy was evaluated through its content validity [22,23,24]. Two independent judges re-coded ‘limitation’, ‘augmentation’, and ‘implementation’ attributes from a random sample of 50 quartets (out of 137). The ‘field’ attributes were omitted as the less equivocal of the attribute types.

The content validity of the taxonomy was inferred by the level of agreement between coders (calculated using a coefficient kappa method [25] as suggested by Boateng, et al. [26]) and by the number of new categories generated. The higher the agreement, the more the categories represent the attributes. Conversely, the fewer new categories that were generated, the more the taxonomy reflects the domain of AC.

The validation process showed an agreement level which is deemed substantial given the obtained kappa values: 0.572 for limitations attributes, 0.650 for augmentations attributes and 0.696 for implementations ones. Finally, only in three cases new categories were proposed by the judges, which suggests a good coverage of the AC domain given the 40 initial categories.

3 Results and Discussion

The aim of this study was to provide a taxonomy of cognitive augmentations useful to navigate the growing field of AC and to guide the choice of augmentations.

The taxonomy’s framework is made of four dimensions intersected in four linking grids which constitutes the quadrants of the compass (Tables 4, 5, 6 and 7 in the appendix). The grids offer a quick overview of the AC field, while the categories and their mutual relationships give insights in specific areas. The individuated categories are described in Table 8.

Several fields of application were found in the AC literature (Table 4). Military, medical, educational, and driving fields presented the largest variety of activities and addressed limitations. The military sector experimented with the most implementations (Table 7). The vast majority of the encountered tasks are operational tasks.

In terms of limitations, not all those faced in the studies are related to cognitive bottlenecks, such as problems of information processing and storage, or an incorrect mental state of the operator. They are also due to physical limitations, like the hypothetical cost/risk of a situation and the unpredictability of human error (variable performance), or by the lack of some sort of knowledge (Table 4).

The augmentations proposed by the AC field to face those problems can be grouped in few categories (Table 5). Aid consisted in managing the task load or by giving assistance during the task. But also through modifying the flow of information during the task, the mental state of the user, or by giving the possibility to simulate scenarios (simulativity).

Finally, augmentations have been implemented using three main strategies of addition, subtraction, or modification (Table 6). Addition of prompts, analysis, cues or experiences during the task, subtraction by reduction or delegation, or modification of the information flow, of some elements of the task or of the task itself.

3.1 Limitations

There are limitations in the construction of the taxonomy. First, the article sampling is far from perfect. Despite the high number of searched papers, the sample was obtained from few keywords and some relevant studies could have been missed. The fact that no articles were dated before 2003, suggests that similar studies could have used different terms before that date (e.g., intelligence amplification).

Second, to maximize objectivity in the coding, only papers which clearly indicated the four types of attributes were categorized. Again, possible relevant studies could have been excluded because not explicit enough to comply with the protocol.

Similarly, the rejection of other types of augmentations apart from the cognitive ones could have excluded some hybrid cases (e.g., a cognitive augmentation for a physical limitation, like in Futami, et al. [27]).

Another limitation comes from the abstraction of the categories which is an inevitably subjective process. This work followed a thorough validation process, with good agreement outcomes, that however involved a small number of independent judges.

Finally, the taxonomy has breadth to cover the whole field but shallow depth of analysis. The intersection of two categories in a linking grid indicates that they have been combined in at least one of the examined papers. However, it doesn’t indicate any evaluation, frequency, or recommendation of that combination.

3.2 Future Directions

The taxonomy gives an overview of what has been done so far in the field of AC. Evident future directions are individuated by the white spaces in the quadrants of the compass. Those graphical gaps indicate unexplored possible applications of augmentations for problems that have been addressed in other methods, perhaps in improvable ways.

Another clear future development is the extension of the taxonomy to the whole field of HA, including social, physical, and sensory augmentations. In this paper the compass is applied in the field of AC, however its dimensions are stable enough to be applicable to any type of augmentation.

To address the limited depth of analysis, in a more extended publication the taxonomy could keep track of the evaluated cases and provide more information to the user. For instance, an example application for each combination in the linking grids and the number of encountered application which fall in that combination.

Looking at the whole field of AC, almost all the analyzed studies involved operational tasks. AC is practically unexplored for applications in tactical and strategic tasks. Fields like management and strategic cognition would benefit from tools that augment cognitive capabilities of decision-making.

AC has already been described as a young research field with no commonly agreed-upon definitions on what it includes or what constitutes an augmentation [4, 6]. Unsurprisingly, form the analysis of the literature emerged a significant heterogeneity in the type of studies, methodologies, language, definitions. A joint effort from scholars in AC for the definition of solid and recognized foundations in the field is deemed necessary. The taxonomy introduced in this paper, like the one from De Boeck, et al. [4], is an attempt in that direction.

Another relevant gap is the absence of an evaluation framework to assess the effectiveness of an augmentation. In fact, only few of the analyzed articles presented an evaluation of the proposed augmentation, some of which proved to be counterproductive [28, 29]. Objective and recognized metrics of cognitive augmentations, similar to the concept proposed by Fulbright [30, 31], would allow a comparison between cases, steering the field of AC towards the most promising applications.