Cooperative Hub for Cooperative Research on Cooperatively Interacting Vehicles: Use Cases, Design and Interaction Patterns

Abstract

This chapter describes central cooperative activities in the research priority program Cooperatively Interacting Vehicles (CoInCar). If the whole research program CoInCar can be seen as a wheel, which is turning research questions into answers, knowledge and hopefully progress for society, the individual research projects described in the other chapters could be seen as spokes of the wheel, and the aspects described in this chapter as an informal cooperative hub of the wheel. Starting with common essential definitions, a use case catalogue was derived and documented. Based on that, cooperation and interaction pattern were sketched and documented into a pattern database. While the details of the research hub described here are specific for this DFG priority program, the general principles of a research hub could be transferred to any other research and development activity.

1 Introduction: The Big Picture—From Cooperative Homo Heidelbergensis to Cooperative Human Machine Systems

In general, movement through space and time is a vital feature of life. Movement in form of mobility, by foot, bicycle, car, train or airplane is an important part of our life as individuals, organizations and societies. Cooperation in contrast to competition is a central aspect of mobility already for a long time, and was already quite important for our development as homo sapiens, as the following example shows.

Fig. 1

One of the spears of Schoeningen, used for cooperative hunting as an example for cooperative movement, cooperative reasoning and acting

Figure 1 shows one of ten wooden spears which were excavated in 1994ff at Schoeningen near Braunschweig in the north of Germany. These throwing spears, dated between 380,000 and 400,000 years old, represent the oldest preserved hunting weapons of prehistoric Europe yet discovered [34]. These spears are not only an early example of weapon technology, but also for an art which is much later called Human Factors (Engineering), for which Homo Heidelbergensis, a prerunner of Homo Sapiens, was already able to combine different techniques like cutting to carve, and fire to harden an effective tool and adapt it to the individual bearer. These spears are also an early example of Human Systems Integration, which in the 21st millennium is understood as integration of humans, technology, organization, and environment, and which was already an important factor for Homo Heidelbergensis: Close to the location of the spears, many horse bones were found. Anthropologists reconstructed that a tribe of Homo Heidelbergensis hunted, rounded up, speared, and ate these horses. Especially the production of the spears, which can be seen as a clever use of or integration with the environment, and the cooperative hunting took a degree of organization, which was not available to other rival species.

It is obvious that movement and mobility in combination with these tools was one of the key factors for success of these homo tribes. But how could these relatively slow species round up and eat other species which were physically much faster and stronger? The key issue can be found not so much on the physical layer, but on the cognitive layer of evolution: Tomasello [35] describes how human cognition evolved together with the ability to create and handle such tools, and especially how the cooperation and shared intentionality e.g. of hunting AND tool manufacturing fostered the evolution of homo towards homo sapiens as one of the most dominant species on this planet. Cooperation and teaming were obviously essential for early hunting. Cooperation and teaming might also be essential for cooperatively interacting vehicles, and for researchers and developers addressing these complex cooperative systems. It is not by accident that communities of research institutions applying for a research grant nickname themselves “hunting communities” or “hunting tribes”, but by insight that the ability to cooperate might be similarly important with hunting and with research.

What Tomasello [35] describes as shared intentionality, other researchers like Norman [28] or Gentner [16] describe as shared mental models (c.f. Fig. 2), where mismatches between the mental model of system designer and system users might lead to dangerous errors in design or use of sociotechnical systems. Mental models are also at the very core of cooperatively interacting vehicles, and of the cooperative research on cooperative interacting vehicles.

Fig. 2

Shared mental models to create shared intentionality, example Homo Heidelbergiensis hunting

Applied to cooperatively interacting vehicles, the setup of cooperatively interacting vehicles might use similar cognitive capacities which already helped homo heidelbergensis to move cooperatively, but might include a new complexity: Here, not only individuals and groups of homo sapiens are involved, but also a new player on the cognitive evolution: The computer. In less than a century from its first invention by Konrad Zuse in 1941, the computer and later Artificial Intelligence (AI) has become a central player in sociotechnical systems. The teaming of computer and humans is already hinted in Wieners famous book about Cybernetics in 1950, where he describes feedback loops as the central mechanism of intelligence both in the animal and the machine. Later Licklider [24] describes symbiotic human—computer systems. Rasmussen [31] proposed the term cooperation, Hollnagel and Woods [22] and Sheridan [33] defined initial principles, Hoc and Lemoine [20] and Hoc [19] described the common ground and know-how-to-cooperate as important parts of developing human computer cooperation.

A major breakthrough was to think of cognition not only as something separated/assigned to individual agents, but also as something which is combined or joined between the different players, i.e. Joint Cognition or Joint Cognitive Systems. Hollnagel sketches how these Joint Cognitive Systems can be nested, from the small to the big, and already prepared the ground for a system of systems approach. System of systems can be understood as the joining of individual systems which “deliver important emergent properties, which have an evolving nature that stakeholders must recognize, analyze and understand” (e.g. Maier [27]). Flemisch et al. [9] described how humans and machine cognition in system of systems can cooperate on levels with different time frequencies, yet still work together like the blunt end and the sharp end of a spear. Flemisch et al. [13] extended this view also to conflicts that can happen between agents in cognitive systems, and how these can be mitigated. Examples for conflicts are different intentions of humans and machines, e.g. vehicle automation, of where to go and how fast. Flemisch et al. [11] describe a holistic bowtie model of meaningful and effective control, which brings together the individual human-machine system with a system-of-systems, organizational, societal and environmental perspective. Cooperation between these layers are—once again—a central key for failure or success of these systems.

As already hinted by Hoc [19], the key of any cooperative activity is to have sufficient common ground and common work space between humans and computers in the form of shared mental models. This proved to apply even more so for cooperatively interacting vehicles. Related to common ground is the concept of inner and outer compatibility (e.g. Flemisch et al. [8]), which describes the ability of interfaces on the outside system border between humans and machines to play together, and the ability of inner mental models to interact in a cooperative way.

Fig. 3

Shared mental models between humans and computers on cooperatively interacting vehicles (Flemisch et al. [12] based on Flemisch et al. [9]

In general, the development of shared mental models does not start from scratch but is always a development and migration. Starting point could be basic image schemes which we inherited from our ancestors (e.g. Lakoff [23], Baltzer [3]), patterns we learned during our life to deliberate discussions in our research and development community on how sociotechnical systems, here cooperatively interacting vehicles, should work together amongst themselves and with the humans involved. In this ongoing effort to shape the mental models, it is important that mental models evolve cooperatively. They are never all up to date at the same time, as Fig. 3 shows.

Applied to cooperatively interaction vehicles, Fig. 3 shows an example of inconsistent mental models, where the human on board of an automated vehicle assumes that the vehicle automation has the control, while the other user assumed that the human was in control. Such a misunderstanding already led to a deadly accident in 2018 with a highly automated vehicle operated by Uber [29].

Figure 4 depicts a very simplified model of how shared mental models in the research community of automated and interconnected driving might have evolved, starting with a simple “black and white” model of manual or fully automated driving, then the intense discussions on different levels of automation, sparked by the theoretic work of Parasuraman et al. [30], boosted with insights like the H(orse)-Metaphor [7], the first formulation of highly automated driving, and its practical solutions like H-Mode [2] or Conduct by Wire [37], leading to the BASt and SAE levels of driving automation [15, 32]. With CoInCar, we entered a new stage of automation, which still uses levels of driving automation, but connects the differently automated users and automations with cooperative driving patterns.

Fig. 4

Vehicle and traffic automation from “on/off-automation” of the 1990ties via “Levels of Automation” of 2000ff to Cooperative Automation 2010ff, as a cooperative development of shared mental models in the research and development community [12]

2 Bringing Researchers Together: Concepts and Definitions Wiki, Ph.D. Workshops

In general, common ground and common mental models for researchers usually do not start with definitions, but with common inspiration, ideas and visions, as vague or fuzzy as they might initially be. Only if this inspirational common ground is assured first, the more tedious work on common concepts and definitions has a chance to succeed. Even then, with complex systems and interdisciplinary teams, it is often impossible to achieve a similar crispness of definitions, as scientist were able to achieve in physical sciences. Especially in the integration of humans, technology, organization and the environment, so many disciplines are involved, that crisp definitions like in physics are highly unpractical, but a higher plasticity of concepts and definitions has to be tolerated from the very beginning, if the definitions should really open the chance to converge between disciplines.

Applied to CoInCar, in a series of workshops in mixed subgroups, concepts and definitions were worked out, documented in a Wiki and deconflicted over the duration of the project (see Fig. 5). It is important to note that the approach was not to deconflict all differences in the definitions—this alone would have consumed most of the research budgets—but to find a common ground just big enough to start cooperations, and to further consolidate the Wiki “on the job”.

Fig. 5

Concept and definition Wiki of the CoInCar project. (Weßel and Herzberger [36], exemplary screenshots taken form the CoInCar intranet)

For further networking within the priority program, a series of structured activities took place for the Ph.D. researchers. For example, every two years there was a cross-project Ph.D. workshop in which the researchers presented their research topics, discussed, and identified links between the subprojects. Furthermore, individual disciplines had regular Ph.D. workshops. One example is the regular meeting for human factors researchers, which met once a month for one hour in a hybrid format. Here, short presentations were given in a rotating process and the researchers own progresses and difficulties were exchanged and discussed within the group in order to benefit from the experiences of the other research groups.

3 Bringing Researchers and Developers Together: Use Case Catalogues

In general, to really understand and master complex sociotechnical systems with all their possible combinations, a system analysis can lay the structure for interdisciplinary teams. Over the years of Systems Engineering and Human Systems Integration, a structuring of systems in problem and solution space (e.g. Haberfellner et al. [18]), and in design space, use space and value space has shown good results of mastering the complexity (for an overview and history of these concepts and their application to the exploration of human-machine systems, see e.g. Flemisch et al. 2022a).

Applied to CoInCar, the alignment of mental models of researchers started with the use space, i.e. the dimensions of use and their combination into use cases and use situations. Based on the positive experience in EU-projects on highly automated driving of working with use cases (e.g. Hoeger et al. [21]), and research efforts to find define a unified ontology for test and use case catalogues in DFG-projects before CoInCar (e.g. Geyer et al. [17]), CoInCar started with the discussion, selection and definition of initial use cases of cooperatively interacting vehicles.

Figure 6 shows the use case tree of CoInCar as an overview of use cases addressed in the consortium. Starting with the use case family of obstacle avoidance, the use case families of lane change, intersection, parking and roundabout are identified, and individual use cases documented. In deconflicting sessions the use cases were discussed and if possible aligned. The use case catalogue also served as a map to explain the priority program, and to onboard new researchers.

Fig. 6

Use case catalogue representing the research topics of the subprojects within the CoInCar project (Canpolat et al. [5], published in the CoInCar intranet SSELab)

4 Bringing Researchers, Developers and Users Together: Pattern Catalogue of Cooperatively Interacting Vehicles

In general, complex systems can be decomposed into system models, use space, design space and value space. This helps with the understanding of the individual components of the system, but not yet with the understanding of the relations and only partially with the recomposing and designing of system variants. Seeing design, use and value space as systems themselves, and taking Luhmann’s argument “Contact happens at the border” [26] of these systems seriously, it is crucial to find a way to describe the interplay of these dimensions as a combination which has a combined effect: How a certain system design is used by the user, and what effects this has on users and the surrounding system. The challenge for this to find a representation that really grasps the essence of a specific design, use and value in a way that it is general enough to be reused, and understandable enough that it can connect researchers, developers and users.

An essential concept to achieve this are patterns, here as design and interaction patterns. Patterns can be traced back to the philosophical theory of Forms (e.g. Plato 427 B.C.). Just neglecting the long philosophical dispute whether forms are something outside of the physical world or just mental models in the brain of the analysts, Christopher Alexander described architecture as a language of design patterns [1]. This concept was transferred to software design patterns by Gamma et al. [14], to human computer interaction by Borchers [4] and to human-machine systems e.g. by Flemisch [6], Baltzer [3]. Based on Alexander’s initial definition, Flemisch et al. [10] understand patters as follows:

A pattern describes something that occurs over and over again. An example for this is a problem and/or its solutions. If this can be observed, and its core can be mapped and modelled, you can either observe and match the pattern over and over again, without ever making the identical observation twice. And/or you can instantiate and design with this pattern over and over again, not necessarily doing it the same way twice. Examples for this are designing, engineering and using of artefacts like human-machine systems. (Flemisch et al. [10] based on Alexander et al. [1])

Patterns bridge the more concrete world of applications with the more abstract world of concepts, and can provide a common mental model of the sociotechnical system and its principal understanding, design and usage. With that, patterns can be a crucial technique to bring designers, engineers, users and other stakeholder together (see Fig. 7).

Patterns can be based on use cases or use situations, and then describe how the use is usually happening with which results. This can be described on different levels of detail, e.g. very general usage (e.g. Baltzer [3]) up to a more detailed description of the interaction happening in a certain use situation (e.g. Flemisch [6], López Hernández et al. [25]).

Fig. 7

(adapted from Flemisch et al. [10])

Human-machine pattern as an interplay of researchers, designers, engineers and users in understanding, designing, engineering and using human-machine systems

Patterns can be freely formed, or transferred between domains, e.g. from the biosphere to the technosphere. A striking example, shown in Fig. 8, for the potential of transferring design and interaction patters is flying, where Otto Lilienthal systematically evaluated the flight of birds, and transferred the most important principles to design patterns e.g. of wings, foils etc., which still form the basis of flying today.

Fig. 8

Design and interaction patters as a connection of decomposing/understanding, recomposing/engineering and using, example flying (Otto Lilienthal, 1874ff)

Applied to CoInCar, patterns influenced the scientific undertaking from the very beginning, e.g. in form of the H-Metaphor as an inspiration for the cooperation between automation and the driver, but also the cooperation between vehicles as a herd or flock. More inspiration came from other domains like dancing, where a common understanding of figures and movements, i.e. patterns, allow dancers to move together and enjoy it (see Fig. 9).

More concretely, the pattern concept was introduced in the second half of the CoInCar focus program, discussed and refined in a series of workshops.

Fig. 9

Example for cooperation patterns and image schemes, derived in other domains and translated to cooperatively interacting vehicles [10]

Fig. 10

Example for cooperation pattern “inform, warn, intervene”, which can be applied between humans, humans and machines, or between machines [3]

Based on this conceptual work between the individual research groups, a first database was built up and filled for a first test. Figure 10 shows a fundamental pattern “inform, warn, intervene” as an example for a cooperation pattern, which was used in a couple of use cases of CoInCar. Figure 11 shows another fundamental family of patterns “Transition of control”, which were investigated in a couple of explorations and experiments in CoInCar.

Fig. 11

Cooperation pattern “Transition of control” as an example of pattern in the online pattern database

5 Conclusion and Outlook

It is important to note that focus programs are usually not as rigidly organized as e.g. excellence clusters or even industrial research and development projects. Organizing a research hub like in CoInCar, based on a concept Wiki, a use case catalogue and a first pattern database was an exploration of ideas, with promising first results, but far from providing complete catalogues or databases which are now ready to use in industry. Nevertheless, these results can provide an inspiration, or a concrete first core for more rigid research and development projects in the realm of cooperatively interacting vehicles, or beyond in the realm of cooperating human-machine systems including human—AI systems.

We see a huge potential in the combination of use cases and design/interaction patterns, which can clearly help to manage the complexity of future cooperative systems. Our vision is that the know-how about human machine patterns and their usage is increasingly collected in easy to access and easy to use data bases, ideally globally (c.f. Fig. 12). The key will be to provide the human machine patterns in a way that it can be easily used in design, engineering, and research activities, so that know-how can flow freely back and forth between researchers, designers, engineers, users and policy makers. This could also be the core for incident and accident databases that, along with cooperative research and development, could make our world safer, more sustainable, and more fun and joy to live in.

Fig. 12

Our vision are Global Databases for Design and Interaction patterns to collect knowledge and know-how about the design, engineering and usage of sociotechnical systems [10]