Analysis of Coordination Mechanisms during Collaborative Problem-Solving on an Interactive Tabletop Display

Abstract

Coordination is an important aspect of group work. Previous studies have shown how collocated multi-user interfaces, such as interactive tabletops, support coordination by providing a shared space that enhances workspace awareness. However, only little is known about the coordination processes that occur during problem-solving on such shared devices and how the design of features and interaction techniques can impact coordination behaviour. In this paper we analyse users’ coordination mechanisms during joint manipulation tasks in Orbitia, an interactive tabletop-based problem-solving activity. The proposed design integrates a series of “breaches” seeking to challenge participants’ collaboration by confronting them to different limitations. We report on a case study with five groups of three users (N = 15), jointly solving tasks while facing different challenges related to the activity interface and the environment. By analysing 135 min of video material along with their transcripts, we identified nine different coordination mechanisms that relate to either coordination of information or coordination of actions. By exploring the occurrences of these mechanisms, we found that sharing unsolicited task-relevant information is a common coordination behaviour that can be observed at interactive tabletops, and that with breaches, more explicit coordination in the form of direct requests, orders, or shadowing is used.

1 Introduction

Coordination within groups can be considered a fundamental and complex phenomenon, required for many everyday tasks and activities. Through coordination, groups concatenate individual actions and make sure that these contribute to a common effort. In case of poor coordination, process losses may occur which leads to reduced productivity and performance (Wittenbaum et al. 2005).

Collocated interactive tabletop displays inherently support coordination by providing a common, shared space, where users can see and work together. They allow them to maintain awareness of each other’s actions, bodily movements and gazes (Fernaeus and Tholander 2006; Antle and Wise 2013), and therefore to coordinate actions with little or no verbal communication (Hornecker et al. 2008). Many situations of collaborative work, however, do not solely involve shared activity, but also require that users can perform individual tasks, with group members moving back and forth between the different types of work. The work of individuals can best be supported by providing multiple entry points with a high level of control, which, in turn, can lead to problems in coordination (Hornecker et al. 2007; Yuill and Rogers 2012). As expound by (Gutwin and Greenberg 1998) the needs of individuals, requiring efficient and powerful control over the application, creates tensions with the needs of groups, requiring workspace awareness to support fluid coordination of actions. Considering these tensions, it is a complex matter to identify the best solutions for supporting coordination on interactive tabletop surfaces.

Existing systems try to avoid coordination problems on interactive tabletops by using design features that enhance the shareability of the system, potentially combined with mechanisms that prevent independent work (e.g., (Marshall et al. 2008; Stanton and Neale 2003; Pontual Falcão and Price 2011)) or encourage reconnection after parallel work (e.g., (Isenberg et al. 2012)). However, to date, only little is known about how coordination processes occur during problem-solving on interactive tabletops, more specifically in situations when shareability or joint awareness is affected. To our knowledge, there is no prior study focusing on users’ coordination during joint problem-solving on interactive tabletops.

Orbitia is a collaborative problem-solving activity on an interactive tabletop, that has been implemented as part of the ORBIT project (Sunnen et al. 2018). The idea of the activity is to provide participants with an experience of successful collaboration, and to help them be aware of their collaboration strategies. The activity therefore can be considered as not only enabling collaboration, but inducing it, i.e., is part of the research topic of technological enhancement of social interaction (Olsson et al. 2020). Orbitia is operated by a group of three users, who need to collaboratively operate a drone and steer a rover and collect minerals on a distant planet. While the drone is accessible as a physical object, the complementary steering controls are spatially distributed on personal control panels. Throughout the activity, participants are faced with different “breaches”, i.e., changes of the interface that impact the visibility of parts of the interface and challenge participants’ joint work. Users need to coordinate their actions and available information in a common effort to be able to find hidden items, plan a route, overcome limitations, and collect the required number of minerals.

In this paper we report on a case study, analysing how verbal coordination occurs during different moments of joint work on interactive tabletops. We first describe the design of Orbitia including the modifications that have been done for this study. We then report on the study design centred around five groups of three users completing Orbitia in a lab environment. As part of the included missions, groups need to make use of features with different characteristics and are faced with different challenges related to the interface. To analyse the data, we made use of a mixed-method approach and analysed 135 min of video material along with their transcripts. To identify coordination mechanisms, we applied a top-down coding approach inspired by a taxonomy proposed by Kolbe et al. to the context of collaborative problem solving on interactive tabletops (Kolbe et al. 2012). This allowed us to structure a codebook to describe coordination behaviour in two main categories: Coordination of task-relevant information exchange and Coordination of actions. Our approach allowed us to identify nine mechanisms that describe coordination during problem-solving on interactive tabletops. By making use of descriptive statistics combined with qualitative data, we derive a series of insights related to the interplay between perceptual access, awareness, and coordination on interactive tabletops. Our findings provide new insights for the design of interactive tabletop applications, in particular for situations where coordination is hindered due to workplace characteristics, user abilities, or design tensions of groupware (e.g., (Gutwin and Greenberg 1998)).

2 Related work

In the following, we will first describe existing theories and models on group coordination, then review previous efforts for understanding and supporting collaboration in contexts that are similar to ours.

2.1 Group coordination

For decades, researchers across various academic disciplines have delved into the study of coordination, analyzing processes and identifying mechanisms (Berntzen et al. 2023). Over time, the concept of coordination mechanisms has evolved. While the initial focus was primarily on explicit forms of coordination (Thompson 1967; Ven et al. 1976), the perspective later broadened. It came to encompass the role of relationships and high-quality communication (Jarzabkowski et al. 2012; Kyriakidou and Özbilgin 2013), as well as the shared cognition and mental models that serve as implicit coordination mechanisms in teams (Salas et al. 2005; Rico et al. 2008).

Malone and Crowston define coordination as the management of interdependencies between activities to achieve a goal (Malone and Crowston 1990, 1994). This conceptualization is applicable across various domains and levels of analysis. For instance, Schmidt and Simone focus on work settings, such as air traffic control or hospital work, where groups are distributed across several sites and need to handle complex tasks with numerous interdependent activities. In these settings, additional prescribed artifacts and procedures become crucial to support their coordination (Schmidt and Simone 1996).

In line with this, Kolbe et al. describe small group coordination as the “task-dependent management of interdependencies among group tasks, members, and resources by regulating action and information flow” (Kolbe et al. 2011). They also use the term “coordination mechanisms” to operationalize coordination. It is important to note that they approach the concept from slightly different angles. Schmidt and Simone lean more towards the structural and static aspects (artifacts and their role) (Schmidt and Simone 1996), while Kolbe et al. emphasize the dynamic and interactive aspects (statements or actions during group interactions) (Kolbe et al. 2011).

Coordination can be disrupted, necessitating adaptability to enhance team effectiveness (Salas et al. 2005). Whenever there is a deviation from expected actions, whether due to individual actions from team members or environmental factors, adjustments are required. This suggests that various coordination mechanisms are appropriate for different situations. Furthermore, these mechanisms are employed dynamically based on workload, available support, or task demands (Kolbe et al. 2012).

Wittenbaum et al. identified two distinct dimensions that describe variations in group coordination mechanisms (Wittenbaum et al. 2005). First, coordination can vary in time, meaning it may occur either before or during the actual group work. Second, it can differ in its level of explicitness: coordination might be tacit, based on unspoken expectations, or explicit, rooted in verbal agreements or formally adopted plans. Building on these two dimensions, Boos et al. introduced the Coordination Mechanism Circumplex Model (CMCM). This model categorizes interactions and communications into four cells: (1) pre-process explicit, e.g., rules, instructions, schedules, outlines; (2) in-process explicit, e.g., division of labor, communication about procedures; (3) pre-process implicit, e.g., assumptions about the expertise of group members and task requirements; and (4) in-process implicit, e.g., mutual adaptation of behavior (Boos et al. 2011).

Previous studies, including those by Espinosa et al. and Rico et al., have noted that implicit and explicit coordination generally share similar patterns, enabling team members to manage their interdependencies (Espinosa et al. 2005; Rico et al. 2008). However, each type of coordination tends to exhibit certain behaviors more prominently. In explicit coordination, behaviors such as articulating plans, defining responsibilities, and seeking information are more frequently observed to address the common task (Entin and Serfaty 1999; Rusk and Ståhl 2022). In contrast, implicit team coordination involves behaviors like providing task-relevant information, offering knowledge or feedback to peers without a prior request, and monitoring peers' progress and performance (Espinosa et al. 2005; Rico et al. 2008). When confronted with challenging work situations or non-routine events, teams often shift towards explicit coordination (Grote et al. 2010; Tschan et al. 2006).

Hence, according to Berntzen et al., there have been efforts to refine coordination mechanisms into a more actionable concept that is pertinent in settings where managing interdependencies is vital for achieving desired outcomes (Berntzen et al. 2023; Okhuysen and Bechky 2009). Consequently, understanding, identifying, and developing these coordination mechanisms benefits researchers in structuring their studies and practitioners in managing coordination tasks by choosing the right strategies. In the following sections, we explore existing efforts to understand and describe coordination mechanisms in contexts similar to those addressed in our study.

2.2 Understanding coordination around collocated multi-user interfaces

In the field of collocated multi-user interfaces, coordination has been a recurrent and important theme to consider when analysing group processes. For instance, in the context of collaborative learning on an interactive multi-touch tabletop, Fleck et al. identified two mechanisms that are involved in the coordination of collaboration: (1) joint attention and awareness, and (2) narrations (Fleck et al. 2009). They argue that both physical and verbal aspects are associated with effective collaborative learning and that in particular physical intrusions into others’ working areas may encourage joint awareness. In a similar context, Davis et al. identified coordination as one of two dimensions to describe dyads’ exploration of an educational multi-touch tabletop exhibit (Davis et al. 2015). More specifically, in their work, coordination describes how much the two users’ actions are targeted towards the same goal and complementing each other (high coordination) or whether they conflict with each other (low coordination).

To be able to understand and describe the coordination behaviour in more detail, there have been efforts for providing taxonomies of coordination mechanisms in different collaborative contexts (e.g., (Berntzen et al. 2023; Tang et al. 2012)). Taxonomies allow for systematically identifying, describing, and understanding events and entities in a domain which brings further cognitive efficiency as it eases the reasoning about observations (Ralph 2019; Berntzen et al. 2023). In addition, taxonomies benefit researchers by providing a technical language which is necessary for a clear and precise communication of results (Ralph 2019).

An example of such a taxonomy is proposed by Kolbe et al. and allows to investigate effective and adaptive team coordination of medical teams during induction of general anaesthetics (Kolbe et al. 2012). It consists of three main categories: explicit and implicit coordination, heedful interrelating, and other behaviour. The main category of explicit and implicit coordination includes in total 26 mechanisms related to either coordination of information exchange (e.g., request for information, questioning information) or coordination of actions (e.g., Giving orders, making plans).

Another example of a taxonomy is provided by Tang et al. who investigated verbal coordination in shooter games and identified five categories to describe the different types of call-outs: Directive, About self, About enemy, Question, and Confirmation. In addition, each call-out may have one or more attributes defining the location, urgency, and repetition (Tang et al. 2012).

To summarise, previous works have proven that coordination was found to be an important part of group work on interactive tabletops. In addition, in other collaborative contexts, taxonomies have shown to be an effective tool for identifying, describing, and understanding coordination mechanisms; though, they are often context-specific and cannot necessarily be applied to other scenarios. Currently, no such taxonomy of coordination mechanisms has been proposed for the context of collaborative work involving interactive shared and individual resources in a collocated space, such as the case in collaborative problem-solving on interactive tabletops. In this paper, we will therefore fill this gap by defining a new taxonomy that is appropriate for studying coordination events and entities in this context. Because of the dynamic and interactive nature of group coordination on interactive tabletops, we use the taxonomy of Kolbe et al. (Kolbe et al. 2011) as a starting point and refine it using observations from our case study.

2.3 Designing for coordination on interactive tabletop displays

Shared, multi-user collocated systems, such as interactive tabletops and surfaces provide unique means to support users in engaging with a system in a group situation. Most crucially is the large, shared screen and the possibility for direct multi-user interaction (Mercier and Higgins 2014; Homaeian et al. 2021). For coordinating actions and information around the available resources, awareness is essential. When users can follow each other’s actions, they know if or when a resource is free to use, and they can anticipate the actions and intentions of other group members. In situations of high awareness, users can coordinate their actions effortlessly and seamlessly, with little or no verbal communication (Yuill and Rogers 2012).

Workspace awareness (WA) is defined as the notion of monitoring the activity of others, which provides context for one’s activities (Dourish and Bellotti 1992; Gutwin and Greenberg 2002). WA involves knowledge about where others are working, what they are doing, and what they are going to do next. There are also various subtypes of awareness (peripheral awareness, mutual awareness, reciprocal awareness, social awareness, task awareness, team awareness, historical awareness), but as Schmidt highlighted, the term “awareness in CSCW” is being used in increasingly diverse ways and is over ambiguous and unsatisfactory (Schmidt 2002). In order to support awareness and avoid coordination problems on interactive tabletops, researchers essentially propose to use design features that enhance the shareability of the system, while limiting possibilities for parallel interactions to prevent independent work (e.g., (Marshall et al. 2008; Stanton and Neale 2003; Pontual Falcão and Price 2011)). Consistent with that, more recently, Klinkhammer et al., studied the effect of different workspace settings on territorial behaviour, collaborative processes, and performance in a collaborative brainstorming task. They claim not only that a different workspace setting leads the groups to process collaboration differently, but also that when personal and group spaces are combined, more advantageous conditions for the collaborative process are provided (Klinkhammer et al. 2018).

Gutschmidt and Richter explored personal space and territorial behaviour in a collaborative enterprise modelling session (Gutschmidt and Richter 2021). They argued that whilst certain areas on a tabletop are predominantly occupied by certain individuals, they are not exclusively occupied by single users. Thus, they conclude that in such tasks and settings, the majority of areas on a tabletop are group spaces and consequently, they witnessed less individual and more collective ownership towards the model. Hence, they came to the conclusion that the type of the task is another factor, that can influence the emergence of personal and group territories (Gutschmidt and Richter 2021; Klinkhammer et al. 2018; Niu et al. 2020).

In the context of collaborative visual analytics around a collocated tabletop display, Isenberg et al. seek to support both individual and common activities (Isenberg et al. 2012). They identify the need to include (1) flexible storage mechanisms that support different work strategies and (2) advanced sharing mechanisms that support reconnection after moments of parallel work and reduce the related coordination cost between group members.

In line with these reflections, Morris et al. argue that social protocols are not sufficient for coordinating actions on a shared tabletop display (Morris et al. 2004). They propose a total of 19 coordination policies that are grouped along two dimensions: conflict type refers to the level at which the conflict might occur (global vs. whole element) and initiative refers to the type of strategy applied to solve the conflict (proactive vs. mixed-initiative vs. reactive).

Beyond the interactive tabletop setting, coordination mechanisms in collaborative problem solving have received some attention in recent years. Datcu et al., in the context of a physical and an Augmented Reality (AR) environment, explored the perception of situational awareness and presence. It is reported that the ability to predict what will happen after the action of a peer, and the understanding of each other’s action did not differ very much in both settings. Though, in the AR setting, players were less aware of the environment around them (Datcu et al. 2016). Cooperation scenarios in Mixed Reality (MR) is another field where similar research is conducted. It is reported that for coordination between participants in real-time collocated MR tasks, direct communication and embodied cues are employed as main mechanisms (Prilla 2019). In line with that, physicality is reported as an anchor for coordination, underlining the importance of the shared physical setting for facilitating collocated collaboration. In the absence of a physical artifact to coordinate around (e.g., in AR based scenarios), the shared physical workspace is used for coordination and participants report the conditions as equally effective. While exocentric awareness, referring to the awareness of others’ actions and intentions, is more maintained due to the available communication cues in the physical condition, egocentric awareness, which is the understanding of the own position relative to others and to the environment, is well maintained in the AR setting (Poretski et al. 2021).

To sum up, previous work consistently revealed the benefits of shared physical settings for coordinating actions. In addition, specific features can be provided to address challenging situations, such as the reconnection after moments of parallel work or conflict situations. In this work, we build upon these results to create a problem-solving activity in a shared tabletop setting, that progressively provides limitations to individual and group spaces and to learn how users deal with such limitations in their coordination strategies.

3 Orbitia

Orbitia is a collaborative problem-solving activity, implemented on a MultiTaction MT550 interactive tabletop surface, using Java and TULIP (Tobias et al. 2015). Groups of three participants need to act as a space mining crew. They need to jointly operate a drone and steer a rover to mine valuable minerals while overcoming obstacles and energy constrains. The overall aim is to provide a tool that can be used in vocational training sessions and allows users to enhance their collaboration skills. With Orbitia, participants should experience successful collaboration, and become aware of and reflect on their collaboration strategies.

Several features of the activity provide participants with the resources to complete their missions. While the design rationale of Orbitia’s features has already been explained (Afkari et al. 2020; Maquil et al. 2021), we give here only a short summary of these and focus on the design rationale of the breaches, which are new in this version of Orbitia.

3.1 Main features

The main space of the activity is a 9 × 11 grid located at the centre of the screen, showing most of the elements (see Figure 1).

Figure 1.

Overview of the activity (Mission 1): (1) the rover located in the base; (2) the canyons (to be avoided, causing rover gets destroyed); (3) the mineral; (4) the rock (to be avoided, causing damage to the rover); (5) the battery; (6) personal control panels; (7) the area affected by the sandstorm; (8) the level/breach instruction; (9) the area revealed using the drone.

Each participant can control certain assets of the activity by a personal control panel, spatially distributed around the table (Figure 1(6)). These contain complementary steering directions and complementary information related to three different roles, i.e., energy, mining, and damage. In a coordinated way, participants need to steer the rover towards the desired cell. Since two directions are not directly available (south- and northeast), the steering is a challenging task as it requires the participants to compensate the missing directions by using the available ones.

Furthermore, each control panel shows the current status with regard to the respective role, i.e., energy level, number of collected minerals, or number of spare wheels. Each movement of the rover consumes one unit of energy, and the participants must take the optimal movements into consideration. Furthermore, participants must keep their eyes on the wheels of the rover. Stepping on any cell containing a sharp rock (Figure 1(4)) would cost them one wheel and after repairing the wheels twice, the rover would be permanently damaged, and the mission would be failed. Another element to be careful about is the canyon (Figure 1(2)); there is no way passing a canyon and falling into a cell containing it, results in destroying the rover and the failure of the mission.

Part of the grid, according to the narrative, is affected by a sandstorm and looks cloudy (Figure 1(7)). All the items located on the affected cells are hidden and participants need to use an active tangible object called the radar drone in order to find and reveal the hidden items.

The radar drone contains a Kniwwelino board (Maquil et al. 2019) with a 5 × 5 LED matrix, an RGB LED and two push buttons. It has Wi-Fi stack to connect the Kniwwelino board to the tabletop application over the internet, making use of MQTT as standard IoT message protocols. The position of the drone on the tabletop is detected via an optical marker and computer vision.

When a participant puts the drone on the grid, an area of nine cells (3 × 3) around the drone is highlighted (Figure 2) and the integrated drone display (LEDs) shows the total number of items hidden within this frame, regardless of whether they are minerals, sharp rocks, or batteries. Moving the drone over the grid enables the participants to scan the area containing hidden items. Pushing the button on the drone reveals the items in the framed area for one second, then, a snapshot is sent to the control panels. Snapshots are shown as small grids of 3 × 3 indicating the location of the revealed items in a complementary way: each control panel indicates only the location of the items respective to that control panel. In each mission, the number of snapshots is limited to three.

Figure 2.

The “Radar Drone”; when put on the table the coverage area of the drone is highlighted (a) and shows the number of items hidden in that area (b). By pressing the button on the drone, the area gets revealed (c), and after one second, the revealed items disappear again but a border of the area remains on the grid (d).

From the perspective of coordination, Orbitia provides several interdependencies among the resources provided to participants. First, the actions for manoeuvring the rover are interdependent. To avoid the sharp rocks and canyons, and to limit the use of energy, the buttons need to be tapped in the right sequence. To do so, actions of each participant are both required and indispensable for achieving that goal, and the group needs to apply strategies for regulating the actions. Furthermore, the status information related to energy, minerals, and spare wheels as well as the locations of items is only available to one participant. To be able to solve the task, the flow of information needs to be regulated and pieces of information need to be shared when it is relevant.

3.2 Missions and breaches

Orbitia has five missions and the core of all is the same: participants are asked to collect two minerals and transport them back to the starting point (the base, see Figure 1(1)). However, there are variations in each mission in terms of configurations of the elements and events. Completing each mission is the prerequisite to advance to the next one, yet no new asset is needed for the higher levels.

Although the objective of each mission is evident, participants are tasked with resolving the disparity between their current state and the desired goal state. However, this undertaking is not straightforward, as participants must cope with several challenges outlined in Section 3.1. The distributed competencies of the participants necessitate joint problem-solving, thereby simulating real-life situations where professionals with diverse expertise come together to solve complicated problems.

In the first mission, participants must accomplish only the main task (i.e., steering the rover and collecting two minerals). As of the second mission, in addition to the usual constrains of the activity, a limiting and surprising event called a “breach” occurs. Calling these events ‘breaches’ is inspired by Garfinkel (Garfinkel 2016) ‘breaching experiments’ where ‘the taken for granted’ processes (in our case, commonly accepted procedures of joint problem solving) are intentionally disrupted by built-in disturbances. The latter seek to ‘breach’ the participants taken-for-granted knowing and ways of playing and collaborating, which then must be coped with and repaired by the participants. In Orbitia, each breach affects one feature of the activity (see Table 1 for details).

Table 1. Name and description of the four breaches in Orbitia.

In each mission, the respective breach is triggered after the first retrieval of the mineral and affects different features of the activity. When a breach triggers, the description and instruction of it appears on the top of the screen, complemented by an automatic voice over. The effect of the first three breaches is permanent and lasts until the end of the mission. However, the last breach is different. After overcoming the limitation of the last breach (returning to the starting point within a time constraint), the breach is ended, and participants only have to deal with the usual limitations of the mission.

The rationale behind adding the breaches into the mission was to affect the features of the activity, which were designed to support collaboration, and to observe how the coordination is maintained during a limiting condition. The breaches serve as abrupt events that simulate unforeseen circumstances, introducing new constraints (e.g., time) or challenging assumed resources (e.g., loss of information) within the ongoing process. The order of the five missions described above was the same for each group and defined to best address the underlying educational objectives, i.e., increase difficulty and come as a surprise for the participants.

The first breach (Blank buttons) affects the complementary task of steering and simulates that one or more participants may forget or lose access to critical information. During the breach, the direction arrows on the steering buttons disappear (Figure 3a). Participants then either need to recall their directions or need to try them one more time to learn and memorize them. In either case, the task gets more challenging as the trial and blind movements may result in loss of resources or destruction of the rover, forcing the participants to reconsider the current strategy. From a coordination point of view, we expected this breach to result in less awareness of the competencies of the peers and the adoption of more information coordination and less directives.

Figure 3.

Four breaches of Orbitia. The blank buttons (a), low brightness of control panels (b), overall sandstorm (c), and low oxygen breach (d).

The second breach (Low brightness) limits the distributed information in the control panels and simulates that the working conditions for a team may deteriorate. The control panels bring positive interdependence to the activity by providing each of the participants with only part of the resources needed to complete the task. They also integrate many sources of information which need to be monitored to solve the task and participants need to share knowledge to figure out a solution collaboratively. When the second breach happens, the opacity of the control panels is reduced and results in less visibility for control panels and their elements (Figure 3b). As in the previous breach, we expected that this would lead to less awareness on others’ information. Thus, to maintain collaborative work, participants would need to explicitly exchange regarding the competencies and information owned by their peers.

The third breach (Overall sandstorm) affects the general visibility of the grid as the main shared area of the activity. It simulates that a team might encounter unforeseen and disruptive external factors that impact their ability to communicate and coordinate effectively. Initially, only part of the grid is affected by the sandstorm (Figure 1(7)). When the third breach happens, the whole grid is covered by sandstorm and all the visible elements on the grid, including the rover, are fully or partly covered (Figure 3c). Since this breach affects the common area of the activity, we expected that it would lead to difficulties for maintaining a common focus and understanding. In particular, we expected users to adopt new strategies for keeping track of the location of the rover and items.

The last breach (Low oxygen breach) simulates that a team may face time limitations or unexpected changes that necessitate swift action and adaptation of their course of action. According to the narrative, the oxygen of the rover is running low, and the rover needs to be taken back to the base in 10 s, or the mission will fail. The breach is designed to limit building a common understanding and acting accordingly by imposing a time constraint that does not allow discussing and planning. Unlike the previous breaches, the effect of this breach is not permanent and after overcoming it, the rest of the activity can be handled normally like the first mission (without a breach). However, when the breach happens, participants are forced to withdraw all the strategies and actions in hand and are induced to join forces to overcome the limitations imposed by the breach. With this breach, we expected to observe a different way for coordination as the participants need to react immediately and overcome the limitation.

A summary of all the breaches including name and details can be found in Table 1.

4 Case study

To be able to understand the use of coordination mechanisms in Orbitia in a meaningful context we conducted an exploratory case study. According to Lazar et al. (Lazar et al. 2017), a case study is an “in-depth study of a specific instance (or a small number of instances) within a real-life context”. To be considered a case study, the research design needs to (1) investigate a small number of cases in-depth, (2) examine them in context, (3) use multiple data sources, and (4) mainly use qualitative data and analysis methods.

For this study, we therefore investigate cases of users coordinating with Orbitia in five different constellations, each constellation corresponding to an Orbitia mission that involves a different breach. To be able to observe the use of the activity in a real-world context, we did not design a protocol with small subtasks, but asked the participants to complete the entire activity on their own, in a similar way as it would be used in an educational context, such as a seminar on collaboration. Although the proposed cases were not happening naturally in the real-world, we argue that the context has still very similar configurations as if during a real-world event. Participants were using the training activity at their own pace, without any further instructions or control by the researchers. The sessions we study in this paper took place in October and November 2020.

For data construction and analysis, we used a mixed-method approach involving multiple sources of data including questionnaires and video analysis. We video recorded the sessions using cameras from five different angles. The recordings were first transcribed, then coded with the aim to describe the coordination behaviour throughout the activity. Then, we derived descriptive statistical information to be able to compare different aspects of the data. To obtain a richer understanding of how users coordinate themselves, we complemented the statistical information by qualitative observations from the video data.

Overall, our research strategy is of exploratory nature as we aim to develop first insights about the processes and dynamics related to coordination mechanisms on interactive tabletops. We explore the nature, frequency and circumstances of group coordination in different situations to generate theories and inform future design of collaborative interfaces. We use a descriptive approach of investigation (Lazar et al. 2017), as we seek to provide a detailed and accurate description about what is happening and base ourselves in existing work and theories.

4.1 Research questions

In this case study we wanted to investigate how groups coordinate themselves in the interactive tabletop-based activity Orbitia. Our research questions are:

RQ1: How frequently do group members apply coordination mechanisms and which types of mechanisms do they use?

RQ2: Is there a difference in how groups use coordination mechanisms when comparing two coordination tasks (use of drone vs use of rover)?

RQ3: How do group members adapt their coordination mechanisms when breaches are triggered?

To specify a coordination mechanism, we are inspired by the definition by (Kolbe et al. 2011): “A coordination mechanism is defined as a statement or action by which group coordination is executed during interaction, whereby the interdependencies of tasks, members, and resources by regulating action and information flow are managed”. They argue that performing joint actions requires not only coordination of actions but also coordination of information and provide ‘commands or affirmations’ or ‘anticipation of actions’ as examples of coordination strategies (Kolbe et al. 2012).

In this study, we adapt the above definition of Kolbe et al. by focusing on verbal communication. Therefore, we have set the scope of a mechanism as a statement that potentially might be complemented by an action by which group coordination is executed during interaction, whereby the interdependencies of tasks, members, and resources by regulating action and information flow are managed. We put the main emphasis on verbal conversation as it is a prevalent form of communication, providing us with the required level of detail to identify different types of coordination mechanisms. To be able to better understand the statements made by participants, we additionally took into account gestures or gaze that complemented the conversation (e.g., pointing gestures during talk). Due to the higher uncertainty in interpretation, we have excluded pure gestures without speech. Building upon the proposed definition, with RQ1, we aimed to investigate which types of coordination mechanisms are occurring throughout the entire activity and how often each type is used.

The rationale behind RQ2 is about the two main tasks of Orbitia: (1) coordinating the use of the drone with the aim to reveal items and find minerals, and (2) coordinating the use of the rover with the aim of collecting the minerals and bringing them to the base (Sunnen et al. 2020). Both tasks are required to solve the missions, and they include different types of interaction techniques (a shared tangible object vs distributed touch-controlled buttons). Therefore, we expect that both tasks involve a different set of coordination mechanisms and would suggest implications for the design of the features.

Finally, regarding RQ3, we expected that participants, in order to deal with the different breaches, would use some types of coordination mechanisms more frequently than others. In particular, since each breach targets a specific collaborative feature of the activity, we expected that, for example, after limiting the common focus, more information exchange would happen; or, after a sudden time constraint, we would observe more orders and less planning among participants. In general, succeeding the overall task of the activity necessitates complex coordination strategies, therefore, limiting one collaborative feature may reinforce other coordination mechanisms or provoke alternative strategies.

Generally speaking, in accordance with our explorative research approach, the emphasis of the RQs lies in describing and understanding coordination behaviour in different moments of a realistic and underexplored situation. Since in this context, there are many potentially influential factors, it is not possible to define an experimental design where all factors would be controlled. Instead, with this research we seek to observe and describe the coordination mechanisms using quantitative and qualitative methods, and do not target a statistical analysis with the identification of causes and effects.

4.2 Participants

Five groups of three participants (N = 15) participated in our study. They were recruited from the authors’ working environments using convenience and snowball sampling. Ten of the participants were male and five females. Two were aged between 25 and 34, 8 between 35 and 44, and four between 45 and 54. We asked the participants about their familiarity with the teammates in a form of a scale of one to five. Five reported very familiar, two were familiar, one was somewhat familiar, two reported hardly familiar, and five of the participants were not familiar to each other at all. Most of the participants (12) were staff from the authors’ research campus with different positions (PhD students, engineers, researchers, administrative staff). The other three had different backgrounds (medicine, art, and geography).

All participants indicated that they work hourly to weekly in a team in work-related activities.

In view of the fact that participants had different backgrounds and nationalities, prior to the study, we made sure that all are confident in English, and we formed the groups in a way that English would become the common language. The language of the application was English and all the explanations and instructions by the authors were also given in English.

4.3 Setup and procedure

Orbitia was deployed on a MultiTaction MT550 in the centre of a laboratory room. The drone was placed on the border of the table.

In accordance with the instructions from our ethical committee, we first informed participants of the objectives and context of the study. We explained them what type of data we would record and how we would store and process it. We also informed participants about their rights to withdraw their consent and ask for the deletion of the data at any time and without giving reasons. We provided them with an information sheet and a consent form to sign. To comply to current sanitary measures of the COVID-19 crisis, participants had to disinfect their hands and wear masks throughout the whole user study.

Participants were then led to the laboratory room. To begin, a researcher showed a demonstration mission and explained the main task to accomplish, and the different resources provided, i.e. the elements, features, and roles. The researcher then informed about the number of missions to complete and explained that along the missions, some surprising events might happen, but despite potential problems, they should continue getting the tasks done. Afterwards, they had the opportunity to ask questions and were invited to try out the features. As soon as they felt ready to move on, the researcher started the application and left the room.

All groups were required to solve the same five missions in the same order. The first without breach and the subsequent four each with a different breach. At the start of the application, an introductory text was displayed on the tabletop and read out aloud by an integrated voice. It explained the narrative and the main objective (finding minerals, minding obstacles, and keeping the rover charged). After that all along each mission, a short instruction was provided at the top of the screen, remembering the overall aim of the mission: “Find two minerals and transport them back to the station!”.

At the beginning of each mission, groups were provided with 20 energy units and 2 spare wheels. In case either energy drops to 0 or the rover is damaged and has no spare wheel left, they lose and must start over (with the same initial values). This is then displayed as a subsequent trial. Only when they complete the trial/mission, they can they move on to the next mission. At the end of each mission, participants made a short break to fill out a questionnaire, then continued with the next mission. Overall, the procedure took between 60 and 90 min.‌‌

4.4 Data collection and analysis

The primary source of data was audio-video. We recorded the Orbitia problem-solving process using four fixed cameras (top, front, left, and right angles) (Figure 4). The video data was first transcribed and then coded to identify patterns of coordination and label the utterances accordingly. To respect privacy, participants were immediately assigned pseudonyms, and all documentation was organized with theses.

Figure 4.

The sessions were recorded using 4 fixed cameras installed in a lab environment.

For the analysis, we considered all video data between the beginning of a mission (i.e., the moment when the mission text was displayed) and the end of the same mission (the moment when the success of the mission was displayed). Hence, in case a group required several attempts, we considered all of them for the analysis (including failed and succeeded ones). In addition, we omitted the time in between the missions, i.e., where participants filled in the questionnaires and the transition time between the levels. The duration of the considered material was therefore between 18:41 min and 42:01 min with an average duration being 27:03 (Table 2).

Table 2. Solving time and success rate in Orbitia.

Prior to coding the data, an initial codebook was set up by adapting the taxonomy proposed by Kolbe et al., to the context of collaborative problem solving on interactive tabletops (Kolbe et al. 2012). We decided to use this taxonomy as a basis, because despite being conceived for another context it grouped concepts proposed by previous work on collaboration on interactive tabletops (e.g., (Fleck et al. 2009)). Two researchers then independently applied the codebook to a random extract of the data (100 turns) and compared their results. By discussing potential inconsistencies and identifying needs for additional categories, we then refined the codebook and either grouped existing categories that could not be clearly separated or added new ones. The refined codebook was then used for an additional random 100 turns extract of the data. This process was repeated iteratively until all coding instructions were clear and considered as complete. The intercoder agreement (percentage of agreement) we then achieved was 86.6% (G3 – Mission 1).

The final codebook to describe coordination behaviour can be found in Table 3. Following the work of Kolbe et al., (Kolbe et al. 2012), we structure codes in two main categories: Coordination of task-relevant information exchange and Coordination of actions. We decided to include coordination of information as in Orbitia, information is purposefully distributed among participants, and therefore we expected each group to repeatedly share information that is required to solve the task. We distinguish four different mechanisms: (A) Requests for information, (B) Providing information of request, (C) Verbalizing interpretation of information, and (D) (Dis)agreement to information.

Table 3. Coding categories of coordination mechanisms (categories marked * were changed, and the ones marked ** were added in comparison to the taxonomy of (Kolbe et al. 2012)).

In the second main category, Coordination of actions, we distinguish between five different codes: (O) Giving orders, (P) Making plans, (Q) Planning and procedural questions, (R) Reaction to plan or action, (S) Verbalizing own or other’s behaviour.

It must be mentioned here that when categorising the turns, we had an eye on the context, i.e., taking into account how the mechanisms relate to the preceding or subsequent statements and actions. For instance, we considered if it was requested through a prior statement or if it was meant to trigger any statements or actions.

More information about the different categories and codes can be found in Table 3. Note that the S category (Verbalizing own or other’s behaviour) is strongly related to what Pinelle et al. call verbal shadowing: “the running commentary that people commonly produce alongside their actions, spoken to no one in particular but there for all to overhear” (Pinelle et al. 2003). Therefore, in this paper, we also use the notion of “shadowing” to refer to this type of mechanism.

After having finalized the codebook, the same two researchers independently coded all groups by labelling each turn according to one or more of the proposed categories. After this step, they compared the codes. In case of differences, the researchers looked together at the video data and discussed it until coming to an agreement about the associated code(s).

In addition to the video analysis, we provided participants with a questionnaire to be filled in after each mission. Aim of this questionnaire was to collect demographic information about the participants as well as to record their perception about task difficulty, coordination, and awareness. The cognitive load of the participants was measured over five subjective items (mental demand, temporal demand, performance, effort, and frustration) using the NASA TLX questionnaire (Hart and Staveland 1988). In addition, we designed another questionnaire based on the team effectiveness questionnaire (Wang and Imbrie 2009) with the aim of assessing the perceived collaboration effectiveness. In this paper, we particularly focus on Coordination and Awareness in which to know about them we asked three questions:

• Team members were familiar with one another’s roles and carried out individual actions in a synchronized manner (Coordination behaviour)

• We accomplished tasks smoothly and efficiently (Coordination efficiency)

• At each moment I was aware of what is going on (Awareness)

Each of the items were rated on a 5-point Likert scale.

5 Results

In the following, we will first present the results from the questionnaire, and then the results of the video analysis, structured by research question.

5.1 Questionnaire results

The overall workload (average from all participants) was 51.2 (SD: 16.48) in Mission 1, 38.6 (SD: 13.85) in Mission 2, 41.6 (SD: 15.58) in Mission 3, 46.2 (SD: 16.13) in Mission 4 and 54.0 (SD: 18.09) in Mission 5. Comparing these values with the results from previous studies, the workload is higher as with a previous version without breaches (M: 31.53, SD: 14.0) (Maquil et al. 2021), but similar as with another tabletop-based collaborative problem-solving activity (M: 44.0 and 55.4 for two different versions tested) (Lahure and Maquil 2018).

Over the five missions, participants reported an increase in almost all the items of the cognitive workload (mental and temporal demand, effort, and frustration) as shown in Figure 5. However, the performance was rated relatively the same over the missions. The increase in the cognitive load can be explained by the introduction of breaches that created new difficulties for participants. For example, the last breach (low oxygen) was designed to leave the participants no time for planning, and therefore significantly affected the temporal demand. Other breaches were limiting the participants to obtain the required information for proceeding with the tasks and more effort was needed to maintain the common knowledge (see Figure 5). This practice also increased frustration as the participants reported.

Figure 5.

Perceived subjective workload after each mission.

The results of the collaboration questionnaire are shown in Figure 6. The overall average values for perceived coordination behaviour, coordination efficiency and awareness are very high: 4.5, 4.45 and 4.36 respectively (SD = 0.69, 0.67, 0.79). The main idea of the breaches was to limit the participants in maintaining their coordination by adding constraints to different features of the activity. Since these constraints challenge the process of maintaining awareness, it could explain the subtle drop in the awareness values in last three missions. However, the reported high values for coordination efficiency and awareness might be an indication that participants managed to adopt other coordination strategies that allowed them to maintain a similar level of understanding on the activity.

Figure 6.

Perceived coordination behaviour and awareness.

We also notice a slight increase in the perceived coordination behaviour which could have its roots in the progressively enhanced familiarity with the activity and the successful adoption of a routine in coordinating main procedures among participants throughout the missions. In the same manner, participants reported an increase over the first four missions in terms of perceived coordination efficiency, with the value dropping in the last mission. This might be explained by the success rate of the missions, that was low in Mission 5 (see Table 2). The repeated failures encountered in the last mission might have affected the participants’ perceived coordination efficiency.

5.2 Types of coordination mechanisms

To understand what types of mechanisms are used by groups (RQ1), we provide descriptive statistics about the number of occurrences of the different types, complemented by observations from the video analysis. To describe our results, we distinguish between a turn, which is the contribution of one speaker before the next speaker, and a mechanism, which might be the entire or just part of a turn but can be uniquely assigned to a category (Figure 7).

Figure 7.

Example of two turns (one per line) and four mechanisms in Group 4.

In total, the considered video data consisted of 3442 turns. Of these turns 2715 were associated to at least one category of the coordination behaviour codebook, resulting in 89.74% of the turns being related to coordination. Of the 3482 turns related to coordination, 641 were composed of two mechanisms, 88 of three mechanisms and 8 of four mechanisms. We did not have a limitation for assigning the turns into categories as sometimes the turns were long and consisted of different phrases which would fit into different categories. After assigning all the turns to the defined categories, we noticed that there is no turn that is assigned to more than four categories.

Turns that could not be associated to a coordination category were, on one hand, personal or general statements related to applied strategies, plans, or actions which are not relevant to the current task. For example, these concerned the expression of feelings (e.g., “Okay. A lot of pressure here guys”), or reflections on previous actions or past events. For example: “Yeah. So, we had one problem, there was not enough energy”. Other turns that were not related to coordination consisted of unfinished words and meaningless phrases, or interjections such as “ummm”. Finally, there were sometimes off-topic discussions that participants started in between the missions and continued at the beginning of the next mission.

The total number of coordination occurrences to solve the five missions per group ranged between 659 (G3) and 1465 (G4), with an average of 851.8 occurrences per group. On average, there was a higher number of action coordination occurrences (M: 56.58%, SD: 6.9) than information coordination occurrences (M: 44.91%, SD: 6.29). The distribution of the types of mechanisms is shown in Figure 8. On average, C (verbalizing interpretation of a situation) was the most frequently observed coordination mechanism (26.32%; M: 224.2; SD: 137.2), followed by R (16.06%; M: 136.8; SD: 24.6) and P (13.85%; M:118; SD: 44.6). The mechanisms that were used least frequently used were B (3.76%; M: 31.8; SD: 9.5), Q (4.06%; M: 34.6; SD: 3.4) and A (5.05%; M:43.6; SD: 16.9).

Figure 8.

Average distribution of coordination mechanisms.

These numbers show us that groups commonly coordinated their information by sharing unsolicited, task-relevant information. Our observations showed that these often concerned the location of items and the current location of the rover (“The battery is there and we are there. There is no rock I think” G3 00:28:13) or the feedback provided by the drone (“Yeah, here is two” G4 00:23:01). Other typical examples include the status of mining, damage, or energy (“Battery. So we're full, almost.” G3 00:38:47), or the available directions (“So yeah, this one right here was straight.” G3, 00:10:07). Sometimes, we also observed groups to share information about personal circumstances, explaining, for example, if they are aware of an information: “Yeah, but I, I remember mine. Do you remember yours?” G1 00:09:51).

Information requests were only seldomly done, with part of them not being answered (on average 43 questions, but only 31.8 answers). In a similar way, agreements and disagreements where not expressed for all shared information (on average 224.2 occurrences of C, but only 102.2 occurrences of D). In our view, one explanation could be that the participants became sufficiently familiar with the task and resources that they were adopting strategies minimizing the effort. According to (Entin and Serfaty 1999), unsolicited communication can be considered as implicit coordination, happening when a shared mental model of the task requirements is built and team members can coordinate the task with minimal resources. With the knowledge participants had on the task and one another’s resources, team members were able to anticipate actions and events and provide task-relevant and timely information without having been asked to do so.

For coordinating their actions, groups used a variated set of mechanisms, including mainly R, P, O and S. We observed that plan making (P) was sometimes expressed more generally (“So we need one. To find one more mineral.” G2 00:15:08), sometimes in more detail, including the exact sequence of actions (“One here and then I do the rest” G3 00:31:16). Noticeable is also that plans are often build together, with one participant complementing the sentence of another participant (see Transcript 1).

Transcript 1. Example of planning in group 3. One participant complements the sentence started by a peer.

00:31:11	FLO:	Then we go and get the, uh…	P
00:31:14	GIA:	The diamond	P
00:31:16	FLO:	Yeah. One here and then I do the rest. Twice. One. Two	P	O	S
00:31:25	LEE:	Oh yes, yes. You can go	R
00:31:27	FLO:	I collect the diamond	S

In the vast majority of cases, orders (O) concerned the next move to be taken. These could be very explicit (“And you go this way.” G1 00:15:14), but most of the time they were relying on contextual information such as previous actions (“Another one” G1 00:02:29) or the knowledge on who has which direction (“Luc it's you.” G5 00:26:33). We also noticed that a common directive was related to stopping the current actions, e.g., “wait” or “stop”.

Some of the verbalizations of behaviour (S) described explicitly what the person is doing (“Okay I grab the battery” G5 00:10:19). However, similar to planning, instances were often becoming considerably shorter over time, e.g., “charge” or even just “yes”, confirming that the button is pressed. Noteworthy is also that verbalizations do not only concern own actions but also actions of other group members, as shown in Transcript 2.

Transcript 2. A participant of Group 4 is verbalizing the action of a peer.

00:03:49	ABE:	Uh, wait. Yep	O	R
00:03:50	AXL:	Yes. Okay	R
00:03:52	ABE:	You collect it	S
00:03:52	AXL:	(Yeah)
00:03:53	ABE:	One more in this direction. And now you go to, to home yeah	S	P

Like in information coordination, questions seem to be of less importance in action coordination than the other mechanisms. The questions in the action coordination phase were used mostly as reassurance on the next action, either in the individual level or in the group level. For example, the group agreed on taking a route towards the mineral and Eli starts the movement. Then, when Sam needs to move, before executing the action he asks: “So, I go just like this? G1 00:12:05” as an attempt to reassure about the coordination of current state. Another example which is in the group level, was pressing the button of the drone. When participants agreed on one location to reveal its items, before pressing the button on the drone, the person in charge was asking a reassuring question such as: “shall I click? Shall I try here? G2 00:19:47”.

Overall, when comparing the distribution of mechanisms between the groups (Figure 9), the results from G4 vary considerably from the remaining groups, both in terms of total occurrences as well is from an individual scale. Indeed, for G4 more than twice as many total coordination occurrences were observed as on average for the other groups (1465 vs M: 698.5, SD: 36.72). Interestingly, the difference does not vary equally across the different categories. The difference is most considerable for the mechanisms C and D where G4 had about 3 times the number of occurrences as compared to the average of the other groups. On the other hand, the number of occurrences of O, Q and R is only little above average.

Figure 9.

Coordination mechanisms per group.

Based on the video data, we could explain this difference by the approach of G4, where group members frequently repeated observations several times. Compared to the other groups, they spend much time to carefully explore the entire field with the drone, and repeatedly communicated the numbers they saw and concluded where the minerals potentially might be hidden. The other groups were less engaged in this task and revealed an area after only a short moment of exploration.

Despite G4, differences can also be seen across the other groups, most noticeable regarding O (giving orders). While G2 and G3 used orders only 41 and 63 times, G1 and G5 for 112 and 113 times respectively. In contrast, G2 was more active in using R (reaction to plan or action) and G3 more active in P (making plans). These numbers show us that groups can have different approaches to coordinate their actions during problem-solving, which is in line with previous work in the field of interactive tabletops studying group dynamics (e.g. (Rick et al. 2011)). A reason for the different approach might be the level of familiarity between teammates. Our results with the system of Orbitia have shown that in particular directives and shadowing are not used in a similar way by each group.

5.3 Coordination in different tasks

To answer the second research question about the coordination mechanisms related to the task, we separated the turns (and related mechanisms) according to the coordination tasks. The overall distribution of the mechanisms, separated by type of task, is shown in Figure 10.

Figure 10.

Coordination of drone vs. rover.

During drone usage, the most frequently used mechanism is C (Verbalizing interpretation of situation), followed by D ((Dis)agreement to information). Our observations revealed that this information typically concerned verbalizing the number shown on the drone or repeating the location of items after revealing them by pressing the button. This might be explained by the functionality of the drone, that intend to provide users, slowly and stepwise, with the required information about the location of items.

In the context of using the rover, the most frequently used mechanism is R (Reaction to plan or action), followed by O (Giving orders), P (Making plans) and S (Verbalizing own or other’s behaviour), representing between 30.90%, and 16.16% respectively of the total coordination mechanisms The frequent use of these mechanisms shows us that all of them were useful for steering the rover and that there does not seem to be a mechanism that is predominantly preferred.

Overall, action coordination mechanisms were used less often during drone usage as compared to rover usage with largest differences related to O (Giving orders) and S (Verbalizing own or others’ behaviour). Here, an explanation might be the tangible nature of the drone. According to (Hornecker et al. 2007), manual manipulation of physical objects provides enhanced visibility and legibility of others’ actions, meaning that observers do not only see the actions, but can also easily make sense of it. This enhanced awareness might allow teams to anticipate actions and react to intentions of others without the need of verbally communicating what happened or what should be happening.

Another explanation for this difference in coordination mechanisms might be the co-dependency (Antle and Wise 2013) embedded in the steering of the rover. Since the steering directions are distributed among the group members, all of them need to contribute in order for the group to be successful. Therefore, all of them need to be constantly aware of what is going on and react at the right moment. As a response for this constant need of joint awareness, group members do additional work to structure their behaviour and provide observable cues to others, i.e. what is called in literature ‘awareness work’ (Hornecker et al. 2008). It becomes apparent here in the form of more extensive planning, shadowing, giving orders, and verbal agreements/disagreements. This observation suggests that the design of the joint interaction techniques can not only create different support of awareness, but also different needs of awareness, that groups address by enforcing action coordination in general, with an emphasis on giving orders and doing shadowing.

5.4 Coordination during different breaches

Concerning the third research question, dealing with the coordination processes, we separated the turns based on the breach that was active at that time. To make the data comparable, we calculated the occurrences over time (i.e., per minute). Furthermore, we only considered the time period when the breach was active, i.e., generally from the moment the first mineral was collected until the end of the mission. Only for M5, we considered the timespan from the moment the first mineral was collected until they failed, or the rover was back to the base. As a benchmark, we also added the data from the same time span in M1, i.e., the first mission where no breach was triggered.

The overall frequency of coordination mechanisms during these timespans is shown in Figure 11. The results show that the frequency of coordination mechanisms during breaches (M2-M5) was higher as compared to M1. The highest frequency was observed during the breach of M2 (blank buttons) with 42.06 mechanisms per minute, the lowest during the breach of M5 (Oxygen) with 30.32 mechanisms per minute. The reason for the higher frequency during M2 might be that this was the first breach happening and was therefore most unexpected and most interfering with their previous way of working. While in M5, the breach came also quite unexpected for participants, almost all groups were required to repeat the mission several times. The general approach during this breach was then to rigorously plan the whole route prior to triggering the breach. Therefore, when putting the plan into action, only little communication was needed to coordinate the actions.

Figure 11.

Total frequency of coordination during breaches.

To analyse the differences in more detail, we had a look at the frequency of each type of mechanism during each breach. As depicted in Figure 12, the results show that A (Requests for information) and B (Providing information of request) are most frequent in M2 (Blank buttons), followed by M3 (Low Brightness) and M4 (Sandstorm).

Figure 12.

Frequency of coordination during breaches.

In our view, when the buttons go blank in M2, participants are deprived of gaining the information related to their peers’ competency and therefore, they explicitly asked for it. This explains why the values of the A and B categories are higher in this mission. Likewise, M3 (Low Brightness) and M4 (Sandstorm) are limiting the participants from the shared information. In both cases, to be able to build a shared understanding of the task, participants rely on obtaining the needed info in an explicit manner (see Transcript 3).

Transcript 3. Group 1 participants are sharing the information explicitly after facing the limitation of the breach in mission 2.

00:09:49	SAM:	So, there's no directions now	C
00:09:50	ELI:	Yeah, but I, I remember mine. Do you remember yours?	D	C	A
00:09:53	SAM:	Yes, this one is like this and this one is like this	B
00:09:57	IVA:	This one is like this and this one (xxx)	B

Figure 12 also shows that, regarding M1-M4, the value of C (Verbalizing interpretation of situation) is similar across the missions and D ((Dis)agreement to information) is slightly increasing. Without breaches, we would expect the task to become more familiar over time and this would lead to an establishment of a routine and less exchange of information in general. However, the breaches were limiting the information sources, and as well limiting the participants to see the immediate effect of their taken actions (e.g., the rover movement under the sandstorm is partly not visible). To overcome this limitation, participants tended to express their own understanding related to the affected areas (C) more frequently, which might have led to this overall constant level in the C category.

Missions M2, M3, and M4, generally seem to involve more O and P as M1, which might be the result of the higher degree of difficulty because of the breaches, that requires the groups to better plan their route and be more cautious about its execution. The highest frequency of orders can be found in M5. This was to be expected, as participants were under time pressure, and therefore were mainly giving directives one to each other (see Transcript 4).

Transcript 4. An example of orders (O) in Group 1 when dealing with the last breach in mission 5.

00:33:55	System announcement:	Low on oxygen, return to the base in 10 s
00:33:57	IVA:	Ah
00:33:58	SAM:	Okay. Oxygen	C
00:33:58	ELI:	10 s? Okay, 10 s, uh-	C
00:34:00	SAM:	Go, go, go	O
00:34:02	ELI:	Go	O
00:34:02	SAM:	Go	O
00:34:03	ELI:	No, no, no, no, wait, wait	O
00:34:05	SAM:	Back. Back	O
00:34:05	ELI:	You, you, you, your turn, your turn	O
00:34:07	System announcement:	Rover destroyed

Noteworthy is also the result that S (Verbalizing own or others’ behaviour) is highest in M4 and M5. In M4, the entire grid as the main shared space of the activity is affected by the breach (overall sandstorm), which makes it difficult for the participants to be sure about the gained information. In such a case, participants compensate the missing visual cues for reassuring about the coordination by verbalizing the next action, either their own or action of the peers (e.g., “My turn again. G3 00:43:05”; “I go up another one.” G2 00:10:35). This could be explained as a feedback on coordination in a way that not only catches the attention of all and raises the awareness to the current task, but also provide opportunities to correct the errors or misunderstandings. The same principle applies to M5 in terms of the S category, however, in this case, instead of compensating the lack of visual cues, participants compensate the lack of time for coordination by verbalizing the action as reassurance.

6 Discussion

This explorative case study was a first step to investigate how users coordinate their work on an interactive tabletop display to jointly solve a problem. We analysed what types of coordination mechanisms participants performed and took into account the subtask they were currently engaged in, as well as which breach was activated. Using a top-down coding approach, we categorized coordination mechanisms, on a first level, into coordination of actions and coordination of information. On a second level, we defined subcategories, taking into account how the mechanisms relate to the preceding or subsequent statements and actions, so considering, e.g., if it was requested through a prior statement or if it was meant to trigger any statements or actions. Through this approach we could identify four mechanisms for coordinating information and five mechanisms for coordinating actions.

Our taxonomy of coordination mechanisms was designed for the context of joint problem-solving and interactive tabletops. As in the original taxonomy (Kolbe et al. 2012), we distinguish between the coordination of actions and the coordination of information, but we do not differentiate between implicit and explicit mechanisms as we focused on verbal communication. The most pertinent difference is related to shadowing (“verbalizing own behaviour”). While in the original taxonomy (Kolbe et al. 2012) this mechanism is considered as being part of heedful interrelating, we found it as an important component of action coordination that covers 8.7% of all observed mechanisms. According to Pinelle et al., verbal shadowing is one aspect of the spoken communication in explicit coordination (Pinelle et al. 2003). They argue that the intention in normal conversation is to convey a specific message, and in contrast, shadowing enables people to stay aware of what the person is doing and why (Pinelle et al. 2003; Clark 2018). In our case, we could observe participants to verbalize behaviour throughout many moments, and describing not only own actions, but also the ones of fellow members.

Our quantitative results of coordination mechanism occurrences showed that the vast majority (89.74%) of verbal exchanges can be considered as being part of coordination. Next to the coordination of actions, the coordination of information covers an essential part - almost half of the coordination mechanisms (M: 44.91%, SD: 6.29). This seems surprising as Orbitia is a shared interface and therefore allows group members to see the same information on the joint screen, and to be aware of what the others see. Since most of the information can be seen by the whole group, we could expect that there is no need to repeat the information aloud. The results from our study show that groups frequently verbalized what they see or how they interpret the information regardless of whether they are steering the rover, operating the drone and/or faced with any of the breaches. We witnessed that participants were exchanging unsolicited information frequently forming their common understanding of the current state of the activity.

By considering the coordination task, we could generally show that coordination mechanisms adapted to the situation and that the tangible nature of the drone, and the co-dependency of the controls might be the reason for this. Previous work has already shown how the modality and arrangement of user interface elements can impact collaboration by facilitating awareness work (Hornecker et al. 2008), enforcing turn-taking (Piper et al. 2006), or promoting equitable participation (Fan et al. 2014). Our results complement these findings by showing that joint interactions with a shared tangible object involved a different coordination behaviour as the ones with co-dependent, distributed touch controls. With the latter, there was overall more action coordination and in particular more orders and more shadowing.

During the breaches in M2-M4, group members were lacking some of the visual cues that they used priorly for coordination. Nevertheless, the results of the questionnaire showed that the perceived coordination efficiency and awareness remained high throughout the whole activity. An explanation for this might be that participants adapted their coordination strategies and used different mechanisms in order to maintain a similar level of awareness and coordination efficiency. In M2 and M3, the affected areas concerned the personal control panels, leading to reduced accessibility of the information related to the peers’ competencies. With our data, we found that in this situation, participants adapted to using direct requests for information as a more explicit manner of communication. In M4, on the other hand, the visibility affected the common area. Our results showed that in these moments shadowing was enforced. M5 was of different nature, and while keeping the visibility of all interface elements intact, it was putting time pressure on participants. In Orbitia, this time pressure led to reduced planning, and enhanced directives and shadowing.

Previous work in different contexts has already shown that shifts to more explicit coordination mechanisms are common in challenging situations. During implicit coordination, according to Entin and Serfaty, team members rely on a shared mental model and common understandings of the task developed over time (Entin and Serfaty 1999). However, when maintaining the shared mental model of the current state of the task is limited, the team shifts to explicit coordination and adopt mechanisms, such as, providing information upon request, requesting help, reassuring, or providing a summary. In our study, we could observe such a shift to explicit coordination with the adoption of direct information requests in M2 and M3. However, we could also see that in the case of the sandstorm, the groups enforced essentially shadowing as an implicit form of communication.

On one hand, this might be an indication that the enforcement of explicit coordination is not the only way of dealing with challenging situations, and that an alternative way is to increase implicit mechanisms such as shadowing. Another aspect, however, that must be considered is that the dimension of explicitness in coordination needs a more fine-grained categorization to be suitable to describe work at interactive tabletops that includes non-verbal communication. Potentially, participants adopted shadowing instead of using pure gestures and in this perspective, shadowing could be considered as a more explicit way of communication in comparison to sole gestures. More work is needed to study explicitness in the coordination on interactive tabletops and needs to include gestures and body language.

6.1 Limitations

Being an exploratory case study, this work focusses on a specific application and is based on a small number of participants that do not serve as sample for a large population. As professionals working in the research field, participants were all quite confident in collaboration and familiar with technology and problem-solving in this context. Furthermore, the study took place in a multilingual country and not all participants were equally fluent in English. This might have impacted the results that cannot necessarily be generalised for other contexts (e.g., different age group).

With regard to the tasks, the order of the missions was not counterbalanced, and the experience from the prior levels might have had an impact on how participants solved them and organized their coordination. Furthermore, our study is focused on verbal communication. The inclusion of non-verbal communication (gestures, posture, gaze) might reveal additional mechanisms, in particular related to implicit coordination.

6.2 Relevance beyond interactive tabletops

Despite the fact that this work was conducted in the context of interactive tabletops, there are several aspects that can be applied to other technological settings. Through their shared screen and the possibility for simultaneous interaction by several users, interactive tabletops provide excellent affordances for collaboration (Mercier and Higgins 2014; Homaeian et al. 2021). Similar characteristics can, however, be provided by other large interactive displays, such as interactive floors, wall-sized displays, or multi-surface environments. These related systems provide one or more common, shared spaces, and potentially, in addition, personal spaces. As the designed problem-solving activity makes use of both personal (control panels) and group areas (activity grid), these could be transferred to other large display configurations in a way that the interdependency of actions and information is preserved. Although characteristics such as size, orientation and physical reachability would impact how and users would distribute roles, explore ideas and collaborate (Rogers and Lindley 2004; Gutschmidt and Richter 2021) the nature of the information and actions that need to be coordinated would still be similar, and therefore, we expect that the taxonomy of verbal coordination can be applied to such settings as well.

Similar observations as ours, such as shifts towards more explicit coordination during breaches (Entin and Serfaty 1999), and less verbal coordination during physical manipulation tasks (Hornecker et al. 2008; Kolbe et al. 2012; Wittenbaum et al. 2005) have been witnessed in related studies that took place in different setups or contexts. Thus, we can also expect that breaches in other display configurations would involve a comparable adaptation of coordination strategies as we observed in our study. Further studies are however needed in order to fully understand the impact of the physical space, task characteristics, and interface design onto the proposed coordination mechanisms.

7 Conclusion

In this paper, we have presented the results of an exploratory case study investigating coordination processes during a joint problem-solving task on an interactive tabletop display. We have described the design of a series of breaches, integrated in an interactive tabletop application, seeking to challenge participants’ collaboration by providing them with different limitations. Using a mixed methods approach, we have analysed the coordination processes of five groups with three participants each. The unique contribution of our work lies in its focus on coordination processes by building on existing theoretical frameworks and studying the verbal communication around interactive tabletops in a context with varying limitations related to the interface.

The results of our study show first insights related to the nature and frequency of coordination as well as its link to design aspects of the interface. We have shown that participants’ coordination mechanisms deal both with actions and information and identified nine types of mechanisms occurring during joint problem-solving on interactive tabletops. The mechanisms were dynamically used and varied depending on the coordination task and the limiting factors related to the breaches. We found that sharing unsolicited task-relevant information is a common coordination behaviour that can be observed at interactive tabletops, and that after breaches, more explicit coordination in the form of direct requests, orders, or shadowing is used. Our results are relevant beyond interactive tabletops and can be used and refined in other collocated settings involving large interactive displays.

With our work, we were able to show how efficient coordination is carried out on an interactive tabletop and how it can be maintained despite limitations in the perceptual access of shared and personal areas. We contribute towards a better understanding of how tangible and multi-touch characteristics of interactive tabletops enable or enhance different coordination strategies. These results can be applied to a broader range of technological systems of shared nature, that might provide different limitations with regard to perceptual access and physical reachability. For instance, in a multi-surface environment, the visibility of a common screen might at some point be occluded by users standing in the way and would then create a similar situation as we investigated during our study. Ultimately, our work serves also to enhance coordination in setups where there is no common display, as it is the case in AR/MR or during remote collaboration. Understanding how users coordinate themselves with shared systems in collocated settings, might allow us to build the required support for improving coordination across space or reality.