Keywords

1 Introduction

In this research study, we present coding schemes for structured video observations of tangible collaborative decision support systems designed for urban planning [1]. Testing the usability of collaborative TUI in the context of urban planning has not been fully addressed in the Human-Computer Interaction (HCI) design literature [2]. Therefore, applying behavioral coding in multimedia to efficiently analyze the user experience in video recordings is important. Behavioral coding of observational videos provides researchers with a lens for studying human interactions. HCI studies have examined video coding for behavioral analysis (e.g. coding schemes for non-verbal behavior of autistic users in interacting with assistive technologies, coding schemes in multimedia surveillance systems for emergency-response systems [3]).

We designed coding schemes to help in understanding the user behavior, evaluating the severity of usability issues, and identifying collaboration breakdowns and problems observed during the evaluation of collaborative TUI systems. The coding schemes were developed as an adaptation of the DEVAN scheme [4] for video analysis and coding behaviors observed in usability testing of interactive systems, which take into account the specific interaction modalities in collaborative TUIs. Examples include the perception of embedding digital information in physical objects (e.g. visual, tactile and auditory perception) as well as gesture-based actions for different application domains. The coding schemes were elaborated iteratively in the context of designing an urban modeling and planning system, until a satisfactory reliability is achieved.

In an observational study we’ve conducted to examine the usability of a collaborative TUI system in the urban planning context [6], we considered different types of coding schemes in the process of analyzing video observations. The aim of this paper is to propose a coding scheme framework for collaborative TUIs that could be followed and applied on performance metrics in task-based usability analysis of TUIs.

2 Coding Schemes

2.1 Coding Scheme Structure

In our study, we considered three types of coding schemes for analyzing user interactions in collaborative TUI systems in the context of urban planning: actions, verbal, and non-verbal gestures. Coding schemes were applied to quantify the occurrences of these interactions between the user and the TUI system (User-to-Object), and between the users (User-to-User & User-to-Many) as shown in Fig. 1.

Fig. 1.
figure 1figure 1

Structure of the coding schemes

Full size image

2.2 Codes for TUIs in Urban Planning

The codes were designed to be applied in task-based usability testing by mapping the codes to incidents and observations in time-stamped observational videos. A team of observers can apply independent coding, and agreement levels can be examined to ensure consistent coding across observers. Trends and patterns of usability and user experience (UX) measures emerge from the quantified dataset of coded observational video recordings.

2.2.1 Actions Codes

The breakdown indication types of DEVAN were applied, such as the ACT codes for wrong actions, and the DISC codes to indicate moments in which users discontinue actions [4]. One breakdown indicator was introduced for interrupting an action during collaborative TUI-mediated interactions; the code was defined as INTER, where an action is interrupted by the system or another action.

2.2.2 Verbal Cues and Non-verbal Gestures Codes

Like in actions codes, verbal cues codes in DEVAN [4] were adapted, such as the PUZZ code for incidents in which users exhibited confusion during interaction with TUIs, and the FRUST code for frustration.

The coding schemes for verbal communication and gestures in collaborative architectural design in [5] were also adapted; they include communication control, communication technology, and design communication. An example of a communication control adaptation is the floor holding code FLO. In the context of collaborative TUI, the gesture-based cue of one user grasping a physical object of the TUI to initiate an action considers as floor holding. Another example is HAN for hand-over occurrences; the coding could indicate relinquishing the floor around the TUI or a physical hand-over of the physical object in the TUI. We introduced several breakdown indicators specific to the context of TUIs as shown in Table 1.

Table 1. Verbal and non-verbal user behavior codes
Full size table

3 Conclusion

In designing collaborative tangible user interfaces (TUIs) for complex systems, the usability analysis of such systems is important. Utilizing behavioral codes to rapidly analyze the user experience has been shown to be effective in testing and designing TUI systems [2, 6]. The work reported in this study contributes to the UX and usability methods’ body of knowledge by introduced an aggregate coding scheme for the usability evaluation of collaborative TUI systems in the context of urban planning and design. Preliminary evidence in applying this coding scheme suggests the efficacy of these schemes in assessing the usability and UX of TUIs. Further work will examine the sensitivity of these schemes in large-scale usability evaluations of TUIs designed for city planning.