KeywordS

1 Introduction

Whiteboard is an important tool to support user’s learning and information exchanging in scenarios such as classrooms and conference rooms. As a presentation media that combined with annotating features, whiteboard allows multiple users to visualise and communicate their thoughts with marker pens that can be massively produced and applied in versatile interaction scenarios [1]. Due to the advances in display technologies, nowadays whiteboard has integrated graphic display functions (e.g. e-whiteboard [2]), and the interaction began to rely on assistant apparatus. In addition, natural interaction technologies are increasingly being embedded in whiteboard interaction (e.g. pressure-sensing drawing [3]). This introduces more peripheral equipment in whiteboard interaction, with which numerous whiteboard applications are designed such as pick-and-drop pen for cross-computers file transfer [4] and cross-devices information sharing system [5]. So far, whiteboard has become not only an interactive tool but also a hub of apparatus that enables various interaction tasks such as idea sketching, sharing, and learning [6].

However, it is both time- and cost-consuming to implement specific apparatus for different whiteboard interaction tasks. For example, digital stylus replaced conventional marker pens, but which required extra system configurations to maintain consistent sensitivity across different whiteboard sizes and platforms [7]. Furthermore, adding or removing interaction modalities in whiteboard interaction faces other difficulties due to that the specifically developed apparatus are often deeply coupled with whiteboard tasks (or functionalities) [1]. Given these tightly-coupled apparatus, whiteboard interaction becomes uneasy of introducing new interaction metaphors derived from natural physical-world objects [8]. For example, whiteboard marker dispenser is designed useful in large size whiteboard, but it runs into practical difficulties when transferring to a small tablet due to the limited screen real-estate [9]. Similar problems are anticipated when users attempt to configure one marker dispenser for other purposes of use.

Increasing attentions are attracted to improving the generality of whiteboard’s peripheral apparatus and meanwhile to lowering the development cost across interaction tasks [7]. To bridge the gap, we designed the cross-objects user interface that was capable of recognising everyday objects (Lego Rubik’s cubes in this case) and configuring these as manipulation tool in whiteboard interaction. Furthermore, to understand effectiveness of the user interface, we constructed a low-fidelity prototype and conducted within-subject evaluation. The main contributions of the study are two-folder. Firstly, it pioneers the implementation of the cross-objects user interface that has generalising use in whiteboard interaction; secondly, it provides new understandings of how effective such user interface is used in whiteboard interaction.

The remainder of the paper is organised as follows. Section 2 give a literature review that mainly covers whiteboard interactions and cross-objects user interface. Section 3 describes methodological details of study as well as the data analysis. Section 4 gives a statement of study results, and Sect. 5 discusses the findings and related implications. Finally, Sect. 6 concludes the study results and implications.

2 Related Work

2.1 Whiteboard Interaction and Peripheral Apparatus

Whiteboard supports people sketching thoughts and sharing ideas. Various sizes and placements of whiteboard have enabled different forms of interactions such as industrial design sketching and multi-user collaborative tasks [10]. Particularly, whiteboard is a great colour tool for demonstrations, as it can highlight important features of objects with the stylus that incorporates different width and colour marks [11]. In addition, the interaction of whiteboard accommodates different types of learning styles [1]. Tactile users can benefit from touching and marking on the board, and such sense can be further developed by combining with other interaction modalities such as music and spoken discussions [12].

Whiteboard interfaces well with other peripherals [10]. With projectors, it displays images that collaborate with user’s marking; and with sensor-integrated stylus, it enables spatial drawing in front of board [7]. As the recent Microsoft Surface Studio demonstrates, not only conventional stylus is embedded in an altered whiteboard system (which is a large size display screen), but also physical objects such as the cylinder dialler can be added as manipulation tool. Given these examples, we see an explicit trend of that the whiteboard interaction is increasingly integrating display and interactive technologies to facilitate different types of natural interactions. So far, apparatus is developed and integrated in whiteboard systems, so to expand the interaction range as well as methods.

However, the generality is often limited when the apparatus is specifically implemented for distinguishing interaction tasks. The limit not only refers to electronic apparatus that are developed for concrete functions, but also covers tangible user interfaces that involve everyday objects as manipulation tool [13]. For example, cross-screen file transferring requires specially developed stylus hardware and software [4]; spatial marking whiteboard needs corresponding pre-use calibrations and configurations that are troublesome to normal users [13]; and in the contexts of education, multiple users hold pens that require special wireless networks to synchronise connection and interaction status [14].

The generality problem is justified for two reasons. Firstly, there are various operation systems and configurable devices currently coexisting in whiteboard interaction environments. For example, whiteboard’s image display functionalities can be realised through projectors and large scale flat screens; and the marking functionality can be implemented by digital stylus and fingers. The diversities of existing system and environment configurations raise the necessity of proposing interaction apparatus for specific scenarios and functionalities. Secondly, the tasks of whiteboard interaction are often distinguishing, that is, for example, selecting a distal item in whiteboard performs differently with spatial-stylus and finger-pointing methods [15]. This stimulates developers and researchers to explore novel whiteboard interactions at a specific task-oriented point of view. In the contrast, the reverse way – to make the interaction more generalising – is less concerned in current studies. The consequences are, firstly, it becomes uneasy of using natural object-based interaction metaphors due to the high implementation cost; and secondly, there are more difficulties in integrating new techniques and apparatus in existing whiteboard interaction platforms and scenarios.

2.2 Tangible User Interfaces and Cross-Objects User Interface

Tangible user interfaces, which were setup to “make computing truly ubiquitous and invisible”, have been considered as one of effective solutions to the generality problem in whiteboard interaction [16]. A body of research in computational systems emphasised on the importance of physical-world object modalities in interactions [17]. In addition, research systems began to rely on the physical artefacts as representations and controls for digital information [17], although the characteristics of the interactions with the artefacts have yet to setup systematic frameworks. To extend this limit, researchers take a significant step in the direction of “graspable user interfaces [16] ”.

As [16] claimed, the digital-world information is being coupled into physical-world objects, so to transform physical objects into configurable interfaces. In the contexts of whiteboard interaction, researchers explored this concept and delivered implements. For example, making whiteboard a smart clustering of free-hand sketches [18], integrating ambient light system to add value in meeting and control rooms [19], and using large interactive surface to track user’s deictic gestures [20]. The advantages provided by these systems include a tighter coupling of physical-world objects and cyberspace interfaces as well as the greater interactivity of whiteboard [21].

However, these systems were often tightly coupled with digital apparatus. For example, a spatial position recognition is uneasy of transforming to another whiteboard [17], not to mention the software toolkits that were tailored for special use [22]. To address this issue, a body of research constructed frameworks that were adaptive to different interactions. For example, implicit interaction was proposed to support interactions across different electronic whiteboards [2, 23] and framework of physical objects was imported to enable cross-device whiteboard interaction [5].

To enhance the generality of the apparatus, researchers used sensing technologies such as RFID to recognise and mark objects [24]. For example, digital stylus equipped with RFID tag has names and attributes to transfer digital files across devices and users [4]. This apparatus still required specific equipment and development work, but it provided inspirations of that everyday objects could be a part of unified user interface.

Object recognition often requires sensor embedment and that involves extra development setups. Electromechanical technique provides a new method of recognising everyday objects for whiteboard interaction, as it is capable of detect uninstrumented and electrical objects without extra equipment. In [25], researchers demonstrated that multiple mundane objects could be recognised through electromechanical signature, and a separate graphical user interface could be displayed on the host device. This technique could be further extended to recognise everyday objects that emit small amounts of unique electromagnetic signals, and configurations and interactions could be done through the host device. This provides the technical foundation of cross-objects user interface that extends whiteboard interaction while keeping high generality across the objects.

3 Method

Given the preceding understanding of whiteboard interaction and requirements for user interface that aims to enhance apparatus generality and meanwhile lower the development cost, we referred to the electromagnetic signal recognition technique and developed a cross-objects user interface for whiteboard interaction. In addition, we conducted a within-subject evaluation to investigate the effectiveness of this user interface. Below we describe methodological details.

3.1 Implement Cross-Objects User Interface

We prototyped a low-fidelity cross-objects user interface that consisted of three separate cubes disassemblied from a 3 * 3 Lego Rubik’s cube by referring to the technique in [25] (Fig. 1). The cubes’ electromagnetic signal patterns were manually extracted and trained beforehand. Thus, when the cubes were touched by user’s hand, the mobile phone in the other hand received electromagnetic signals through user’s body as conductive antenna, recognised the cubes on touch, and launched configuration interfaces in mobile phone. By changing the settings displayed in the mobile phone, the users configured functionalities of the cubes in whiteboard interactions.

Fig. 1.
figure 1

Disassembled cubes as the cross-objects user interface

Full size image

The cubes attached magnetic pads to stick on and move around the whiteboard surface. The whiteboard was a conventional board at sizes of 900 * 1800 mm. A projector was setup in front of the whiteboard to display contents during the interaction. In addition, a webcam was attached on the top surface of the projector to recognise positions and numbers of cubes, so the contents were accurately displayed around the cubes. Both the projector and webcam were connected to a laptop. The mobile phone ran an application that stored the cubes’ configurations and communicated with the laptop through Bluetooth. A full map of study apparatus setup is illustrated in Fig. 2.

Fig. 2.
figure 2

Study apparatus setup

Full size image

3.2 Evaluation of Cross-Objects User Interface

To understand effectiveness of the user interface in whiteboard interaction, we conducted within-subject comparable evaluation. We recruited 15 undergraduate students. None of them had any previous experiences of cross-objects user interface, and none self-reported any body movement impairments. The participants were required to trial the cross-objects user interface and complete three whiteboard-related tasks: namely, sketching, erasing, and file transferring between the cubes. These represented the mostly frequent whiteboard interaction tasks [12]. Procedural flows of the evaluation procedures were as follows.

  1. 1.

    The participants received a 5-min introduction of the user interface given by experimenter, and then were given a 2-min trial of the cubes on whiteboard with mobile phone.

  2. 2.

    On the end of pre-study practice session, the participants were required to make open sketches on the whiteboard by firstly configuring the cubes as a drawing marker. The configuration process was done through the mobile phone. The sketches were an open task to circumvent unnecessary pressures on participants.

  3. 3.

    The participants were then asked to reconfigure the cubes to erase the sketches drawn earlier.

  4. 4.

    The participants were asked to reconfigure the cubes, displayed the contents stored in the cubes, and transferred these contents from one cube to another. This was an open task which aimed to encourage participants’ exploration of the use of user interface.

  5. 5.

    After accomplishing all required tasks, the participants fulfilled 5-Likert questionnaires which contained 9 effectiveness-related questions and 1 overall satisfaction-realted question (see questions in Appendix 1). The questions asked participants to compare with conventional whiteboard interaction (including normal markers and plain whiteboard) and then to give ratings.

  6. 6.

    Following the questionnaires, informal interviews were hosted by experimenters and the feedbacks were logged.

The evaluation of effectiveness of the cross-objects user interface adopted three criteria derived from [26]. The criteria were used in evaluation of various interactive products such as intelligent tutoring systems with proven reliability and validity, particularly the criteria were suitable to reflect effectiveness of conceptual products that were not massively applied [27]. The criteria consisted of:

  1. 1.

    timeliness which measures how quickly a system is able to provide the user with the outputs (e.g. sketching and erasing in this case) they require; this reflects how the new user interface can help participants accomplish the given tasks;

  2. 2.

    throughput which measures how much work is done by the system over a period of time; this reflects how much work the new user interface can afford to be a productive tool in critical whiteboard interaction tasks; and

  3. 3.

    utilisation which measures the proportion of time a system resource is busy; this reflects how much resources of the new user interface are available to support participants’ interactions.

The evaluation collected 15 questionnaires and interview logs, respectively. Each question in the questionnaire required the participants to compare with conventional whiteboard interaction and to give two respective ratings, one for the conventional whiteboard interaction and the other for the cross-objects user interface. The conventional whiteboard interaction was chosen as the comparison benchmark of evaluation, because the results were more intuitive when comparing the cross-objects user interface with the natural physical-world objects. All questionnaire results were transcribed into databases for later analysis, and the interview logs were examined by experimenters to annotate the key phrases.

Statistical analysis was conducted to compare the differences in the aspects of effectiveness between the conventional whiteboard interaction and the cross-objects user interface whiteboard interaction. The results are shown in Fig. 3.

Fig. 3.
figure 3

Results of effectiveness of the cross-objects user interface

Full size image

As Fig. 3 shows, the analysis reported significant differences in the results of question 5, 8, and 9 (Independent-Sample T-test, p = 0.03, 0.046, and 0.049, respectively). The rest questions did not report any significant differences between the interaction with conventional markers and whiteboard and the cross-objects user interface.

The overall reliability and validity of evaluation were great, as the results of question 10 showed, all participants successfully completed required tasks and rated equal overall satisfaction levels. The results of question 5 confirmed the participants’ expectation of interactive functionalities embedded in whiteboard. The results of question 8 showed that the participants’ willingness to deploy the cross-objects user interface was affirmative, and the results of question 9 showed that the participants were happy to extend the cross-objects user interface in broader applications. The results of other questions showed some advantages of conventional whiteboard interaction over the cross-objects user interface interaction. For example, question 1 (measuring the timeliness of learning) indicated that the conventional markers and whiteboard were still easier and quicker to learn, which was validated by the results of question 6 and interview feedbacks.

As the results of question 3 indicated, the participants might not be the same confident as in conventional whiteboard interaction when facilitating with the new user interface. The results of question 4 and 7 showed some advantages of the cross-objects user interface, although no significant differences were reported. Importantly, the results of question 2 (measuring the timeliness of naturalness) reported equal naturalness of the cross-objects user interface as conventional markers and erasers, which confirmed that the physical cubes and related metaphors of the cross-objects user interface were understandable to the participants. This was validated by interview results, as the participants commented “it (the cubes) feels easy to understand, it’s also quite intuitive”.

4 Results

The preceding evaluation findings confirmed the overall effectiveness of the cross-objects user interface for whiteboard interaction, as the interface achieved equal results compared with conventional whiteboard interaction, particularly in several aspects of effectiveness the cross-objects user interface gained greater advantages. Taking all evidences together, we claimed that the cross-objects user interface was effective and it had great potentials to support various whiteboard applications with lightweight configurations.

5 Discussion

The intention of this research was not to deliver a one-for-all solution to the generality problem in whiteboard interaction, rather it was motivated to explore new avenues that could possibly incorporate physical-world objects to implement cross-objects user interface to support multiple whiteboard interaction tasks with less development work. Admittedly, many studies have proposed different systems that were aimed at maximising the use of apparatus across whiteboards. This study did not make significant differences in this regard.

However, the study stepped further towards the coupling of physical and virtual world and the coupling of everyday objects and digital information by adopting electromagnetic technique. Consequently, the study proposed the cross-objects user interface that was almost same natural as the conventional markers and whiteboard, and importantly the evaluation study showed that the new user interface was more configurable and adaptable across multiple whiteboard interaction tasks. Therefore, the main contributions of this study lay on two points, one is the implementation of the cross-objects user interface for whiteboard interaction, and the other is the understanding of the effectiveness of the cross-objects user interface. The latter understanding supplies generalising implications for future tangible user interface design and augmented reality user interface research.

The findings drawn from this evaluation are generally aligned with the conclusions of previous studies. For example, tangible objects as manipulation tools are naturally intuitive [28] and tangible bits take advantage of users’ cognition habits that have been accumulated through biological evolvement [16]. Also, new understandings were added. For example, despite users learned the cross-objects user interface quickly, they appeared less confident in interaction.

Regardless of evaluation findings, the design of cross-objects user interface as well as its trials in the study give broader implications for user interface design in ubiquitous computing era. The form of using physical objects in whiteboard interaction is not new, as this has been done in previous projects such as [9, 13,14,15,16]. However, these objects were still bound to specific colours, shapes, and materials. Given the electromagnetic technique, researchers are now easy of introducing interaction metaphors from physical world, and they can configure the apparatus with minimum development efforts. Also, the cross-objects user interface realised foreground and background interaction environments, which focused on the interactive objects and the peripherals, respectively [29]. The peripherals are often neglected in conventional HCI research, but the cross-objects user interface helps users coordinate foreground and background interaction simultaneously.

In addition, the electromagnetic technique-based mundane object recognition can be further extended to a wider range of applications. In [25], the researchers have attempted to apply this technique in different everyday objects such as door handle, motor-bike, and toaster. Initially this technique was proposed to recognise electrical and electromechanical objects by identifying objects’ electromagnetic signal patterns (also called ‘signature’), and then enabled to infer object states. However, when this technique is able to accurately recognise objects’ electromagnetic signatures, it could be extensively used as a connector between the objects and ubiquitous networks. As partially demonstrated in this study, the cubes could be assigned links that point to remote digital files assisted by the host device, which could be a wearable smart watch or any other digital artefacts. Therefore, not exaggeratingly, the cubes could be containing the whole internet contents when it is recognised with a unique identification and working links to contents. The cross-objects user interface also shed some lights in envisioning future user interface design, for example, an invisible user interface that supports user grabbing a box of milk in store and complete payment by touching the wallet.

Due to the early stage prototype development and preliminary exploratory study, the cross-objects user interface and the evaluation are noted with some limits. Firstly, low-fidelity prototype of cross-objects user interface was used in study. The interface was functional and fulfilled expected study requirements, however, to some extent it affected the participants’ overall satisfaction. Secondly, the evaluation used a relatively small number of participants. We do not deny that a larger sample would be adding more credits to the study, but we also believe that the evaluation study that strictly followed methodological procedures provided a reliable and valid ground of understanding. Finally, the electromagnetic technique adopted in this study still has some limits due to its early stage development, these included the negative influences of object sizes and locations. The technical limits will be especially pushed forward in future work.

6 Conclusions

We have proposed a cross-objects user interface for whiteboard interaction, and evaluated effectiveness of the interface. Compared with conventional whiteboard interaction, we analysed how the cross-objects user interface performed in the perspectives of timeliness, throughput, and utilisation. Our study showed that the new interface was as natural, responsive, and easy to learn as conventional whiteboard interaction, and in some aspects (e.g. versatility) the cross-objects user interface outperformed over the conventional one. Given these findings, implications for future tangible user interface design for whiteboard interaction as well as for broader ubiquitous computing-related application design are discussed.