1 Introduction

Nowadays, effective collective decision-making and knowledge sharing are necessary for group collaboration. The traditional collaboration devices, sand-table simulation using tokens on a customized interface, have been digitalized and facilitated by a multi-user multimedia interactive system with a shared digital medium (e.g., tabletop). The system is compatible with diversified collaborative tasks and supports information synchronization and an individual’s need for information input and inquiry. The system is composed of a large horizontal interactive tabletop displaying a shared digital overview, physical cursors on the tabletop, and a personal interface on each individual’s mobile device (Fig. 1). The system could support both shared context display and personal operation by efficient information synchronization. A movable physical cursor is paired with a mobile device, and multiple cursors could work simultaneously to facilitate each user’s operations. Users adopt cursors to select objects on the tabletop and acknowledge other members’ cursor activities. Two types of pointing techniques, direct pointing and proximity pointing, are developed to realize the selection and further operations.

Fig. 1
figure 1

Direct and proximity pointing techniques applied in a collective decision-making task. Note: Users can pair their smartphones with physical cursors (colored blue, grey, and rose in Fig. 1). The smartphone serves individuals, and the shared tabletop enhances information synchronization and shared interface. Details of the shadow area on the shared interface are shown in the circles above. The gray square marked “1” refers to a cursor. The top-left shows direct pointing that provides a single option for confirmation, and the top-right demonstrates proximity pointing that displays four potential choices for selection and confirmation in the second step

Full size image

Direct pointing follows the traditional mouse-based interaction but replaces digital cursors as the physical ones and mouse as mobile devices. As the Browse-Move-Click model (e.g., operating a mouse), a user selects a target object by identifying the target, moving a digital cursor to select it, and clicks the mouse to confirm the selection. Following the same model, a physical cursor is located on top of the target object, and requests confirmation by clicking the single option “confirm” on a smartphone. However, a physical cursor may fail to locate the target object precisely due to compact items and occlusions by physical cursors. Proximity pointing sets a two-step procedure to tolerate location inaccuracy. It first narrows down the options, shows four potential targets close to the cursors on a smartphone, and then requires confirmation among them.

The pros and cons of both pointing techniques under different conditions remain unknown, resulting in arbitrary designs. Therefore, we empirically evaluate the usability of the pointing techniques at an individual level. We examine how the pointing techniques influence users’ speed and accuracy in completing different tasks on interfaces of different information densities and reveal the foundation for group-level collaboration supported by an interactive multimedia system.

The remainder of the paper is organized as follows. Section 2 provides a motivating example. Section 3 reviews related work. Section 4 explains the design rationality. Section 5 proposes hypotheses and illustrates the experiment design. Section 6 presents the experiment design. Section 7 presents the experiment results. Section 8 analyzed the results. Section 9 discusses practice implications, followed by future work and conclusions.

2 A motivating example

The increasingly tight collaboration in work requires full technical support to synchronize dynamic information within group members. Although many shelter places are established already in some disaster relief plan, overview of shelters and the related-information is extremely important to help officers to make quicker responses with unknown factors and uncertainty, especially dealing with a public emergency. For example, three government officers from different departments have to coordinate for a public emergency. They are planning temporary shelters to cope with an approaching hurricane. The evacuation and disaster settlement decisions at the onset of crisis require fast and accurate sharing of multiple-resourced information including status of shelter occupancy and surrounding traffic condition. Each officer constantly inquiries and provides the department-owned confidential information to coordinate others’ decisions while tracking the dynamics of the collective overview. As information system has increasingly well-documented and integrated the data from different departments, the officers are enabled to use a physical cursor to authorize the information accessible through the tabletop-centric display and process the retrieved detail information on their own mobile devices, and transparently reveal their focus on the overall picture to other group members. Physical cursors balance the tension between sharing transparency and confidentiality. Using the cursor paired with a mobile phone, the officer would select a public library as a potential shelter to find the essential supply(e.g., electricity, water and food).

The direct pointing allows the officer to place his cursor on the library and confirm the selection by clicking the “select” button on the smartphone (Figs. 2 and 3). The cursor is easy to be located on the target and gets the users free from the time-consuming and error-prone typing of the location. Meanwhile, it synchronizes what officers focus on the overview picture of the disaster-prone area with all members.

Fig. 2
figure 2

Direct pointing on tabletop

Full size image
Fig. 3
figure 3

Direct pointing on mobile device

Full size image

The officer places his cursor on Princeton Library in New York on the tabletop (See Fig. 2 direct pointing on the tabletop), and the phone displays the library name for selection confirmation (Fig. 3a). After confirmation, the detailed information of this library shows (Fig. 3b). To check about Yale Club library, the officer switches his attention back on the tabletop, moves the cursor, confirms on his mobile phone (Fig. 3c) and check for details (Fig. 3d).

Direct pointing is easy to understand but has obvious shortcomings. First, a physical cursor of 1.125 in. square inevitably occludes critical information of a target object. Novice users may have to check whether the occluded information is displayed precisely on mobile by first lifting up and then putting down the cursor. Second, it is difficult to precisely spot a target object among dense surroundings or/and when multiple cursors are located nearby. For instance, in Fig. 2, the officer may have to adjust the location of the physical cursor multiple times to occlude the Princeton library accurately for selection. In such a situation, proximity pointing could be a solution.

When proximity pointing is adopted, the mobile phone shows four potential targets closed to the physical cursor as choices for further confirmation. For example, the officer places his cursor among multiple libraries (Fig. 4. A black cursor on the map), and his smartphone renders four options adjacent to the cursor (Fig. 5a) to facilitate the selection among the four. The details of the selected library would show after confirmation (Fig. 5b). The officer reviews other libraries by clicking the back button and confirming another option (Fig. 5c) which reduces the distance to move the physical cursor on the tabletop. Proximity pointing tolerates the ambiguity in locating the cursors without compromising by changing the area/scale of shared information space [20] on tabletop and allows multiple cursors located within a small area.

Fig. 4
figure 4

Proximity pointing on tabletop

Full size image
Fig. 5
figure 5

Proximity pointing on mobile device

Full size image

Using pointing techniques and operating on the tabletop, the officers may have to select multiple objects, count available resources, and write about notifications. When the officer selects multiple objects, a successive target object could be adjacent to or isolated from its preceding selected one. The selection of two adjacent objects was nominated as cluster selection mode; otherwise, isolated selection mode. Selection mode reveals the (dis)advantages of the pointing techniques. To discuss the allocation of relief goods, the officer may have to count the number of available shelters and take notes to remind other officers about exceptions. Overall, pointing techniques as the fundamental technique of the interactive system should be systematically explored in terms of usability when specific selection mode and for different tasks.

3 Related work

The shared digital medium, such as a tabletop or a digital whiteboard, is widely utilized for diversified collaborative tasks[5, 24]. Furthermore, increasing shared digital mediums are used for investigating the hybrid/virtual environment with diversified tasks. Many frameworks [12, 25] are developed to facilitate the development of tabletop-centric applications in the hybrid environment with specific physical awareness tools [13, 17] which allow users to be aware of current activities and contextual information. Although using awareness tools can significantly improve collaborative task performance[29], the compromise between multi-user transparent information representation and individual user’s privacy [38] remains an issue. Tabletop-centric cross-device interaction[37] is introduced as a solution.

The completion time and accuracy of a given task are important parameters for evaluating the effectiveness as a criterion of usability. In single device interaction, Fitts’s law [11] uses moving time to measure the difficulty in pointing and selecting a target and set up the criteria for techniques innovation. Indirect-control pointing devices (e.g., a mouse and a joystick) are created and optimized. There are many techniques proposed to improve the completion time by reducing the moving distance. For example, Pie menus [19] reduced the moving distance from each selection options compared dropdown list to improve the average selection time. Drag-and-Pop and Drag-and-Pick [6] automatically moved potential targets to the cursor based on the cursor moving direction, thus reducing the moving distance. On the other hand, some other techniques are developed to increase the size of targets to enhance the accuracy. Radial menus [40] which is another version of Pie Menu enhance the selection accuracy by increasing the target size by implementing a direction-based selection. The bubble cursor [15] dynamically increased the size of an object when the mouse moves toward it. The increasing popularity of the touch-screen drives the technique innovation based on a direct-control pointing device (e.g., a touch-based screen). However, the new technique brings other problems, such as the occlusion issue (i.e., a user cannot see the content beneath the finger) and the fat finger issue (i.e., the small target content is too small to point by a finger). TapTap [36] used a double-tap gesture to enlarge the target and facilitate the selection using fingers to solve the problems. Offset cursor [32] allowed a user to select a target by moving and placing his/her fingertip in proximity to the target object and thus minimized the occlusion. Shift [41] displayed a small window by the user’s finger to reveal the content beneath the finger. Single-device-based pointing techniques are applied to cross-device interaction. However, its usability is rarely documented nor well explored.

The pointing technique in our research is extended for the classical interaction paradigms of a large display with small devices for display and operation. Past techniques substitute a mouse that remotely points and selects objects on a large display by mobile devices. The touch screen of a mobile device tracks finger movements to support remotely selecting [27, 28]. Contactless pointing techniques used a built-in camera to read visual codes and select objects in a distant display (known as “Point & Shoot” and “Sweep” technique; [1, 3]). Contact pointing techniques are also developed. It either placed a mobile device on top of a tabletop as a magic lens displaying the contents beneath the mobile device [21, 30] or used a mobile device as a stylus targeting an object on a large display [18, 37].

Physical cursors were introduced into cross-device interaction and drives the development of two-handed interaction [34, 35]. One hand moved a physical cursor and pointed objects on the tabletop, and the other handheld a mobile device to confirm the selection, which causes fewer errors than single-handed interaction [21], and boosts the performance when both-hand-collaboration are required [23]. Thus, our cross-device interaction highlights the physical-cursor-based two-handed operation.

The pointing action is realized based on proxemics interaction. Proxemics interaction explores fine-grained proxemics relations (e.g., distance, orientation, movement, identity, and location) to support implicit and explicit actions in cross-device interaction [4, 14, 26]. A serial of studies provides solutions to proxemics interaction, utilizes an external camera to capture visual markers that indicate the locations of users’ devices on a flat surface [33]. detect the pointing direction between two devices through acoustic signals[31], locate the relative positions of adjacent devices [22], employ the orientation of mobile devices or other objects as a reference frame to deploy the position of mobile devices.[7, 16] However, how the hardware-based technique is perceived and experienced needs further exploration.

4 Design rationality

We introduce a physical cursor (See Fig. 6) placed on a tabletop in an interactive multimedia system. The cursor is identified by its unique visual marker and assigned to its user, enabling multiple-user cross-devices interaction. The cursor’s identity recognition and position tracking are implemented on the hardware of Microsoft SUR40. The tabletop could respond to the cursor’s real-time location and display the requested information on the user’s mobile device.

Fig. 6
figure 6

A physical cursor with a unique visual marker

Full size image

We implemented two-handed interactions with the physical cursor. One hand locates a physical cursor on the tabletop, and the other hand holds a mobile device to confirm the selection, which supports navigation and selection better than one-handed interactions [8]. When the direct pointing is adopted, the smartphone shows the target object’s name beneath the cursor for confirmation. When proximity pointing is adopted, the smartphone renders a two-by-two grid, and each cell displays potential targets’ names closest to the cursor (See Fig. 5a) according to the cardinal direction. The potential targets would be displayed in a layout similar to how they are located on the tabletop display. If the cursor accurately occludes a target object, the object would be rendered on the smartphone as the single option and selected. If the target objects are located among a cluster, four potential options could be rendered simultaneously and selected at once at maximum.

We manipulate the density of displayed items, among which the cursor is operated. Many objects and intricacy details in a given unit of space shape a dense (vs. sparse) layout, perceived as compact and complex, and increases the difficulty in accurately locating the cursor on the target object.

We designed two types of selection tasks to detect how users optimize the moving time using pointing techniques. The first type of the selection task only highlights a single object for selection at a time; the highlight remains until the object is selected through the mobile, a user has no other vision to anticipate the next movements. The second type of the selection task highlights multiple targets simultaneously so that a user can overview in advance and may minimize the moving route by anticipating the next movement. We nominate them as a single-object display and a multi-object display perspectively. We also simulate reading exercises to assess the user performance in the count and write task. Especially the user’s performance would be evaluated by writing down the displayed item on the paper after the selection which also tests user’s ability of global and local information processing in this task. The details of tasks would be explained in the Section 6 Experiment.

Overall, we designed and tested the time and accuracy of the two types of pointing techniques when they performed in the selection task with varied display density and count and write tasks.

5 Hypothesis

To adapt to a cross-device interaction, users have to switch their attention between a tabletop and a mobile device, which conflicts with their habituated schema of single-device interaction. They may perceive a strong sense of uncertainty and are motivated to verify the relationship between the content displayed on two devices. In a verification process, direct pointing causes occlusion and impedes learning, adaption, and uses. Specifically, users can not help repeating to lift the cursor, peek at the content, and put down the cursor, such that confirming the occluded objects on the tabletop is consistent with the information displayed on the mobile. The verification becomes even frequent when users process unfamiliar content with high cognitive load. By contrast, proximity pointing avoids occlusion and supports an instant review of the target object. Users do not have to memorize the content beneath the physical cursor. Therefore, they have more cognitive resources to improve operation accuracy.

  • Hypothesis 1a: Proximity pointing causes less time to select an object than direct pointing.

  • Hypothesis 1b: Proximity pointing causes less selection error than direct pointing encountering increasing complexity of tasks.

Given that a set of targets need to be selected, users may shorten the moving distance by optimizing the sequence of selection actions. In proximity pointing, a mobile device displays four objects adjacent to the physical cursor. This feature benefits selecting adjacent objects. One cursor movement may facilitate the selection of four targets at most. In an ideal situation (as Fig. 7 presents), proximity pointing allows a user to select at most four adjacent objects (highlighted in red) with one single movement of the cursor. Especially if a user happens to move the physical cursor to location T1 (See Fig. 8), the total movements are reduced to one. In a less ideal situation, locations T3, T5, T6, and T8 allow a user to select two targets with one single movement (See Fig. 8). In contrast, using direct pointing, a user must move the physical cursor four times to select four target objects. Having been enabled to select multiple objects, users may locate the cursor among objects, actively optimize total moving distance, and improve operational efficiency. In other words, the advantage of proximity pointing is strengthened by cluster selection. Therefore,

  • Hypothesis 2: The advantage of proximity pointing in selecting an object is strengthened by cluster selection.

Fig. 7
figure 7

In cluster selection, there are nine possible movements (i.e., T1 to T9) to select a target on the large display. Red symbols are targets

Full size image
Fig. 8
figure 8

Different displays on the mobile device based on the location of a physical cursor in Fig. 8

Full size image

When processing a vast amount of information in a large space, a user has to narrow down their attention and focus on local information each time. The attention switch between the tabletop and mobile display causes interruption. Users would be forced to re-target the local information of interest on the tabletop. The direct pointing technique enables the cursor to directly anchor the target object for selection, and increases the inclination to use the cursor to mark the content. Using the cursor as a visual anchor, the users perceived it easier to focus on their target and process the information more accurately in a dense layout. Therefore,

  • Hypothesis 3: Cursors used as anchors mitigate the disadvantage of directing pointing on the performance to process information that is displayed in a dense layout.

6 Experiment

6.1 Experiment design

  1. 1)

    Overview

This study investigates how the relative spatial location of a physical cursor on a tabletop affects the usability of tabletop-centric cross-device interaction. The experiment is a mixed factorial design, with pointing technique (i.e., direct pointing and proximity pointing) as between-subject factors and selection mode (i.e., cluster selection vs. isolated selection) as within-subject factors (See Table 1). We manipulate the possibility that a user adopts cluster selection by displaying either single or multiple objects for selection. We assume that users may use the continuous selection as an optimized way and are likely to conduct cluster selection when multiple objects for selection are given.

  1. 2)

    Tasks

Table 1 Independent and dependent variables
Full size table

The study includes three types of tasks (i.e., selection, read-and-count, and read-and-write). The selection task tests the fundamental differences in pointing and selecting objects between two pointing techniques. Read-and-count and read-and-write tasks test users’ reading performance when processing a large amount and a small amount of information after selecting an object, respectively. To minimize the effect of content familiarity, we collected the symbols and English words as stimuli, with which participants are not familiar. To keep participants engaged, we addressed the significance of the research and encouraged participants to complete the tasks correctly. Those tasks are described in detail as follows.

Selection task

In the selection task, participants were asked to point and select a highlighted target object from 24 symbols that are distributed in a 6 × 4 grid on the tabletop.

Reading performance tasks

The reading performance tasks test how two pointing techniques affect individual’s target/key information processing at a cognitive load level. When making a serial of judgments and decisions on reading process, individuals have to effectively read, and perform visual-based analysis, such as tracking key objects, and timely update the consensus with other collaborators. The physical cursor, on the one hand may help the visual object tracking and locating, and on the other hand may occlude the visual content and slow down the reading. Therefore, we design the basic tasks in collaboration to evaluate the role of the physical cursor and the two pointing techniques. In the tasks, we ask the participants to perform fundamental ways (i.e., search and write) to retrieve two kinds of elemental information (i.e., symbol and alphabetical words) in reading. Searching is a fundamental way to help the reader with tracking and searching the key information in the article and papers. E.g., readers would search the keyword in the article to navigate the relevant paragraph. Since counting the key information could emphasize the process of searching, in the task, the participant is asked to count the specific information. Meanwhile, taking notes is common when people learn the essential knowledge for preparation and review. However, taking notes is inappropriate on the shared display. Therefore we request the participants to write down the alphabetical words on another white paper. These two tasks would be described in the following:

  • Read-and-count Task. The tabletop is divided as a 6 × 4 grid, where each cell is represented with a unique index rendered at its center. Each object’s index, rather than its content, is displayed on a mobile device for confirming a selection since the content is too large to be displayed. After successfully selecting a designated object, the participant is asked to count the number of a specific symbol in the target, which has 36 symbols distributed in a 6 × 6 grid. The size of each symbol is similar to the size of the physical cursor and whose majority parts would be occluded.

  • Read-and-write Task. The read-and-write task adopts the same layout as the read-and-count task. We design to display an index rather than a word on a mobile device for confirmation, such that a participant won’t retrieve a word from the smartphone. Participants are asked to read and memorize a 7-letter string displayed on the tabletop in their selected object’s center and write it down on a piece of paper. To exclude the influences from processing fluency caused by existing words, we made up the string stochastically from 26 English letters without duplication. The physical width of the word is similar to the width of the physical cursor.

  1. 3)

    Apparatus

Our study adopts a Samsung SUR40 as a tabletop and a Google Nexus 4 as a personal smartphone. The Samsung SUR40 has a 40“ multi-touch LCD screen with a built-in IR camera that can recognize 50 simultaneous touchpoints through the Microsoft PixelSense technology. The Google Nexus 4 has a 4.7” touch screen and supplements the tabletop in the cross-device interaction.

6.2 Participants

Sixty-two participants (50 males and 12 females) were recruited by email at a mid-west university in the United States. All participants are currently using smartphones. Twenty-six participants own an Android phone, and thirty-four own an iPhone, one owns a Windows Phone, and one owns other types of smartphones. We survey the time each participant spent on a cell phone each week. On average, 17 participants spend over 18 h, 23 participants 12–17 h, 18 participants 6–11 h, and only 4 participants less than 5 h per week. We also gauge the familiarity with a multi-touch tabletop using a 5-point Likert scale (1 = unfamiliar and 5 = familiar). 48 of 62 participants reported that they are unfamiliar (3 and below) with a multi-touch tabletop.

  1. 4)

    Procedure

The experiment consists of a pre-test questionnaire, a training section, a test, and a post-test survey. The entire experiment took about 30 min on average.

  1. Step 1.

    Pre-test questionnaire. Participants were asked to report their gender and experience of using smartphones and tabletops, including the type of their smartphones, the frequency of smartphone use, and the experience on the multi-touch tabletop.

  2. Step 2.

    Training. All the participants were randomly divided into two groups, and each group used one pointing technique. Before hands-on, the researcher explained the features of the corresponding pointing technique and demonstrated how the pointing technique worked. Then, participants were asked to practice the pointing technique. We prepared four training scenarios that cover different selection and display modes.

  3. Step 3.

    Test. The experiment adopts mix factorial design, in which the pointing technique is a between-subject factor, the task categories, display mode and selection mode are with-in subject factors. Each participant was asked to complete 12 trials (See Table 2), i.e., four trials for each type of task. Those trials were randomly arranged to avoid the learning effect and the influence of the sequences of task types. Participants were asked to press a button on the tabletop to start the test when they were ready. The system automatically recorded the completion time of each trial.

Table 2 12 different trials: trails would be randomized on each participant
Full size table

To prevent users from reading the information before tasks, the tabletop first only displays the index of each object in the read-and-count and read-and-write tasks. The content of each object is rendered on the tabletop only after a user successfully selects this object.

  1. Step 4.

    A Post-test survey. After 12 trials, participants were asked to fill out an online questionnaire regarding their subjective feedback. The questionnaire was designed based on the ISO 9241 Ergonomic design of visual display terminals used for office work requirements for non-keyboard input devices [10]. It was used to evaluate the usability of cross-device interaction [2]. The questionnaire inquired mental effort, physical effort, finger, and arm fatigue, perceived difficulty, and perceived joy when using a pointing technique. Each question was rated based on a 5-point Likert scale (1 = very low, and 5 = very high). After filling out the questionnaire, participants were asked to provide open comments on the pointing technique they were using. The accuracy and operation speed were automatically recorded by our system.

7 Result

In total, we successfully collected 744 trails data from 62 participants. Each participant in total spent 11 min 33 s (with SD = 2 min 32 s) to complete the 12 trails. The total completion time ranges from 7 min 47 s to 18 min 2 s. On average, the completion time of each task, the total number of errors, and the perceived confidence of each trail in sequence are shown as the Fig. 9.

Fig. 9
figure 9

The average completion time, total errors, and perceived confidence of each task in sequence

Full size image

7.1 Completion time

Table 3 presents the completion time of three types of tasks. The result indicates that proximity pointing in general causes less time than direct pointing, as H1a predicted.

Table 3 Main effects under three tasks
Full size table

Table 4 demonstrates the error of each task. In the task 2 and task 3 (p < 0.05), it shows direct pointing causes significantly more selection errors than proximity pointing (task 2:Mproximity = 0.313, SDproximity = 0.903, Mdirect = 0.825, SDdirect = 1.776, p = 0.004; task 3: Mproximity = 0.234, SDproximity = 0.918, Mdirect = 0.542, SDdirect = 1.289, p = 0.031). Overall, All the results indicate proximity pointing significantly causes less selection errors than direct pointing Mproximity = 0.224, SDproximity = 0.918, Mdirect = 0.561, SDdirect = 1.429, p < 0.001). The conclusion supports H1b. Task 1 is simpler than task 2, and 3. The results show that direct pointing causes more selection errors than proximity pointing when tasks are complex and require more cognitive resources (p < .05), but not when the tasks are simple (p = .139).

Table 4 Results of selection error
Full size table

We analyzed the learning effect using a generalized linear mixed model. We tested the fixed effect of the trial sequences, controlling the random effect of individual difference, the fixed effect of task categories, display mode and selection mode. The results reveal a significant influence of the trial sequences on completion time (p = .000). The estimated completion time on each task in sequences decreases from 68 s when the first task to approximate 55 s when the last two tasks. The interactive effect of pointing technique and task sequence was statistically insignificant (p = 0.532), i.e., the two pointing techniques don’t show significant difference in the learning effects. We did not capture significant learning effect of trial sequences on errors or confidence.

We also performed a 2 (pointing technique: direct pointing vs. proximity pointing) × 2 (selection mode: cluster selection vs. isolated selection) analysis of variance for three tasks, with the completion time as the dependent variable. The analysis revealed an interactive effect of pointing technique and selection mode on the completion time of both selection tasks and read-and-count tasks, but no interaction in read-and-write tasks. The interaction effect (See Fig. 10, p < .05) in the selection task suggests that the advantages of proximity pointing are strengthened by cluster selection and its consequences, including a short moving distance and few operations in verification, as predicted in H2. Specifically, when cluster selection, users of proximity pointing performed faster than direct pointing (Mproximity = 10′918, SDproximity = 5′699, Mdirect = 13′904, SDdirect = 4′331, p = 0.001). When isolated selection, users of two pointing techniques do not perform significantly differently in completion time (Mproximity = 17′052, SDproximity = 6′313, Mdirect = 17′498, SDdirect = 7′502, p = 0.72).

Fig. 10
figure 10

Completion time in the selection tasks

Full size image

The interactive effect (See Fig. 11, p < .05) of pointing technique and selection mode in the read-and-count task is consistent with that in the selection task, i.e., the advantage of proximity pointing (vs. direct pointing) in completion time is enhanced in the context of cluster selection. Particularly, in the context of isolated selection, users of proximity pointing spend significantly less completion time than using direct pointing (Mproximity = 70′951, SDproximity = 22′915, Mdirect = 86′904, SDdirect = 21′173, p < 0.001). By contrast, in the context of cluster selection, users using two pointing techniques are not statistically different in their completion time (Mproximity = 81′916, SDproximity = 25′308, Mdirect = 83′005, SDdirect = 20′305, p = 0.793). The conclusion support H3.

Fig. 11
figure 11

Completion time in the read-and-count tasks

Full size image

In the experiment design, we expected that the benefit of proximity pointing could be strengthened in multi-object display mode since proximity pointing enables us to select at most four targets that surround a physical cursor and motivates users to optimize the overall moving route of total operations. As a result, we did not find a significant interactive effect of the pointing technique and display mode on selection time (p > .1). As we observed, participants were not aware of the potential optimization of cluster selection. Instead, they persist in repeating moving the cursor and select a single object each time, nullifying the expected interactive effect of the display mode and the pointing technique.

7.2 Subjective feedback

Table 5 shows the average rating of each question in the post-test questionnaire. Mann Whitney U test reveals that proximity pointing is perceived significantly easier than that in direct pointing, while there is no significant difference in other questions. Also, the open comments suggest that the occlusion of a physical cursor in the direct pointing significantly affects the readability.

Table 5 Results of post-questionnaire
Full size table

8 Discussion

The pointing technique is the most fundamental operation that supports diversified functions in group collaboration. We explore how users habituated to single-device interaction using a mouse learn and adapt to multiple device interactions. This research compared proximity pointing and direct pointing in different contexts, including task types, selection mode, and display mode. Regarding the completion time, the empirical evidence supports the arguments on the advantages of proximity pointing in completion time and rate of correctness (H1a and H1b). Furthermore, the advantages of proximity pointing in completion time are strengthened by cluster selection on selection tasks (H2). Direct pointing causes occlusion and results to its disadvantage. However, as a trade-off, it can be used to help the users focus on the content and facilitate reading. The result suggests direct pointing improves the reading of the content among a large amount of information in a dense layout (H3).

First, our conclusions contribute to the literature on tabletop-centric cross-device interaction by indicating the advantage of the proximity pointing technique. Besides few studies of proximity pointing technique in single-device interaction (e.g., offset cursor), the direct pointing technique has dominated the mainstream of designs in both single-device interaction and cross-device interaction. The use of proximity pointing in cross-device interaction is overlooked and remains unexamined. Our findings indicate the advantage of proximity pointing in cross-device interaction and highlight the significant role of proximity pointing in cross-device interaction. With the increasing demand from collaboration and the need for information synchronic, our findings provide a fundamental underpinning for the future development of the multi-device interaction system. The empirical evidence from individual users would hold when a group of uses operates using the system since the disadvantage of direct pointing would be amplified by the increasing occlusions caused by peer users.

Second, our research sheds light on the understanding of information redundancy [9, 39] in arranging the proxemics relations between physical cursors and digital objects. The advantage of proximity is supported by the redundancy of information display and the two-step information release design. The lack of information redundancy in the direct pointing motivates users to repeatedly lift up the cursor and put it down to check about the occluded content. Comparing memorize a selected object, users would like to see the object immediately for updated information. Direct pointing, with inevitable occlusion issues, disallows the quick power of “seeing.” To summarize, the comparison confirms the necessity of information redundancy in cross-devices interaction.

Furthermore, to coordinate the proximity pointing, we separate the confirmation into two steps. The redundancy information released in two-step is proved effectively improve usability. Users figure out the selection among pre-selected four options in a 2 × 2 grid on a smartphone and then click as confirmation, rather than confirms the only option. One might argue that four options are distracting, making it difficult to recognize and distinguish the target. Apart from the visual appearance cues, users could confirm the selection based on the relative location of the target to the physical cursor on a tabletop. The relative position as a prominent cue reduces the perceived difficulty, as justified by the post-test survey. From the observation, proximity pointing users tend to place the physical cursor close to the target object, making the relative location easy to process. In summary, motivated by a direction-based interaction design (e.g., radial menu), proximity pointing reduces users’ cognitive load, avoids the potential distractive influences of the information redundancy, and provides a premium solution over the traditional content-based verification. Overall, the carefully designed information redundancy can facilitate the adoption of the physical cursor and improve human-machine information synchronization.

Third, we highlight the priming effect of the pointing technique on the following reading-aid adoption in information processing. The rules of the pointing may generate a spill-over effect on the following reading and even confirmation behavior. In direct pointing, occlusion requires the users to memorize the content underneath the cursor. Therefore, the cursor is considered as the symbol standing for the content. Such a direct connection implies the users to place the physical cursor as a visual anchor to highlight their focus. We observed the phenomenon in both read-and-count and read-and-write tasks. Even though the occlusion caused a serial of actions, including troublesomely lifting up and putting down the physical cursor, the users persisted in following the rules of “placing-on strictly” but not placing it beside the target. This observation suggests that users prefer to keep consistent with instinctive uses of the cursor and extend the application of existing rules but reject to develop new rules for specific needs. On the other hand, in proximity pointing, the physical cursor is located approximately to the target, free from occlusion issues on the shared display. Multi-user collaboration on a shared display would alleviate the occlusion, making the searching and navigation distracted. One might criticize proximity pointing that the cursor can not accurately show its selected object to other collaborators. The issue could be solved without any layout changes on the shared display but simply highlight the selected object.

Fourth, we explore whether users optimize the moving distance to select multiple objects when using proximity pointing. Although having been informed about the shortcut for cluster selection, users did not adopt the shortcut. Maybe users are occupied with learning the basic functions in the limited rounds of tests and lack the cognitive sources to explore advanced skills. Future studies may conduct more extended tests to reveal the skill advances by self-taught.

Future research should explore the interaction of layout and pointing techniques on usability. So far, the evidence implies that the cursor used as an anchor might improve users’ performances exclusively when dense layout. A cursor might hook the user’s attention and help them focus on their selected content more efficiently when they shift attention between tabletop and mobile devices. As our results, in the condition of cluster selection in read-and-count tasks, the positive influence of anchoring point offsets the disadvantages of direct pointing in completion time caused by occlusion and moving operations. The effect in read-and-write tasks was not found, suggesting dense layout might be the boundary condition for the influences of the cursors used as an anchor. Further research may rule out potential confounds caused by the tasks and provide more evidence.

Our research highlights the two types of pointing technique that supporting offline collaboration and examines the fundamental rules in adopting proper pointing techniques contingent on the task types. Future research should further test the tasks requiring high collaboration and examine how the tabletop-based collaboration system influences the effectiveness of group decision makings. For example, it could be an interesting topic how system facilitates transparent information sharing and changes the role of stakeholders in planning and business negotiation scenarios. Future research may recruit participants to perform collaborative works in a natural setting and reveal interesting findings at a group level.

9 Practice implications

Our findings provide implications for the design of tabletop-centric cross-device interaction. First, we recommend the adoption of proximity pointing in cross-device interaction. The design of direct pointing is motivated by a popular Browse-Move-Click model and proved to be an optimal choice in traditional tabletop applications with a mouse. By contrast, direct pointing is accepted well by users in the context of tabletop-centric cross-device interaction. The understanding of proper pointing techniques needs to update in cross-device interaction.

Both pointing techniques apply two-handed interaction. However, the users who adopt proximity pointing may have difficulty in clicking a target object in a 2 × 2 grid at an unachievable distance of a single thumb. Consequently, the user has to utilize the second hand to complete the selection on the mobile and impede the selection. Therefore, in proximity pointing, it is desirable to set the location of the 2 × 2 grid on a smartphone easy to access using a single thumb.

In proximity pointing, the cursor avoids occlusion but is seldom used as an anchor to aid information processing. On the contrary, in the direct pointing, the cursor occludes but is used to help to read. We did not find evidence that users would spontaneously adjust the rules to optimize the uses. Therefore, it is necessary to help users of proximity pointing realize the potential anchor role of the cursor. Also, cluster selection as an advanced technique needs intentional teaching and learning.

10 Conclusion

This paper examines the effect of proximity pointing and direct pointing using physical cursors on completion time and errors in the tabletop-centric cross-device interaction. Three types of tasks, i.e., selection task, read-and-count task, and read-and-write task, were designed to test both pointing techniques. The results confirm the advantage of proximity pointing in the completion time and the correctness rate, revealing the moderation effect of the selection mode and pointing techniques on completion time. The system should include an accessible 2 × 2 grid with a single thumb and a visual anchoring marker to improve the usability of proximity pointing. We recommend proximity pointing with a physical cursor since it avoids occlusion and potentially reduces the total moving distance for general uses (except for reading tasks in a dense layout). Our research sheds light on the understanding of information redundancy and the procedure to release information in promoting usability.