Investigation of Transferring Touch Events for Controlling a Mobile Device with a Large Touchscreen

Abstract

When users hold large mobile devices equipped with a large touchscreen in one hand, the region distant from the thumb is too distant for users to control. This forces users to change their hand posture so that their thumb can reach to the top half. To address this problem, we explore a technique that transfers touch events on the bottom half of a touchscreen to its top half. This technique may allow users to control all regions of a large touchscreen by using only the bottom half. Thus, users can control a mobile device without changing hand posture. We conducted a user study to investigate the feasibility of our technique. From the results, our technique is marginally faster than direct touch and thus, might be feasible.

Keywords

1 Introduction

While the vast majority of users want single-handed interaction with mobile devices [9], large mobile devices equipped with a large touchscreen have been spreading widely. This trend presents a new problem that we call the reachability problem: when users hold such large mobile devices in one hand, the region distant from the thumb is too distant for users to control [15]. This forces users to change their hand posture so that their thumb can reach the top half.

In this research, we explore a technique for controlling a large device (Fig. 1) to address the reachability problem. This technique transfers touch events on the bottom half of a touchscreen to its top half. This design allows users to control the top half using only a bottom half. Therefore, the design may allow users to control all regions of a large touchscreen without changing hand posture and thus grip the device stably. We also report the user studies we conducted to investigate the feasibility of our technique.

Fig. 1.

Our technique allows users not only to point but also to perform touch gestures at the top half of a large mobile device without changing their hand posture. As an example, suppose that a user wants to select the text “HCI International 2015” in an Web browser. (a) In our technique, a red X-shaped cursor appears as visual feedback. The user selects the initial text “HCI” by long-tapping using our technique. (b) She points to the handle using our technique. (c) To adjust the handle to the end text “2015”, she drags her thumb in the bottom half without removing it from the touchscreen. (d) She finishes the text selection by removing her thumb from the touchscreen after adjusting the handle (Color figure online).

2 Related Work

There have been many studies using a cursor controlled by dragging to address the reachability problem. MagStick [11] is a cursor that moves in the opposite direction of a thumb and sticks to a target. CornerSpace [15] uses a UI shown by BezelSwipe [10] to determine the cursor’s rough position. Extendible Cursor [6] is a cursor activated using BezelSwipe or a touch gesture using a wide area of the finger; users can point at a distant position by moving the cursor quickly (i.e., the control-display ratio is over one) by dragging. While these techniques are designed for pointing, our technique is designed to be used for not only pointing but also for other touch gestures (e.g., dragging or long tapping). ExtendedThumb [7] is activated with a double tap and shows a cursor moving in the direction of the thumb’s movement. A touch event (e.g., tapping and dragging) following a user’s touch up is transferred to the cursor position (e.g., tapping an icon at the top half of a touchscreen). Our technique also transfers touch events but does not require the position to which touch events are transferred to be determined.

Many studies have use indirect pointing techniques other than cursors to solve the problem of “fat fingers” [12]. When the user touches the screen, Shift [14] creates a callout showing a copy of the occluded screen area and places it in a non-occluded location. The callout also shows a pointer representing the selection point of the finger. LinearDragger [1] is the target acquisition technique for small and clustered targets. LinearDragger allows users to point at a target from a clustered group of selectable objects by scanning the objects one by one while continuously dragging.

ThumbSpace [5] was designed to improve access to the borders and corners of the screen. It uses an on-demand “radar view” that the user can trigger at the center of the screen. Interacting directly on this radar view allows the user to reach all locations on the screen. Thumbspace thus works as an absolute pointing touchpad superimposed on the standard touchscreen.

Other studies that have allowed users to move targets to near their thumb to make distant objects easily accessible. LoopTouch [13] provides a roll gesture, where a thumb moves on the front of the device and the same hand’s index finger on the back of the device, close to each other. This gesture rolls the whole screen. Because users can make the whole screen roll by this gesture, they can move a target near to their thumb. To detect this gesture, LoopTouch requires a mobile device equipped with a touch panel on the back. Sliding Screen [6] is activated by BezelSwipe or a touch gesture using a wide area of the finger; this moves all objects on a whole screen to the touched position. In the study of Nagata et al. [8], users could pull all the objects near their thumb; among them, users could touch the target directly. Drag-and-Pop and Drag-and-Pick [2] allow users to pull icons on a desktop screen near to the cursor by dragging.

In contrast to these techniques, our technique is indirect. However, it incorporates the advantage of LoopTouch and Sliding Screen: the users can virtually bring a distant region of a screen near to their hand. This advantage gives us a chance to design a single-handed control technique for large mobile devices that allows the users to perform various touch gestures on the distant region, while still maintaining grip stability.

3 Transferring Touch Events from the Bottom to the Top

Our technique transfers touch events on the bottom half of a touchscreen to the top half (Fig. 1). This design leads to a stable single-handed grip because users can control a large mobile device without changing hand posture: users control the top half of a large touchscreen using only its bottom half. Our technique has a mode that transfers touch events on the bottom half to its top half. In this mode, our technique colors the bottom half red (Fig. 1) and displays a cursor at its corresponding point in the top half as visual feedback when somewhere on the bottom is touched.

As a use case of our technique, Fig. 1 shows a text selection. Suppose that a user wants to select the text “HCI International 2015” in an Web browser. First, the user long-taps the start of the target text while using our technique to transfer the touch event on the bottom half to the top half (Fig. 1a). Second, she points to (Fig. 1b) and moves (Fig. 1c) a handle to the end of the target text using our technique. Finally, she completes selection by removing her thumb from the touchscreen after moving the handle from the start to the end of the target text (Fig. 1d).

4 User Studies

To investigate the feasibility of our technique, we measured its performance in terms of two typical tasks on mobile devices: pointing and dragging. To clarify the pros and cons of our technique, we compared our technique (Transfer) with the existing direct touch (Touch) as the control condition.

The two controls were performed under the following two dummy conditions: non dummy (ND) and with dummy (WD). In the ND condition, only one target square or one target seek bar was displayed at a time on the screen as shown in Figs. 3a and 7a. On the other hand, in the WD condition, one target or one seek bar was displayed with 23 dummy targets or 5 dummy seek bars, as shown in Figs. 3b and 7b. These dummies were displayed to serve as hints to point out the place where events will be transferred. We explained this dummy design to participants and asked them to consider this hint positively. In the Transfer condition, we asked participants to use our technique to control all the targets in the top half, even if their thumb could reach the target, and to use direct touch to control all the targets in the bottom half.

4.1 Participants and Experimental Environment

The participants were 10 undergraduate and graduate students from the engineering department, one undergraduate student from the art department, and one professional designer of user interfaces aged from 22 to 26 (M = 22.5, SD = 0.5). Two were left-handed, and four were females. None had ever used our technique. All used their smartphones daily.

The user studies were conducted in the environment shown in Fig. 2. For the smartphone in this experiment, we used a Samsung GALAXY Note 3 SCL22 (79 $\times$ 151 $\times$ 8.4 mm, 5.7” display). We chose this device because it is almost the same size as the device used in the user study of Yu et al. [15], one of previous studis on the reachability problem. We asked the participants to sit on a chair and control the smartphone in one hand as they usually do. To control experimental conditions between participants, we also asked the participants to put their elbow on a desk and hold the smartphone without supporting it by using the desk or their bodies.

Fig. 2.

Experimental environment.

We asked the participants to perform the tasks as accurately as possible. This instruction is based on the design principle that focuses on accuracy rather than speed. If the participants could not reach the target (which they frequently could not under the Touch condition), we asked them to change the hand posture of the hand holding the smartphone. If participants could not change their hand posture in midair, we permitted them to touch the smartphone to the desk to support it.

4.2 Task

We asked the participants to perform a pointing task and then a dragging task. In addition, the order of the two dummy conditions was ND and WD. We asked them to complete 12 sessions (4 learning sessions, 4 training sessions, and 4 real sessions) for each task in both techniques and both dummy conditions. We divided the participants into two groups to counterbalance the order effect between the two technique conditions: one group performed the tasks under the Transfer condition first; the other performed them under the Touch condition first.

We asked them to answer a questionnaire after they had finished all the tasks. The experiment lasted approximately two hours per participant, including the questionnaire. All participants were paid 1640 JPY (approximately 16 USD) for their participation.

4.3 Study 1: Pointing

In each trial, the participants pointed to a target square, which was blue (e.g., the blue square in Fig. 3b). The target square was the same size as the application icons displayed on the default home screen of this device. Participants had to successfully perform each trial before proceeding to the next trial.

A participant started each session by touching the touchscreen. A new target square was automatically presented by the experimental software when a trial was completed.

In this task, the independent variables were target position (24) and technique (Transfer and Touch). Each participant performed 12 sessions in each combination of positions, techniques, and dummy conditions, thus performing 1152 (24 $\times$ 2 $\times$ 12 $\times$ 2) trials in total. The target position was presented in randomized order.

Fig. 3.

Positions of target squares in (a) ND and (b) WD.

Results

Trial-times for both conditions: We analyzed the time to complete a trial (trial-time) in real sessions. The two left and two right bars in Fig. 4 show the mean trial-times of both techniques in ND and WD, respectively. The mean times were 1044 ms in Transfer-ND and 935 ms in Touch-ND. In turn, the mean times were 901 ms in Transfer-WD and 976 ms in Touch-WD. Repeated measures analysis of variance showed a significant main effect on dummy condition (p(11) = .045 $<$ .05). With pairwise t-tests, there was a significant effect on dummy condition in Transfer (p(1) = .000 $<$ .050). In addition, there was a marginal significance on the technique condition in WD (p(1) = .082 $<$ .100). This result suggests that pointing in Transfer was marginally significantly faster than that in Touch in WD.

Trial-time for each session: We show trial-times of all conditions in each session in Fig. 5. In ND, the trial-times in all the sessions after the second are faster than that in the first one (p(11) = .000). This result suggests that participants learned how to use our technique during the first session. Thus, our technique can be learned with little effort (approximately 0.55 min).

Error Rate: We analyzed the error rates in the real sessions. The two left and two right bars in Fig. 6 show the mean error rates of both techniques in ND and WD, respectively. Note that this study was designed such that participants had to successfully perform each trial before proceeding to the next trial, even if this required multiple trials. As such, all trials were ultimately “successful”. However, we classified the cases where trials were not cleared on the first attempt as errors, and analyzed this error data. The mean error rates were 12.76 % in Transfer-ND and 6.51 % in Touch-ND. In turn, the mean error rates were 3.99 % in Transfer-WD and 5.99 % in Touch-WD. Repeated measures analysis of variance showed a significant main effect on dummy condition (p(11) = .000 $<$ .05). With pairwise t-tests, there was a significant effect on dummy condition in Transfer (p(1) = .000 $<$ .050). In WD, while Transfer is more accurate than Touch, we found no significant effect.

Fig. 4.

Trial-times for each technique and dummy condition.

Fig. 5.

Trial-times of all conditions for each session.

Fig. 6.

Error rates for both techniques and both dummy conditions.

Considerations. Although, in WD, the trial-time in Transfer was marginally significantly faster than that in Touch, there was no significant or marginally significant effect on the error rate. These results suggest that the trial-times are faster in Transfer-WD than in Touch-WD because our technique does not include time for changing the hand posture in Transfer. In the questionnaire, some participants said that it is easy to perform this task was easy to perform in Transfer because they did not need to change their hand posture.

To examine this hypothesis, we compare the trial-times of each region in Tables 1, 2, 3, and 4. We mirrored the data of the two left-handed regions (e.g., A1 and A4). Even in ND where no hint was displayed, the trial-time in Transfer in region A1 was shorter than that in Touch. Note that A1 is the most distant region from the thumb. Furthermore, in the dummy condition, the trial-times in Transfer in row F, which is the lowest region, were shorter than those in Touch. This result suggests that our technique does not require users to change hand posture.

These features are far more valid in WD where hints are displayed. In WD, the trial-times in Transfer in row A, which is the highest region, and rows D–F which are lower regions, are shorter than those in Touch. Because of this, the trial-time in Transfer are shorter than those Touch. Thus, our technique allows users to grip a smartphone stably (Tables 5 and 6).

In rows B and C, the trial-times were shorter in Touch than in Transfer, and the error rates were lower in Touch than in Transfer except Region B4. The possible cause of this derives from the fact that users were required to touch in rows E and F, which are the most difficult regions to touch, and to point to rows B and C using Transfer. However, this problem could be solved by allowing users to use Transfer and Touch arbitrarily by expanding touch vocabulary.

Table 1. Trial-times of each region in Transfer-ND.

Table 2. Trial-times of each region in Touch-ND.

Table 3. Trial-times of each region in Transfer-WD.

Table 4. Trial-times of each region in Touch-WD.

Table 5. Error rates of each region in Transfer-WD.

Table 6. Error rates of each region in Touch-WD.

4.4 Study 2: Dragging

We asked the participants to adjust a seek bar by dragging its knob. We illustrate the seek bars in both dummy conditions in Fig. 7. Following the scroll experiment of Cockburn et al. [3], we displayed a target as a range of value (target range) whose center is determined randomly. The range is bounded by two red bars. The participants were instructed to drag the knob and release their hand from it when the knob was within the target range. The action to start the session was the same as that of the pointing task.

In this task, the independent variables were seek bar position (6) and technique (Transfer and Touch). Each participant performed 12 sessions in each combination of factors and dummy conditions, thus performing 288 (6 $\times$ 2 $\times$ 12 $\times$ 2) trials in total. The seek bar position was presented in a randomized order.

Results

Trial-time for both conditions: We analyzed the trial-times in the real sessions. The two left and two right bars in Fig. 8 show the mean trial-times of both techniques in ND, respectively. The mean trial-times were 2235 ms in Transfer-ND and 2151 ms in Touch-ND. In turn, the mean times were 1900 ms in Transfer-WD and 1978 ms in Touch-WD. Repeated measures analysis of variance showed a significant main effect on dummy condition (p(11) = .006 $<$ .05). With pairwise t-tests, there were significant effect on dummy condition in Transfer (p(1) = .001 $<$ .050) and a marginal significance on dummy condition in Touch (p(1) = .059 $<$ .100).

Trial-time for each session: There were no significant effects on the trial-times in 12th session in ND and the trial-time of 1st session in WD (Transfer: p = .263 $>$ .050, Touch: p = .186 $>$ .050) (Fig. 9). This result suggests that dummy does not significantly affect the trial-time (if there were significant effect, the trial-time of the first session in WD would have been shorter than that of the 12th session in ND). Moreover, the trial-time of each technique in WD was significantly shorter than ND, and the trial-time of Transfer is slightly shorter than that of Touch in WD. These results might be due to the learning effect. Therefore, after finishing learning, the trial-time of Transfer should become shorter than Touch in ND.

Fig. 7.

Positions of the seek bars in (a) ND and (b) WD (Color figure online).

Error Rate: We calculated and analyzed the error rates in the real sessions in the same manner as in Study 1. The two left and two right bars in Fig. 10 show the mean error rates of both techniques in ND and WD, respectively. The mean error rates were 24.00 in Transfer-ND and 13.54 in Touch-ND. In turn, the mean error rates were 11.80 in Transfer-WD and 14.23 in Touch-WD. Repeated measures analysis of variance showed a significant main effect on dummy condition (p(11) = .012 $<$ .05). With pairwise t-tests, there was a significant effect on dummy condition in Transfer (p(1) = .002 $<$ .050). With pairwise t-test, there was a significant effect between the dummy condition in Transfer (p(1) = .002 $<$ .050) and techniques in the ND condition (p(1) = .007 $<$ .050).

Unlike trial-time, there is no significant error in dummy condition in Touch. Thus, in ND, participants could adjust the seek bars as accurately as in the WD condition, although they took longer to adjust that one. On the other hand, in the Transfer condition, the error rate in WD is lower than that in ND. Thus, our technique requires a hint in seek tasks, as well as in pointing tasks.

Fig. 8.

Trial-times for both techniques and both dummy conditions.

Fig. 9.

Trial-times of all conditions for each session.

Fig. 10.

Error rates for both techniques and both dummy conditions.

Table 7. Trial-times of each region

Table 8. Error rates of each region

Considerations. Table 7 compares the trial-times of each region. In the ND condition, the trial-time in region A, the upper-most, in Transfer is shorter than that in Touch. Furthermore, in WD, trial-times in the all regions except C are shorter in Transfer than in Touch, because the control is easier in Touch than in Transfer in region C.

In WD, the error rates in Transfer in regions A and B are lower for over 10 % than those in Touch in Table 8. The possible cause of this derives from the fact that adjusting using Touch is difficult due to “fat fingers”, although our technique is an indirect control and nothing hide the target range. In the questionnaire, one participant said that it was hard to see the target region in the bottom half.

5 Conclusion and Future Work

We proposed a technique for transferring touch events for controlling a large device. The studies using our technique revealed that it is marginally faster than direct touch and thus might be feasible.

However, we found two areas to be further explored in our technique. First, our technique requires a hint pointing out the place where a touch event will be transferred. To this end, we will implement visual feedback of the place using hover events that can be sensed in some smartphones. Second, to use our technique realistically, we must explore the interaction design for mode switching between the two modes: the mode where touch events on the bottom half are transferred to the top half and when they are used on the bottom half as usual. For a possible solution, we are testing using the strength of touch to classify them (similar to Forcetap [4]): users touch strongly to transfer touch events. Future work will need to explore more designs.