Camera Mouse + ClickerAID: Dwell vs. Single-Muscle Click Actuation in Mouse-Replacement Interfaces

Abstract

Point-and-click interface modalities are a pervasive method of interacting with graphical user interfaces. Users of mouse-replacement interfaces use alternative input devices to replace the mouse for pointing and clicking. We present a comparison of click actuation modalities with users of the Camera Mouse, a motion-tracking mouse interface. We compare dwell-time click generation against detecting a single intentional muscle contraction with an attached sensor (ClickerAID). A preliminary evaluation was conducted as well as an in-depth case study with a participant with the neuromuscular disease Friedreich’s Ataxia. The case study shows modest temporal differences among the test conditions in movement time and throughput, though the participant subjectively favored the ClickerAID interface.

1 Introduction

Users of mouse replacement interfaces must perform two disjoint tasks while using graphical user interfaces. These tasks involve first positioning the mouse pointer (“pointing”) followed by selecting the user interface element under the pointer (“clicking”). Here we investigate two alternative mouse selection techniques in the context of a mouse-replacement interface: dwell-time selection and single-muscle contraction activation.

Our investigation evaluates selection options for people who use the Camera Mouse¹ [1, 7] – a computer-vision-based mouse-replacement interface that tracks head motion to move a mouse pointer on the screen. The Camera Mouse allows clicking via a configurable dwell-time selection. In order to use dwell selection, the user must keep the mouse pointer relatively still over the target for a specified period of time. Users with different motor abilities have varying levels of success with dwell-time selection. Configuration options are shown in Fig. 1.

Fig. 1.

CameraMouse dwell-time click configuration options. The feature tracking is selected in the video window. The click configuration can select the dwell radius and dwell time as well as single- or double-clicking.

One common issue is the “Midas Touch” problem [5], the unintentional selection of a target. This can be due to a short dwell duration or to the user resting or reading information on the screen. It is particularly a problem when a user interface contains interactive elements that take up a large portion of a screen, thus leaving only a few “rest areas” on the screen.

Another common problem with dwell-time selection is the inability to select small user interface elements. This may be due to a dwell-time setting that is longer than the user can comfortably maintain. In addition, users with involuntary motions may have difficulty holding the pointer still for any period of time.

ClickerAID [4] is an alternative selection modality. It uses intentional muscle contractions to actuate a mouse event. It is not limited to any one particular muscle; rather, the user decides what works best for him or her (eyebrow, jaw, cheek, chin, etc.). This makes the system very flexible. The ClickerAID configuration interface is shown in Fig. 2.

Fig. 2.

ClickerAID configuration. The top window depicts the signal from the piezoelectric sensor. Configurable threshold, offsets, and gains are used to control and tune event actuation. The type of click functionality can be selected by the user. Image credit Felzer and Rinderknecht [4].

The user chooses a suitable muscle group that he or she is able to reliably control (e.g., the brow muscle). The muscular activity of that muscle group is acquired using a piezoelectric sensor in contact with the skin directly over the muscle and, in the brow example, kept in place with a flexible headband. ClickerAID compares the intensity of the input signal to an adjustable threshold. Whenever the threshold is exceeded, the software detects a contraction event and executes a corresponding code segment (resulting in the emulation of a mouse click). ClickerAID was developed to provide mouse click functionality based on a previous system: HaMCoS (HAnds-free Mouse COntrol System) - which used the same single-muscle actuation to provide all mouse functionality [3].

In the following sections, we present an evaluation of Camera Mouse’s visual tracking of a user’s head motion combined with ClickerAID’s muscle contraction actuation. With this combination, the Camera Mouse is used to position the mouse pointer and the ClickerAID interface is used to actuate selection. This is compared against dwell-time selection from the Camera Mouse alone.

2 Preliminary Evaluation

2.1 Participants and Apparatus

We performed a preliminary evaluation of dwell-time selection versus ClickerAID selection while participants used the Camera Mouse. Ten participants, five female, mean age 29, participated in the preliminary evaluation. Half performed the dwell-time condition first, while the other half performed ClickerAID first. All performed a baseline Touchpad condition last.

Fig. 3.

FittsTaskTwo - User interface test software highlights a target circle. When the user clicks on the target, the next circle is highlighted. The test software is configurable for different size targets and distance between targets. The overlaid arrows indicate the order that targets are highlighted.

The interface test was conducted on a 13-inch laptop screen set to a resolution of 1366 $\times$ 768, viewed from a distance of approximately 2.5 feet. For the Camera Mouse, a Logitech® HD Pro Webcam C920 was used, and ClickerAID was operated with the help of a headband sensor monitoring the muscular activity of the brow muscle. The following Camera Mouse settings were used for all participants: medium horizontal and vertical gain, very low smoothing, and dwell-time click area was set to “Normal” and 1.0 s.

2.2 Procedure and Design

The preliminary investigation employed an interactive evaluation tool called FittsTaskTwo² ([6], p. 291). Users performed selection tasks that required them to first position the mouse pointer over a target and then select the target before moving to the next target (Fig. 3). We performed a comparative experiment using this interactive tool to analyse the performance of users with the different interaction modalities. A log file recorded the mouse trajectory and click events.

Each participant’s session contained four sequences of thirteen targets at amplitudes 300 and 600 and widths 50 and 80 pixels. The main independent variable was input method with the following three conditions:

• CM_DWELL – Camera Mouse with 1.0 s dwell time,

• CM_CA – Camera Mouse with ClickerAID,

• Touchpad – standard pointing method.

The experimental protocol therefore included 156 trials (3 $\times$ 4 $\times$ 13) for each participant. The dependent variables were movement time (speed), throughput (speed and accuracy – bits/s), error rate (%), and target re-entries.

2.3 Results and Discussion

The Touchpad baseline was clearly the better method for the dependent variables movement time and throughput. The mean movement time was lower for the Touchpad (1302 ms) compared to CM_CA (2226 ms) and CM1000 (2609 ms). The differences were statistically significant ($F_{2,18} = 49.4$, $p < .0001$). Bonferroni-Dunn post hoc comparisons revealed significant differences between all pairs ($p < .05$). See Fig. 4a.

For throughput (speed and accuracy), the Touchpad had a better mean (2.10 bits/s) compared to CM_CA (1.43 bits/s) and CM1000 (1.28 bits/s). The differences were statistically significant ($F_{2,18} = 12.3$, $p < .0005$). Pairwise, CM_CA had 11.7 % higher throughput than CM1000. Figure 4b summarizes these results. Bonferroni-Dunn post hoc comparisons show significant differences only between the Touchpad condition and the CM conditions ($p < .05$), but not between the CM_CA and CM1000 conditions. Nine out of ten participants had faster movement time with CM_CA than CM1000, and eight out of ten had higher throughputs in those respective conditions.

Fig. 4.

Preliminary evaluation: (a) movement time; (b) throughput, which combines speed and accuracy. Error bars show $\pm$1 SE.

For the accuracy measures, the dependent variable error rate again favors the Touchpad baseline, while target re-entry favors the CM_CA condition (see Fig. 5). The Touchpad condition averaged 0.12 re-entries/trial compared to 0.10 for CM_CA and 0.15 for CM1000. There was no significant effect on target re-entries ($F_{2,18} = 0.9$, ns). Error rate demonstrated larger differences with means of 3.8 % for Touchpad, 8.1 % for CM1000, and 10.8 % for CM_CA. The differences were statistically significant ($F_{2,18} = 4.2$, $p < .05$). However, Bonferroni-Dunn post hoc pairwise comparisons did not show statistically significant differences in either error rate or target re-entries.

Although CM1000 had slightly lower error rates than CM_CA on average, individual participant performance was not consistent in comparison to the throughput measures. Some participants had larger error rates in one or the other condition, while others had equal error rates. While observing the trials, we noted that a few participants attempted to quickly click in the CM_CA condition without stopping long or at all, causing them to miss the target. In the CM1000 condition the dwell-time clicking forced them to stop and hit the target. With the Touchpad, some participants had zero errors, while others had errors caused by inadvertently moving (i.e., dragging) the pointer while attempting to tap the touchpad.

Fig. 5.

Preliminary Evaluation - Accuracy measures (a) error rate; (b) target re-entries (a low value generally means good cursor control). Error bars show $\pm$1 SE.

On subjective measurements, all but one participant said that the ClickerAID was either “somewhat easier” or “much easier” compared to dwell-time clicking. Almost all participants felt they had more control with ClickerAID and the ability to intentionally click rather than waiting for a click to happen. Many mentioned that it was difficult to keep the mouse moving to avoid unintentional clicks and then to hold still in a small area to intentionally click. The one user who preferred dwell-time clicking stated that the sensory feeling of the headband made her uncomfortable and that she worried her face movements might trigger an unintended click. This participant was not the same participant with faster movement time or higher throughput in that condition.

Several years of experience with Camera Mouse users with various abilities suggest that individual user’s abilities may be the determining factor in efficacy and preference between the two click actuation modalities evaluated here. For users of the Camera Mouse who have difficulty holding the head still to activate the dwell-time clicker in a small region, the ClickerAID activation may provide a more reliable and intentional activation option. However, users who have difficulty actuating the muscle required by the ClickerAID sensor may have better performance with the dwell-time option. The preliminary evaluation included only subjects with typical motion abilities.

3 Case Study

In addition to the preliminary evaluation with multiple participants, a case study documenting the experiences of a single 44-year-old male computer user, the second author, who has the neuromuscular disease Friedreich’s Ataxia, has been conducted. The purpose of the study was to determine whether there are any (objective or subjective) advantages for this particular user of Camera Mouse with ClickerAID compared to the usual dwell method, or to the participant’s ordinary pointing technique.

3.1 Apparatus

The study took place in the participant’s work environment. The click targets were displayed on a 33-inch TV monitor, set to a resolution of 1920 $\times$ 1080, viewed from a distance of approximately 2 feet. For the Camera Mouse, the same webcam was used as in the preliminary evaluation, and ClickerAID was again operated with the help of a headband sensor monitoring the muscular activity of the brow muscle.

Fig. 6.

Experimental setup: (a) Camera Mouse window during the generation of a click with ClickerAID; (b) view from behind; (c) trackball used as ordinary pointing device.

Figure 6 shows the entire setup. Figure 6a depicts the exact moment when the participant raised his eyebrow to generate a click; the ClickerAID window flashes red as a confirmation. The ClickerAID window is shown on top of the Camera Mouse window, and normally illustrates the muscular activity of the brow muscle as a green curve on black background. To interact with a PC, the second author regularly uses OnScreenDualScribe [2], which efficiently replaces both the standard keyboard and mouse with a small manually-operated keypad. He infrequently uses the trackball device depicted in Fig. 6c for pointing; however, to make the case study as comparable as possible to the preliminary evaluation, this standard device was chosen as a baseline.

Furthermore, the second author is currently developing a pointing-driven variant of OnScreenDualScribe for users who cannot employ their hands. The idea is as follows. OnScreenDualScribe reduces operating the entire keyboard to pressing just a few keys. Obviously, not all of those keys can always emulate the same functionality. To remind the user of the currently valid associations, an “avatar” of the keypad is displayed on screen. Making the buttons of the avatar clickable naturally leads to a point-and-click keyboard replacement.

Compared to conventional onscreen keyboards, this extension has the advantage of needing less screen space. In particular, users of non-manual mouse-replacement interfaces might appreciate that they have to span smaller distances between keys. Although being able (within limits) to use both hands, the second author had the additional goal when doing the case study to look into potential benefits for himself and people with similar symptomatologies. The manual device was the natural choice since the on-screen clickable version of OnScreenDualScribe requires external mouse control.

3.2 Procedure and Design

The case study employed the same evaluation tool as in the preliminary evaluation. It consisted of two sessions, each containing four sequences of thirteen targets at amplitudes 400 and 800 and widths 50 and 100 pixels. The main independent variable was input method with the following conditions:

• CM1500 – Camera Mouse with 1.5 s dwell time,

• CM2000 – Camera Mouse with 2.0 s dwell time,

• CM_CA – Camera Mouse with ClickerAID,

• Trackball – standard pointing method.

The experimental protocol therefore included 416 trials (2 $\times$ 4 $\times$ 4 $\times$ 13) plus any sequence repeats. The participant’s subjective evaluation of the Camera Mouse configurations resulted in the following settings: low horizontal gain, a medium vertical gain, and high smoothing. He used these settings in all trials. The difference between the two sessions was that the dwell-time click area was set to “Normal” in the first session and “Small” in the second. The dependent variables were the same as in the preliminary evaluation.

3.3 Results and Discussion

While developing the experiment, the participant tried shorter dwell times, but these resulted in frequent sequence repeats due to unintended clicks. A zoomed camera image was also tried, but the quick motion made it too sensitive for accurate dwell-time clicking.

Despite attempts to configure the input methods to optimal settings, the evaluation may not lead to a clear outcome among the conditions. Results showed better performance with the 1.5 s dwell-time setting of CM1500 compared to the 2.0 s dwell-time of CM2000. Here, we will compare the better of the dwell-time conditions against the ClickerAID condition. When considering the dependent variable movement time, CM1500 had the lowest mean movement time of 5960 ms compared to CM_CA (6713 ms, 13 % slower). For throughput, CM1500 again had the better mean of 0.488 bits/s compared to CM_CA with 0.454 bits/s (7 % less). Figure 7 summarizes the results of all four conditions.

Fig. 7.

Case Study - Temporal performance for the four conditions: (a) movement time; (b) throughput, which combines speed and accuracy.

Fig. 8.

Case Study - Accuracy measures (a) error rate; (b) target re-entries (a low value generally means good cursor control).

When considering the accuracy of user input, the dependent variables error rate and target re-entry favor the Trackball condition (see Fig. 8). The Trackball condition averaged 0.375 re-entries/trial compared to 0.894 for CM1500, and 0.692 for CM_CA. Error rate demonstrated the largest differences with mean percentage of 1.9 for Trackball, 19.2 for CM1500, and 18.3 for CM_CA. Pairwise comparisons between the dwell-time and ClickerAID conditions shows CM1500 had 29 % higher target re-entry rate and 4.9 % higher error rate than CM_CA. The accuracy of the trackball is not surprising given that it was the slowest input condition – it is a well-known trade-off that users make fewer mistakes when given more time. In addition, since the pointer stops moving when the trackball is not being touched, it’s likely that the user would not click the button when the pointer is off the target – contributing to the low error rate.

While differences in these four dependent variables were considerable, we note that tests for statistically significant differences could not be performed given that there was only one participant. The differences for movement time and throughput were modest compared to those for target re-entries and error rate.

Subjectively, the participant preferred the ClickerAID condition, as it demanded less physical effort than the trackball, while allowing him to “stay in control”. This feeling of control was in contrast to the dwell-time conditions where the user must wait for a click to occur, and is similar to what was noted by participants in the preliminary evaluation. The difference in this effect is illustrated in Fig. 9. When using the Camera Mouse, the participant is unable to keep the mouse pointer completely still; it is constantly moving, with sometimes larger, sometimes smaller amplitudes. When near the desired target, the pointer often drifts out of the intended area while – in the dwell-time conditions – waiting for a click to be issued. When that happens, the pointer has to be repositioned quickly, which regularly results in overcorrections. An extreme example is the four circled targets in Fig. 9a. With ClickerAID, the participant does not have to wait for a click to be issued by the dwell timer; instead, he can intentionally actuate a click. This ability allows the participant to correctly actuate a click on a target and move directly to the next target, as shown in Fig. 9b near targets T3 and T10 and the path between.

Fig. 9.

Staying in control: (a) Waiting for a click to be invoked while dwelling; (b) intentional clicks in the CM_CA condition.

Fig. 10.

Several “strokes” needed to maneuver the mouse pointer from one target to the next in the Trackball condition.

Given that the participant in the case study is able to use both hands, it might not have been expected that the Trackball condition would be so slow. The loss of speed is related to the physical size of the trackball (about two inches diameter); when moving across the screen, especially in the large outer radius sequences, the participant usually needs several “strokes” to span the required distance. This means that he must roll the ball all the way in one direction, then lift his hand and move it back in the opposite direction, and then put his hand down and roll again – very cumbersome. This pattern is evident in the path between targets T3 and T9 in Fig. 10.

4 Conclusion and Future Direction

The empirical evaluation in the case study indicates considerable difference in movement time and throughput among the conditions evaluated, while much larger differences were observed for the accuracy measures which favor the Trackball condition. Nevertheless, the subjective preference for the ClickerAID indicates that the different clicking modalities may offer an improved experience for people who use mouse-replacement interfaces. This subjective preference for the ClickerAID over dwell-time was similarly observed in the multi-user preliminary investigation. A further evaluation will expand the case study methodology to a larger number of participants with a variety of motion abilities, which should allow for statistical analyses across several users. An additional future direction may include the pointing-driven variant of the replacement tool the second author is currently developing for users who cannot employ their hands.