Advertisement

The dynamic grouping keyboard: a general keyboard optimization approach for users with motor impairment

  • Yizheng Gu
  • Chun YuEmail author
  • Yuanchun Shi
Regular Paper
  • 89 Downloads

Abstract

An important feature of software keyboards is its ability to flexibly adapt the interface. In this paper, we present the Dynamic Grouping Keyboard, as a general approach to optimize text entry for users with motor impairments. Each time after a key is selected, the Dynamic Grouping Keyboard adapts the grouping of keys to optimize the interaction cost, such as the number of keystrokes and the distance of cursor movement, while maintaining the key locations to preserve keyboard layout. We identify the technical challenge, and propose a two-step algorithm for optimization: The first step estimates the amount of information that the current step should provide, while the second step minimizes the interaction cost by applying dynamic programming. Two lab studies with twelve able-bodied users were conducted, in which three keyboard cases for motor impairment were tested. Results showed that compared to an optimal static grouping (ambiguous) keyboard, the Dynamic Grouping Keyboard reduced interaction cost by 23–75%, and improved text entry rate by 16–71% across all tested cases. Subjective user feedback showed consistent preference for the Dynamic Grouping Keyboard due to its faster input speed and less fatigue.

Keywords

Grouping keyboard Keyboard layout Dynamic adaptation Optimization Language model 

1 Introduction

Text entry is a basic and fundamental interaction task for not only normal people, but also users with motor impairment. To facilitate text entry of users with limb impairment, low strength or limitation of movement, a number of techniques have been proposed using a joystick (Song 2010), a trackball (Wobbrock and 2006) or stylus-based devices (Wobbrock et al. 2003) and so on. In these works, researchers have found that for users with motor impairment, optimizing time was not the only objective; motion-related performance, such as movement distance (Harbusch et al. 2003; Song 2010; Zhai et al. 2000), and number of keystrokes needed (Gong 2007) often mattered more.

In this paper, we present the Dynamic Grouping Keyboard (DGK for short), which dynamically adapts the keyboard to facilitate text entry for users with motor impairment. In particular, it adapts the grouping (ambiguous) keys (i.e. keys overloaded with multiple letters) rather than altering key locations (Fig. 1). As a result, users can still utilize their familiarity with the keyboard layout to seek keys. As a general approach, DGK can optimize a variety of objectives, including time and various motion-related objectives.
Fig. 1

A user inputs “the” with the Dynamic Grouping Keyboard. In this example, the keyboard layout is dynamically adapted to minimize cursor movement during text entry. The white lines serve as visual boundaries specifying the groups of letters. The highlighted buttons are selected during the input

The technical challenge is how to determine the adaptation behavior of DGK. To address this, we propose a novel two-step algorithm for optimization: the first step estimates the amount of information (i.e. entropy) that the current step should provide, while the second step minimizes the interaction cost by applying dynamic programming. This algorithm provides real-time performance, and can support varying language model, different types of keyboards, and constraints on the keyboard adaptation behavior.

We evaluated DGK in two lab studies. In the first study, we tested a scanning keyboard with the objective to minimize interaction time. In the second study, we tested a cursor-based QWERTY keyboard with the objective to minimize distance of cursor movement. Two pointing devices were used: a joystick with slow speed and a trackball controlled by the chin. Evaluation results showed that compared to an optimal static grouping keyboard, DGK reduced interaction cost by 23% and 75%, and improved text entry rate by 16%, and 71% respectively. Subjective user feedback also showed consistent preference for DGK due to its faster input speed and less fatigue.

1.1 Related work

We review software keyboard techniques that optimize performance in various ways, as well as the state-of-the-art algorithms that were leveraged for keyboard optimization.

1.2 Ambiguous keyboard

Ambiguous keyboards are useful for situations where input has either human limitations [SAK (Mackenzie and Flezer 2010) for accessibility] or device limitations [for reducing keyboard size (Green et al. 2004; Komninos and Dunlop 2014; Li et al. 2011), T9]. Ambiguous keyboards overload individual keys with multiple letters, and use a language model to predict a target character after each keystroke. The design space includes the number of keys [T9, 1-line keyboard (Frank Chun Yat Li et al. 2011), one-row keyboard (Nathan Green et al. 2004), and NineType (Huang 2019)], letter-key assignment (Dunlop et al. 2012; Dunlop and Levine 2012; Kushler 1998; Lesher et al. 1998), and the prediction algorithm. To improve prediction accuracy, researchers have proposed various methods to predict a target character after each keystroke, using n-gram models (Mackenzie and Flezer 2010), domain-specific language models (Harbusch et al. 2003), sentence information (Hasan and Harbusch 2003), and even the semantic relatedness and a part-of-speech (Gong 2007).

1.3 Dynamic and adaptive keyboard

The adaptation of keyboards can take on many forms, like adapting the visual representation to facilitate visual search (Kühn et al. 2001) or adapting keyboard parameters to meet disabled people’s needs (Song 2010). Our research focuses on adapting the keyboard layout to optimize performance.

Fluctuating Optimal Character Layout (Bellman and MacKenzie 1998; Schadle 2004) rearranges the layout so that the most likely next characters are closer to the cursor to minimize KSPC—the average number of keystrokes per character. However, it also requires the user to locate each character in an unfamiliar layout, thus adding visual search time to the task.

Other techniques preserve the key locations. The Fisheye keyboard (Raynal and Truillet 2007) expands the key under a stylus to facilitate key acquisition on small screens. BigKey (Faraj et al. 2009) expands the size of keys that are probably to be needed next. Floodkey (François et al. 2010) divides a keyboard in Voronoi with key size proportional to its probability of being pressed. Gunawardana et al. (2010) identified the negative effect of key-target resizing and proposed an anchored key-target method, which aims to provide an input method that is robust to errors while respecting usability principles. SpreadKey (Merlin and Raynal 2010) dynamically recycles the needless characters (with low probability) to provide several times the characters with a high probability (improvement 51–65%). Keyglass (Raynal 2014) and QuickSuggest (Gargi and Gossweiler 2010) provide additional keys surrounding a current character to allow the user to rapidly get to the most likely character. The Semantic Keyboard (Raynal et al. 2014) adjusts the control/display ratio to make the cursor cross faster over keys containing low-probability letters (considering the prefix already entered).

1.4 Keyboard optimization

One important objective for keyboard layout optimization is to minimize distance or time of cursor movement. Zhai et al. (Zhai et al. 2002) used a Metropolis random walk algorithm to gain 50% improvement. Bi et al. (2012) applied this approach for multiple languages, reducing travel distance by 50%. Bi et al. (2010) also extended the optimization by imposing layout familiarity. Oulasvirta et al. (2013) minimized the movement for two-thumb text entry on touchscreen devices.

Multi-object optimization can also be achieved. Dunlop and Levine (2012) took into account finger travel distance, spell correction abilities, and similarity to the standard Qwerty layout for optimization. They simulated the Pareto front for optimization. Fitts’ law modelling shows a predicted 11% improvement in entry speed without taking into account the significantly improved error correction potential and the subsequent effect on speed. In addition, Bi et al. (2014) proposed to perform Multi-Objective Optimization of Touchscreen Keyboard. The independent variables were the parameters in intelligent keyboard algorithm. Again, they used a Pareto multi-objective optimization and the Metropolis algorithm.

Most keyboard optimization approaches described in prior literature are based on random search, such as Metropolis (Zhai et al. 2002) and simulated annealing (Oulasvirta et al. 2013). The advantage of such “black box” approaches are convenient to use, because good solutions can be found quickly and no assumptions must be made about the objective function. The drawback is that such methods offer no guarantee for finding the global optimum, nor do they inform distance to it. To address this limit, Karrenbauer and Oulasvirta (2014) modeled keyboard optimization as an Integer Programming problem, and solve it by branch-and-bound. The structured non-random search approach not only improved the outcomes, but also guaranteed bounds, and thus could increase designers’ confidence over the quality of results.

1.5 The optimization algorithm

At a moment, the grouping layout of DGK can be defined as the current set of grouping keys on it. Each time the user selects a grouping key, he/she provides an amount of information that narrows down the set of candidate words. This process repeats until the last letter is input and the accumulative information is adequate to specify the intended word. On the other hand, the grouping layout of DGK determines the interaction cost to select an individual grouping key on it. In general, grouping keys that contain fewer letters, which are smaller, are associated with higher interaction cost. For example, they might be more difficult to select. In fact, the interaction cost depends on not only the grouping key that contains the needed letter but also the grouping layouts of DGK at the current step and even at the previous step. Therefore, the optimization is to determine the dynamic grouping behavior of DGK, which provides the required information to specify words in the vocabulary and at the same time minimizes the interaction cost.

However, if a DGK is defined by its dynamic behavior, the space of solutions would be too large to be resolved. To break down the problem, we treat each step independently. For each step, our algorithm first estimates the amount of information that should be provided for the current step, and then optimizes a grouping layout to provide that required information with minimal interaction cost. We next describe the details of our two-step algorithm. The complete definition of the optimization problem for DGK can be found in Appendix 1.

Step #1 Estimating the required information

Figure 2 illustrates the immediate state of when a user inputs a word: the user has entered a sequence of ambiguous keys, KS1…KSm. We can then derive a set of candidate suffixes (denoted as \( S \)) by matching the aforementioned sequence to the dictionary. Now, the question is: how much information should the next step provide? We next provide our solution.
Fig. 2

The immediate state of a user inputting a word

Let us first consider a simple case. Suppose the potential suffixes are of the same length n. Then the information for a single step should be:
$$ H_{single step} = \frac{H\left( S \right)}{n}, $$
(1)
where \( H\left( S \right) \) is the entropy of all candidate suffixes, which can be computed based on their probabilities.
However, in reality, suffixes have different lengths. To deal with this, we compute the required information for each suffix length respectively, and then weight and sum them up according to their probability (Eq. 2). The result is the expected amount of information that is required for the current step:
$$ H_{expected} = \mathop \sum \limits_{k = 1}^{n} p_{k} \frac{H\left( k \right)}{k}, $$
(2)
where \( H\left( k \right) \) represents the entropy of suffixes that have a length of k, and \( p_{k} \) represents the sum of probabilities of suffixes that have a length of k.

Step #2 Optimizing the grouping layout

In this part, we first describe how to compute the amount of information a grouping layout can provide as well as the definition of interaction cost. We then describe our dynamic programming method we used to resolve the optimization problem.

The Information a Grouping Layout ProvidesThe information a grouping layout (\( gl \)) provides is computed as the entropy of the grouping keys within it (Eq. 3):
$$ H\left( {gl} \right) = - \mathop \sum \limits_{gk \in gl} p_{i} \left( {gk_{i} } \right)\log p_{i} \left( {gk_{i} } \right), $$
(3)
where \( p_{i} \left( {gk_{i} } \right) \) represents the probability of \( gk_{i} \). In general, the more information a grouping layout provides, the more grouping keys it has.

Moreover, we assume that the cost of selecting one key from a grouping layout is highly correlated to its entropy. Therefore, the grouping layouts for each step would ideally provide the same amount of information (entropy), in order to achieve uniform interaction cost over individual key selections. \( H_{expected} \) estimates the information amount for the current step by considering the local redundancy of the language model.

1.6 Interaction cost

We followed (Mackenzie and Felzer 2010) to define the interaction cost for inputting a word on DGK, which can be divided into the coding cost and scanning cost. Coding cost represents the cost of selecting each individual grouping key, while scanning cost represents the cost of selecting the target word from the candidate list. In practice, the coding cost is determined by the input device as well as the interface design:
$$ {\text{cost}}\left( {\text{w}} \right) = \mathop \sum \limits_{i = 1}^{len\left( w \right)} Coding\left( {l_{i} } \right) + {\text{Sanning}}\left( w \right). $$
(4)

1.7 Definition of the dynamic programming problem

We apply the dynamic programming approach to resolve this problem. To do this, we first define a graph representation of this problem. Take a QWERTY keyboard for example.

Then, any grouping layout can be determined by a sequence of cutting lines (Fig. 3). A cutting line is defined as one that traverses the keyboard from the top to the bottom, and partitions the keyboard layout into a left area and a right area. We denote a cutting line as Fi,j,k, and the left area as Ai,j,k., where the subscripts represent the locations where the line crosses each row. Thus, a grouping key can be considered the difference of the left areas corresponding to two adjacent cutting lines. For example, the grouping key {FGHCVB} is the difference of A6,3,3 and A6,6,6.
Fig. 3

An example grouping layout, which can be represented as a sequence of cutting lines: F0,0,0->F6,0,0->F6,3,3->F6,6,6->F10,10,10

Figure 4 shows the directed graph that describes sequences of cutting lines for all grouping keyboard layouts, in which a vertex represents a cutting line, and an edge represents a grouping key. The graph has a single source vertex (F0,0,0), and a single sink vertex F10,10,10. Each individual path from the source vertex to the sink vertex determines a unique grouping keyboard layout. Note that there is a path from Fi,j,k to Fi’,j’,k’ iff i ≤ i’, j ≤ j’, and k ≤ k’; there is an edge between Fi,j,k to Fi’,j’,k’ iff the difference of Ai,j,k to Ai’,j’,k’ is a rectangle area according to our constraint on the shape of grouping keys. Further details of resolving this dynamic programming are described in Appendix 2.
Fig. 4

The directed graph representing all grouping layouts. The highlight path represents a grouping layout corresponding to Fig. 3

1.8 Dual-objective optimization

General dynamic programming is often used to solve single-objective problems, like finding the shortest path in a graph. However, our problem has two objectives. To address this, we extend simple dynamic programming by storing a Pareto front (regarding entropy and interaction cost), rather than a single value, in each vertex (Fi,j,k). A Pareto front stores a set of best solutions so that no other path from F0,0,0 to Fi,j,k (corresponding to sub-grouping layouts within Ai,j,k) can better achieve either objective. Figure 5 illustrates the Pareto front of F10,10,10. Based on dynamic programming, the Pareto front of a vertex can be computed using the Pareto fronts of all immediate previous vertexes in the graph, until the Pareto front of F10,10,10 is computed. We can thus choose one solution from the Pareto front of F10,10,10 according to some appropriate criterion.
Fig. 5

Illustrating the Pareto front for the sink vertex (F10,10,10 for QWERTY). Solutions in the Pareto front are displayed as black points, while gray points represent those that are not Pareto efficient. The highlighted point is the solution finally chosen because it has a lowest cost to entropy ratio, and also falls within the tolerance of entropy

After deriving the Pareto front for the sink vertex, we have all the grouping keyboard layouts that are Pareto efficient regarding entropy and cost. Then, we can choose one solution from the Pareto front. Here, the input for this decision is the required entropy (H) that should be provided by this step. In our current implementation, we adopt a heuristic method. We first look at the range of 0.9H–1.1H to see if there are any solutions within this range. If there are, we choose the one with the lowest cost/entropy ratio, representing the most efficient solution. If not, we extend the range gradually (with a step of 0.1H) until we find a solution. Figure 5 illustrates this strategy.

1.9 Constraints on keyboard behavior

Sometimes, it is necessary to impose some constraints on the behavior of dynamic grouping keyboards, for example, to specify the minimal size or avoid grouping keys of irregular shape. Fortunately, our design of the optimization algorithm makes it easy to incorporate such constraints. To do so, one only needs to remove illegal edges from the graph before applying dynamic programming. For example, all edges representing a single key should be removed, as well as their involved paths. By constraining the behavior of the dynamic keyboard, the search space can also be reduced. The specific constraints imposed on the behavior of the dynamic grouping keyboard are determined by the interaction design and purpose.

1.9.1 Study 1: scanning keyboard

We applied DGK to a scanning keyboard (Ashtiani and MacKenzie 2010). We compared the performance of DGK with static grouping keyboards. Both simulated study and real-user study were conducted.

1.10 Study setup

Figure 6 shows the interface of DGK and a static grouping layout for scanning keyboards. A-Z letters were organized in alphabetical order on a one-dimensional interface. The system highlighted grouping keys in turn with a time period set to be 1000 ms (Ashtiani and MacKenzie 2010). Users input a letter or a group of letters by pressing a confirm button on an external physical keyboard when the grouping key that contained the target letter was highlighted. After a grouping key was selected, the scanning started again from the beginning. Once all letters of the intended word were input, users selected the Button # to switch to word selection mode. Words matching the input sequence were displayed in a candidate selection region ordered by word frequency. The keyboard provided last/next page buttons for users to navigate candidate words, and a Backspace button, with which users could delete the letter or the word that had been just inputted.
Fig. 6

The interface of DGK (above) vs. a static grouping layout (below) for scanning keyboards

In this case, we determined the objective of optimization as to minimize input time. For scanning keyboards, it was the number of periods required to input words. Therefore, we can compute the interaction cost like this: coding cost was defined as the position of the needed grouping key on the keyboard, and the scanning cost as the position of word in the candidate list.

1.11 Simulation

We simulated four vocabularies with 2000, 5000, 20,000 and 50,000 words respectively. The simulated performance was calculated as the average cost of all words in the vocabulary weighted by their frequency. According to the results, static grouping keyboards with less than 5 keys often generate too many candidates, which makes some low-frequency words difficult to be input. Therefore, we tested 5-Key and 7-Key keyboards in the simulation. As shown in Fig. 7, DGK was superior to other static grouping keyboards. Take a vocabulary of 5000 words for example. With DGK, it required 9.31 periods to input a word on average. In comparison, the costs for 5-Key and 7-Key layouts were 12.59 and 15.48 respectively. According to the results, we chose the 5-Key keyboard as a baseline when evaluating the performance of DGK for actual use.
Fig. 7

The simulated performance of the DGK and three static layouts

1.12 Participants

We recruited twelve able-bodied participant (8 males) aged from 20 to 34 (M = 21.75) to take this experiment. None of them had used a scanning keyboard before.

1.13 Design and procedure

We used a within-subject design. The only independent factor was Technique (DGK vs. the 5-Key keyboard). Six participants tested DGK first while the others tested the 5-Key keyboard first. Each participant input 10 phrases sampled from the MacKenzie and Soukroeff corpus (2003), which were the same for the two tested techniques. Before testing, participants familiarize themselves with interaction of scanning keyboards until they would not easily miss an intended key when it was highlighted. Usually, 5–10 words were enough for the practice. Between phrases, participants could rest for up to 30 s to avoid fatigue. After the experiment, we interviewed them about their subjective feedback. On average, the experiment took 30 min.

2 Result

2.1 Error rate

Error rate was computed as MSD, the minimum number of insertions, deletions or substitutions to transform the inputted word into the intended one. The mean MSD were 0.24% for DGK and 0.88% for the static keyboard. The difference was not significant (F1,11 = 1.13, p = 0.31).

2.2 Input time

The text entry rate was measured with WPM (Word Per Minute; We suppose a word to contain 5 letters on average). On average, it required 15.96 time periods to input a word with DGK, and 18.04 time periods with the 5-Key keyboard. Accordingly, the mean rate was 3.79 WPM for DGK and 3.26 WPM for the static keyboard. The actual performance of DGK was 16% faster than that of 5-Key keyboard. RM-ANOVA showed the difference was significant (F1,11 = 26.30, p < 0.001).

2.3 Subjective user feedback

Figure 8 shows the subjective feedback of participants. RM-ANOVA found significant effects of Technique on perceived speed (F1,11 = 22.40, p < 0.001), accuracy (F1,11 = 5.50, p < 0.05), learnability (F1,11 = 11.96, p < 0.01) and preference (F1,11 = 49.00, p < 0.0001). In spite of being relatively less accurate and more difficult to learn than the static 5-Key keyboard, DGK offered a faster text entry rate and less fatigue. As a result, participants preferred DGK (M = 3.75, SD = 0.45) to the 5-Key keyboard (M = 2.58, SD = 0.51) in overall.
Fig. 8

Subjective user feedback for scanning keyboard. The scores are from Not Good (1) to Good (5)

2.4 Study 2: mouse input

In this study, we adapted DGK to a cursor-based QWERTY keyboard. We used two pointing devices: a joystick with its speed tuned down to emulate the difficulty of users with motor impairment to control, and a trackball that users control with their chins. We also compared DGK to static grouping keyboards, and conducted both simulation and real-user studies.

2.5 Study setup

Figure 9 shows the interface of DGK and 5-key keyboard in this experiment. To input a letter, users first moved the cursor within the grouping key containing that letter, and then clicked a confirmation button (on the joystick controller or the trackball controller). After all letters of the intended word was entered, users pressed a selection button to switch to word selection mode. Pushing the joystick or swiping the trackball would advance the selection to next word. Deletion was done with another button on the controller. To simulate the difficulty of motion control, we slowed down the cursor movement speed by restrain the speed of the cursor no faster than 1 key per second.
Fig. 9

Dynamic Grouping Keyboard (above) vs. 5-key ambiguous keyboard (below) for cursor input. The white dots are the clicking point of each letter

In this case, the objective of optimization was to minimize the distance of cursor movement. We defined the coding cost as the distance of cursor traveled from the last key to the next key. In particular, to calculate moving distance, we assumed users would move the cursor to the center of the nearest letter in grouping key that was needed (which contained the intended letter). We computed the scanning cost based on the position of the intended word displayed in the candidate list. We converted the time of selecting a word into the distance of cursor movement according to the selection time and the speed of the cursor tested in a pilot study.

2.6 Simulation

Used the same simulation method in Study 1, we again found DGK was superior to the other two static grouping keyboards. The results are shown in Fig. 10. Take the vocabulary of 5000 word for example. To input a word, users on average moved 2.54, 9.39 and 12.19 key-width with DGK, 5-Key and 10-Key keyboards respectively. The relative advantage of DGK was even greater than that found in Study 1. This is probably because the two-dimensional space created a large space for utilizing the optimization power of DGK. Accordingly, we chose 5-Key keyboard to compare in our real-user study.
Fig. 10

The simulated performance of the Dynamic Grouping Keyboard vs. static layouts

2.7 Participants

We recruited 12 able-bodied participants (9 males) to take part in the study. Participants were aged from 20 to 25 (M = 21.42), all of them familiar to the QWERTY layout (M = 4.58).

2.8 Design and procedure

We studied each input device independently. We did not intend to compare the difference between input devices. The 12 participants were divided into two groups. The first group tested only the joystick while the second group tested only the trackball. Each participant entered text on both DGK and the 5-Key keyboard, with their presentation order counterbalanced. The only independent factor was Technique (DGK vs. the 5-Key keyboard).

Before testing, participants practiced 1-2 phrases to familiarize them with the interaction. They then entered the same five phrases with DGK and the 5-Key keyboard respectively. The tested phrases were randomly sampled from the MacKenzie and Soukroeff corpus (2003), and were the same for the two tested techniques.

3 Result

3.1 Error rate

With a trackball, the mean MSD were 0% for DGK and 0.22% for the static keyboard. With a joystick, the mean MSD were 0.38% for DGK and 0.10% for the static keyboard. RM-ANOVA showed for each input device, the difference was not significant (F1,5 = 2.50, p = 0.17; F1,5 = 2.16, p = 0.20).

3.2 Text-entry rate

With a trackball, the mean text entry rate was 4.76 WPM (SD = 1.00) for DGK and 3.83 WPM (SD = 0.94) for the static keyboard. With a joystick, the mean rate was 7.03 WPM (SD = 2.27) for DGK and 4.11 WPM (SD = 1.14) for the static keyboard. RM-ANOVA showed for each input device, the difference significantly higher than the other (F1,5 = 10.30, p < 0.05; F1,5 = 15.91, p < 0.05).

3.3 Subjective user feedback

Figure 11 shows the subjective feedback of participants for inputting with the trackball and the joystick respectively. RM-ANOVA showed for both input devices, there were significant effects of Techniques on perceived speed, accuracy, mental fatigue, physical fatigue, learnability and preference (for all, p < 0.05). The results were quite consistent with the result in Study 1. Again, participants found DGK was relatively less accurate and more difficult to learn than the static 5-Key keyboard, but they preferred DGK for its faster text entry rate and less fatigue.
Fig. 11

Subjective user feedback of input with the trackball (above) and the joystick (below)

4 Discussion and conclusion

In this paper, we describe a novel approach for software keyboard optimization, the Dynamic Grouping Keyboard, which dynamically adapts the grouping of individual keys, without altering key location on the keyboard. This feature lets users can leverage their familiarity with the location of keys to fast seek intended letters, while at the same time provides the flexibility and benefit of dynamic keyboards.

The unique paradigm of the Dynamic Grouping Keyboard also imposes a question on how to control the behavior of adaptation. We identified the challenge, and proposed a novel two-step algorithm: Both the estimation of information entropy and the dynamic programming with dual objectives are for the first time applied in literature. Meanwhile, our algorithm provides a new perspective for keyboard optimization: The optimization is to optimize the allocation of input resource on the keyboard (e.g. cursor moving distance, key size, and so on) to letters, in order to provide the necessary information for specifying text or words. We hope our work can push the limits of keyboard optimization algorithm, and inspire future research.

Another benefit of the Dynamic Grouping Keyboard is that it supports dynamic change of the language model, such as deleting/adding words, adjusting word frequency or even replacing the entire lexicon. As its optimization is done in real time, it will not suffer from the performance issue. Such a feature might be useful for allowing the keyboard to adapt to different usage context. For example, each application (e.g. office email, chatting, shopping, and watching TV programs) may have a separate language model. In contrast, for the static keyboard, dynamic change of language model is impossible, because every time the language model is changed, usually a brute-force algorithm has to be run to derive an optimal layout.

A limitation of the two studies was we experimented DGK with able-bodied users, rather than users with motor impairment. In Study 2, we tuned down the display/control ratio of the pointing devices to simulate the difficulty of motor control. In despite of this limitation, our two studies indeed gave insights and demonstrated the advantage of DGK. Moreover, our derived text entry rates were 3-7 WPM, which overlapped the range of text entry rates found for motor impairment in literature (Ashtiani and MacKenzie 2010; Merlin and Raynal 2010; Song 2010; Wobbrock and Myers 2006). This partially supported the validity of the constraints we imposed on the device.

Another issue we did not research in depth was the cost of dynamic grouping—users had to visually confirm new grouping keys before moving the pointing device. This definitely prevents users from pre-programming the input actions. For example, on a static keyboard, expert users can pre-program the movement to acquire the next key even before the current key has been selected. From the perspective of users, however, using DGK allowed them to wait for the intended key to be more accessible before actually moving. Moreover, according to users’ subjective feedback, they could get used to the adaptation, and perform simultaneous visual search and input action with practice. In the future, it deserves to quantify the cost of adaptation, and examine the trade-off between the cost and the theoretical benefit in a systematical way.

Finally, compared to the generality of DGK, the range of keyboard layouts, input devices and optimization objectives investigated in our studies was still limited. Meanwhile, we deem DGK also has the potential to facilitate ordinary users when the input is not constrained by users but by the input device or environment, such as in mobile or wearable interaction. In the future, we plan to investigate the Dynamic Grouping Keyboard in various interaction platforms and scenarios, to obtain a more complete understanding of its performance and usability issue.

Notes

Acknowledgements

The authors thank all participants. This work is supported by the National Key Research and Development Plan under Grant No. 2016YFB1001200, the Natural Science Foundation of China under Grant No. 61672314 and No. 61572276, and also by Beijing Key Lab of Networked Multimedia.

References

  1. Ashtiani, B., MacKenzie, I.S.: BlinkWrite2: an improved text entry method using eye blinks. In: Proceedings of the 2010 Symposium on Eye-Tracking Research & Applications (ETRA ‘10), 339–345 (2010). http://doi.org/10.1145/1743666.1743742
  2. Aulagner, G., François, R., Martin, B., Michel, D., Raynal, M.: Floodkey: increasing software keyboard keys by reducing needless ones without occultation. In: Proceedings of the 10th WSEAS international conference on Applied computer science (ACS’10, pp.), Hamido Fujita and Jun Sasaki (Eds.). World Scientific and Engineering Academy and Society (WSEAS), Stevens Point, Wisconsin, USA, 412–417 (2010)Google Scholar
  3. Bellman, T., MacKenzie I.S.: A probabilistic character layout strategy for mobile text entry. In: Proceedings of Graphics Interface ‘98. Canadian Information Processing Society, Toronto, Canada, 168–176 (1998)Google Scholar
  4. Bi, X., Ouyang, T., Zhai, S.: Both complete and correct?: multi-objective optimization of touchscreen keyboard. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ‘14), 2297–2306 (2014). http://doi.org/10.1145/2556288.2557414
  5. Bi, X., Smith, B.A., Zhai, S.: Quasi-qwerty soft keyboard optimization. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ‘10), 283–286 (2010). http://doi.org/10.1145/1753326.1753367
  6. Bi, X., Smith, B.A., Zhai, S.: Multilingual touchscreen keyboard design and optimization. Hum.-Comput. Interact 27(4), 352–382 (2012)Google Scholar
  7. Dunlop, M., Levine, J.: Multidimensional pareto optimization of touchscreen keyboards for speed, familiarity and improved spell checking. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ‘12), 2669–2678 (2012). http://doi.org/10.1145/2207676.2208659
  8. Dunlop, M.D., Durga, N., Motaparti, S., Dona, P., Medapuram, V.: QWERTH: an optimized semi-ambiguous keyboard design. In: Proceedings of the 14th international conference on Human-computer interaction with mobile devices and services companion (MobileHCI ‘12), 23–28 (2012). http://doi.org/10.1145/2371664.2371671
  9. Faraj, Khaldoun Al, Mojahid, Mustapha, Vigouroux, Nadine: BigKey: A Virtual Keyboard for Mobile Devices. In: Ambient, Ubiquitous, Interaction, Intelligent, Jacko, Julie A. (eds.) Human-Computer Interaction, pp. 452–459. Springer, Berlin Heidelberg (2009)Google Scholar
  10. Gargi, U., Gossweiler, R.: QuickSuggest: character prediction on web appliances. In: Proceedings of the 19th international conference on World wide web (WWW ‘10), 1249–1252 (2010). http://dx.doi.org/10.1145/1772690.1772894
  11. Gong, J.: Semantic & syntactic context-aware text entry methods. In: Proceedings of the 9th international ACM SIGACCESS conference on Computers and accessibility (Assets ‘07), 261–262 (2007). http://doi.org/10.1145/1296843.1296903
  12. Green, N., Kruger, J., Faldu, C., St. Amant, R.: A reduced QWERTY keyboard for mobile text entry. In; CHI ‘04 Extended Abstracts on Human Factors in Computing Systems (CHI EA ‘04). ACM, New York, NY, USA, 1429–1432 (2004). http://doi.org/10.1145/985921.986082
  13. Gunawardana, A., Paek, T., Meek, C.: Usability guided key-target resizing for soft keyboards. In: Proceedings of the 15th international conference on Intelligent user interfaces (IUI ‘10), 111–118 (2010). http://doi.org/10.1145/1719970.1719986
  14. Harbusch, K., Hasan, S., Hoffmann, H., Kühn, M., Schüler, B.: Domain-specific disambiguation for typing with ambiguous keyboards. In: Proceedings of the 2003 EACL Workshop on Language Modeling for Text Entry Methods (TextEntry ‘03), 67–74 (2003)Google Scholar
  15. Hasan, S., Harbusch, K.: N-Best hidden Markov model super tagging to improve typing on an ambiguous keyboard. In: Proceedings of Seventh International Workshop on Tree Adjoining Grammar and Related Formalisms: 24–31 (2003)Google Scholar
  16. Karrenbauer, A., Oulasvirta, A.: Improvements to keyboard optimization with integer programming. In: Proceedings of the 27th annual ACM symposium on User interface software and technology (UIST ‘14), 621–626 (2014). http://doi.org/10.1145/2642918.2647382
  17. Komninos, A., Dunlop, M.M.: Text input on a smart watch. IEEE Pervasive Comput. 13(4), 50–58 (2014)CrossRefGoogle Scholar
  18. Kühn, M., Garbe, J., Informatik, I.: Predictive and highly ambiguous typing for severely speech and motion impaired users. In: Proceedings of the UAHCI 2001 conference on Universal Access in Human Computer Interaction, New Orleans, Louisiana (2001)Google Scholar
  19. Kushler, C.: AAC: Using a Reduced Keyboard. Eric.ed.gov (1998)Google Scholar
  20. Lesher, Gregory W., Moulton, Bryan J., Jeffery Higginbotham, D.: Optimal character arrangements for ambiguous keyboards. IEEE Trans. Rehabil. Eng. 6(4), 415–423 (1998)CrossRefGoogle Scholar
  21. Li, F.C.Y., Guy, R.T., Yatani, K., Truong, K.N.: The 1line keyboard: a QWERTY layout in a single line. In: Proceedings of the 24th annual ACM symposium on User interface software and technology (UIST ‘11), 461–470 (2011). http://doi.org/10.1145/2047196.2047257
  22. Mackenzie, S., Felzer, T.: SAK: Scanning ambiguous keyboard for efficient one-key text entry. In: ACM Trans. Comput.-Hum. Interact. 17, 3, Article 11 (July 2010), 39 pages. http://doi.org/10.1145/1806923.1806925
  23. MacKenzie, S., Soukoreff, R.W.: Phrase sets for evaluating text entry techniques. In: CHI ‘03 Extended Abstracts on Human Factors in Computing Systems (CHI EA ‘03), 754–755 (2003). http://doi.org/10.1145/765891.765971
  24. Mathieu Raynal, I., MacKenzie, Scott, Merlin, Bruno: Semantic keyboard: fast movements between keys of a soft keyboard. In: Miesenberger, Klaus, Fels, Deborah, Archambault, Dominique, Peňáz, Petr, Zagler, Wolfgang (eds.) Computers Helping People with Special Needs, pp. 195–202. Springer, Berlin Heidelberg (2014)CrossRefGoogle Scholar
  25. Merlin, Bruno, Raynal, Mathieu: Evaluation of spreadkey system with motor impaired users. In: Miesenberger, Klaus, Klaus, Joachim, Zagler, Wolfgang, Karshmer, Arthur (eds.) Computers Helping People with Special Needs, pp. 112–119. Springer, Berlin Heidelberg (2010)CrossRefGoogle Scholar
  26. Oulasvirta, A., et al. Improving two-thumb text entry on touchscreen devices. In: Proceedings of. CHI, ACM (2013), 2765–2774. http://doi.org/10.1145/2470654.2481383
  27. Raynal, M., Truillet, P.: Fisheye Keyboard: Whole Keyboard Displayed on PDA. In: Human-Computer Interaction. Interaction Platforms and Techniques, Julie A. Jacko (ed). Springer, Berlin Heidelberg, 452–459 (2007)Google Scholar
  28. Raynal, M.: KeyGlasses: semi-transparent keys on soft keyboard. In: Proceedings of the 16th international ACM SIGACCESS conference on Computers & accessibility (ASSETS ‘14), 347–349 (2014). http://doi.org/10.1145/2661334.2661427
  29. Schadle, Igor: Sibyl: AAC system using NLP techniques. In: Miesenberger, Klaus, Klaus, Joachim, Zagler, Wolfgang L., Burger, Dominique (eds.) Computers Helping People with Special Needs, pp. 1009–1015. Springer, Berlin Heidelberg (2004)CrossRefGoogle Scholar
  30. Song, Y.C.: Joystick text entry with word prediction for people with motor impairments. In: Proceedings of the 12th international ACM SIGACCESS conference on Computers and accessibility (ASSETS ‘10), 321–322 (2010). http://doi.org/10.1145/1878803.1878892
  31. Wobbrock, J., Myers, B.: Trackball text entry for people with motor impairments. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ‘06), 479–488 (2006). http://dx.doi.org/10.1145/1124772.1124845
  32. Wobbrock, J.O., Myers, B.A., Kembel, J.A.: EdgeWrite: a stylus-based text entry method designed for high accuracy and stability of motion. In: Proceedings of the 16th annual ACM symposium on User interface software and technology (UIST ‘03), 61–70 (2003). http://doi.org/10.1145/964696.964703
  33. Huang, Y.-C., Wu, F.-G.: NineType: An Ergonomic Ambiguous Keyboard DesignGoogle Scholar
  34. Zhai, S., Hunter, M., Smith, B.A.: The metropolis keyboard—an exploration of quantitative techniques for virtual keyboard design. In: Proceedings of the 13th annual ACM symposium on User interface software and technology (UIST ‘00), 119–128 (2000). http://doi.org/10.1145/354401.354424
  35. Zhai, Shumin, Hunter, Michael, Smith, Barton A.: Performance optimization of virtual keyboards. Human-Computer Interaction 17(2–3), 229–269 (2002).  https://doi.org/10.1145/1866029.1866043 CrossRefGoogle Scholar

Copyright information

© China Computer Federation (CCF) 2019

Authors and Affiliations

  1. 1.Tsinghua National Laboratory for Information Science and Technology, Department of Computer Science and TechnologyTsinghua UniversityBeijingChina

Personalised recommendations