Abstract
With interactive displays, such as touchscreens, becoming an integrated part of the modern vehicle environment, predictive displays have emerged as a solution to minimize the effort as well as cognitive, visual and physical workload associated with using in-vehicle displays. It utilises gesture tracking in 3D as the basis of an input modality enabling interface component acquisition (pointing and selections). Nevertheless, the predictive display technology has the potential to facilitate and assist human computer interaction for motion impaired users, for example, those with cerebral palsy, tremors and spasms, in various scenarios. It also has a wider application in inclusive design addressing general ranges of impairments, such as those arising from ageing. This paper explores the potential of this promising technology and proposes that a predictive display, which was developed to aid drivers in a situationally induced impairment due to using non-driving interfaces in a moving car, can be applicable to the health induced impairment arising from perturbations due to physical movement disorders. It is concluded that 3D predictive gesture tracking can simplify and expedite target acquisition during perturbed pointing movements due to a health/physical-capability impairment.
Keywords
1 Introduction and Background
Interactive displays, such as touchscreens, are becoming increasingly prevalent in the modern vehicle environment, progressively replacing traditional in-vehicle mechanical controls such as switches, knobs and buttons [1, 2]. This is due to their ability to present large quantities of information related to in-vehicle infotainment systems, facilitate intuitive interactions via free hand pointing gestures (particularly for novice users) and offer additional design flexibilities (for example, the display can be adapted to the context of use via a reconfigurable Graphical User Interface GUI) [1–4]. However, undertaking a free hand pointing gesture to acquire (pointing and select) a target on the display, e.g. a GUI icon, requires dedicating a considerable amount of attention (visual, cognitive and physical) that would be otherwise available for driving [4], with potential safety implications [5]. Due to road and driving conditions, the user pointing gesture can be subject to high levels of perturbations leading to erroneous on-screen selections [6]; attempts to rectify an incorrect selection or adapting to the noisy environment can lead to even more distractions, i.e. Situationally Induced Impairment and Disability (SIID). Therefore, intent-aware displays [7], which can infer, notably early in the free hand pointing gesture, the intended on-screen item can simplify and expedite the selection task (even under perturbations). They can significantly improve the usability of in-car touchscreens by reducing distractions and workload associated with interacting with them.
Additionally, with the proliferation of the increasingly ubiquitous touchscreen technology in everyday use, target acquisition (pointing and selection) on a graphical user interface has become part of modern life and a frequent Human-Computer Interaction (HCI) task. Pointing reliability and accuracy is of a key importance for the design of effective GUI. This has triggered an immense interest in approaches that model pointing movements and assist the pointing task by reducing the cursor pointing time and improving its accuracy [8–22]. This can be achieved via pointing facilitation techniques, such as increasing the size of the target icon, altering its activation area, dragging the cursor closer to the target, etc. However, such strategies can be effectively applied only if the intended GUI item is known a priori [11–17]. Such studies focus on pointing via a mouse or mechanical-device in a 2D set-up to select a GUI icon(s) and often focus on able-bodied computer users in a stationary input situation. However, the pointing-selection task can be particularly challenging or even overwhelming at times for users with a motion-visual impairment [17–22], i.e. due to Health Induced Impairment and Disability (HIID). For example, in [18] a method that is based on an advanced state-space particle filter technique is used to smooth the 2D pointing mouse cursor trajectory such that it compensates for HIID-related-perturbation leaving the cursor to move only in the intended direction.
On the other hand, inclusive design is a user-centered approach that examines designed product features with particular attention to the functional demands they make on the perceptual, thinking and physical capabilities of diverse users, including those with impairments and ageing. Inclusion refers to the quantitative relationship between the demand made by design features and the capability ranges of users who may be excluded from the use of the product because of those features [25]. Therefore, formulating solutions that facilitate HCI for people with a wide range of HIID, including those that arise from age and not only severe forms of physical disability, is crucial. Most importantly, an inclusive design approach extends beyond the scope of conventional usability methods as it must accommodate extremes of capability range or situational contexts of task or stress, that are not normally accommodated by product design. A predictive display presents itself in this context as a means to extend the usability of the interactive displays to a diverse population of users, for example motion impaired or able-bodied users, elderly or young users, expert or non-expert users as well as situationally impaired users.
The transferability of HCI solutions for HIID to SIID scenarios (and vice versa), was proposed in [22, 23]. It assumes that any human user can be impaired (disabled) in their effectiveness by characteristics of their environment, the task and the design of the GUI. Such impairment may take the form of perceptual, cognitive and physical movement functional limitations, which translate into inability [18, 25]. For instance, attempting to enter text on an in-car touchscreen (e.g. for navigation) whilst driving in an off-road environment presents difficulties in perceiving the interface for multiple tasks (seeing on-screen icons, outside driving environment and vehicle controls), performing the attentional tasks necessary for safe driving (track/correct vehicle movement, maintaining car controls as well as monitor/correct semantic an texting task), and carrying out the required physical movements (pointing, pressing, steering, braking, etc.).
Figures 1 and 2 depict 2D and 3D pointing trajectories for several on-screen selection tasks, respectively. In the former, several mouse cursor trajectories pertaining to two users (one suffers from severe motor impairment) carrying out a number of target acquisitions on a computer screen using a mechanical mouse. Figure 2 displays the 3D pointing gesture track recorded by a gesture-tracker (namely Leap Motion Controller) whilst a user interacts with a touchscreen in a car under different road/driving conditions. This clearly demonstrates the similarities between perturbations in the pointing movement due to situational (especially when a car is driven on a harsh terrain) and health induced impairments. Thus, solutions devised for predictive in-vehicle displays can be applied to tackle HIID perturbations in the pointing movement.
The developed predictive display for automotive applications utilises a gesture tracker, which captures, in real-time, the pointing hand/finger location(s), in conjunction with probabilistic inference algorithms to determine the intended destination on the interactive surface (e.g. touchscreen). The prediction results for each of the GUI selectable icons are subsequently used to decide on the intended endpoint and accordingly alter the GUI to assist the selection process. Several such pointing gesture trackers, which can accurately track, in real-time, a pointing gesture in 3D, have emerged lately, e.g. Microsoft Kinect, leap motion controller, and Nimble UX. They are motivated by a desire to extend HCI beyond traditional keyboard input and mouse pointing. The Bayesian destination predictor applied here relies on defining a Hidden Markov Model (HMM) of the pointing motion in 3D, effectively capturing the influence of the intended endpoint on the finger/hand movement [7]. This is distinct from previous HCI research on endpoint prediction in 2D scenarios, e.g. [11–15], which often follow from Fitt’s law type analysis and uses a static setting/model. The Bayesian HMM approach permits capturing the variability among users as well as the noise of the movement tracking sensor via Stochastic Differential Equations (SDE) that represent the destination-motivated pointing motion in 3D.
The remainder of this paper is organised as follows. In the next section, we describe the adopted Bayesian intent inference framework used in the predictive displays and outline the flexibility of this formulation. Pilot results from the automotive domain are shown in Sect. 3. Finally, the applicability of predictive displays in inclusive design is discussed in Sect. 4 and conclusions are drawn.
2 Bayesian Intent Inference with Hidden Markov Models
Bayesian inference with HMM allows the flexible modelling of the pointing motion with either HIID or SIID via a stochastic differential equation. The variability in the pointing movement, e.g. due to the user behavior and/or impairment, can be introduced through the noise element of the state (position, velocity, acceleration, etc.) evolution equation. Additionally, the noise generated from the employed sensor, e.g. a particular gesture tracker, can be incorporated via the measurement noise in the observation equation. Most importantly, the statistical filter utilised to infer the state or intent/final destination of the tracked object (e.g. mouse cursor in 2D or pointing finger for free hand pointing gestures) can be applied to the same class of motion models (e.g. Kalman filtering for linear models) despite changing the adopted pointing movement process/model. The effectiveness of the state-space-modelling for removing unintentional impairment-related pointing movement were demonstrated in [17, 18, 26]. Nevertheless, the main objective of employing HMM in predictive displays is to determine the icon the user intends to select on the display as early as possible; removing unintentional HIID/SIID-related pointing movement, although desirable, is not essential.
2.1 Destination Motion Models
Since the pointing motion is intrinsically driven by the endpoint (i.e., the intended on-screen icon), destination-reverting models such as the linear Mean Reverting Diffusion (MRD) and Equilibrium Reverting Velocity (ERV) models can be suitable for predictive displays under health or situationally induced impairments. Following the integration of their respective SDEs and assuming that the intended destination is \( {\mathcal{D}}_{i} \), linear destination reverting models can be expressed by
where \( {\mathbf{s}}_{i,k - 1} \) and \( {\mathbf{s}}_{i,k} \) are the hidden model state vectors at two consecutive time instants \( t_{k - 1} \) and \( t_{k} \). For example, the state \( {\mathbf{s}}_{i,k} \) can include the true pointing-finger location in 3D and other higher order motion dynamics such as velocity as in the ERV case. Matrix \( {\mathbf{F}}_{i,k} \) is the state transition and \( {\varvec{\upkappa}}_{i,k} \) is a time varying constant (both are with respect to the \( i^{th} \) destination \( {\mathcal{D}}_{i} \)), and the motion model dynamic noise is \( {\mathbf{w}}_{k} \). Therefore, for \( N \) possible endpoints on the display (i.e. selectable GUI icons), \( N \) such models can be constructed. The (also linear) observation model is given by
where \( {\mathbf{n}}_{k} \) represents the noise introduced by the sensor. For more details on the destination reverting models and their characteristics, the reader is referred to [7, 27].
To demonstrate the ability of the destination reverting models to capture a wide range of possible pointing behaviors in 3D, Fig. 3 depicts several possible velocity profiles of the pointing finger (during target acquisition tasks via free hand pointing gestures) as per the ERV motion model. Each of these plots is obtained by setting a different value for the damping parameter of the ERV model along the \( x \), \( y \) and \( z \) axes via \( {\mathbf{F}}_{\varvec{k}} \) in (1). The figure clearly illustrates that by using ERV, a range of possible pointing velocity profiles can be modelled, for example, reflecting the motor-ability and/or reach of a user interacting with a touchscreen positioned at a considerable distance from the seating positions such as with an in-vehicle interactive display for controlling the infotainment system. It is noted that the bottom plot in Fig. 3 resembles the expected velocity profile of the point gestures of an in-car touchscreen user without any impeding HIID or SIID.
2.2 Intent Inference
Predictive displays aim to establish, in real-time, the likelihood of each of the selectable icons of the displayed GUI being the intended destination of the undertaken pointing task (e.g. of a pointing gesture). For example, at time instant \( t_{k} \) where the available pointing object (finger/cursor) observations (positions) are \( {\mathbf{m}}_{1:k} = \left\{ {{\mathbf{m}}_{1} ,{\mathbf{m}}_{2} , \ldots ,{\mathbf{m}}_{k} } \right\} \), the system calculates
The intended destination, which is unknown a priori, is notated by \( {\mathcal{D}}_{I} \) such that \( {\mathcal{D}}_{I} \in {\mathbb{D}} = \left\{ {{\mathcal{D}}_{1} ,{\mathcal{D}}_{2} , \ldots ,{\mathcal{D}}_{N} } \right\} \). It is noted that the location of the items in \( {\mathbb{D}} \) are known, however, no assumptions are made on their distribution-layout. After evaluating \( {\mathcal{P}}\left( {t_{k} } \right) \) in (1), a simple intuitive approach to establish the intended destination at \( t_{k} \) is to select the most probable endpoint via
.
Decision criterion other than (4) can be applied. For the linear destination reverting models, Kalman filters can be used (one per nominal destination) to calculate \( P\left( {{\mathcal{D}}_{i} = {\mathcal{D}}_{I} |{\mathbf{m}}_{1:k} } \right) \) in (3) as per [7, 27]. Adopting nonlinear motion or observation models can lead to advanced statistical inference methods such as sequential Monte Carlo or other related methods [28] being required for online filtering.
2.3 Smoothing Noisy Trajectories
The results of the \( N \) statistical filters applied to determine (3) can be utilised to remove the unintentional perturbations-impairment-related movements as shown in [7]. However, in certain scenarios (e.g. infrequent severe perturbations) where it is desirable to maintain a simple linear motion model for the intent inference functionality, a pre-processing step/stage can be added such that the raw pointing data is filtered, e.g. using a particle filter [18, 26]. The filtered track is subsequently used by the destination inference module.
3 Pilot Results
Figure 4 depicts selected pilot results of using a predictive display in an automotive context. The benefits are assessed in terms of the technology ability to reduce the effort/workload associated with interacting with an in-vehicle display. In this scenario, a gesture tracker is employed to produce, in real-time, the locations of the pointing hand/finger in 3D, which are then utilised by the intent predictor. Here, the predictive display auto-selects the intended on-screen icon once a particular level of inference certainty is achieved (the user need not touch the touchscreen surface to make a selection). This figure shows the measured subjective workload using NASA TLX forms when the prediction and auto-selection capability is on and off. In the latter case, the experiment becomes a conventional task of interacting with a touchscreen where completing a selection operation entails physically touching the intended on-screen icon. Figure 4 illustrates that the predictive display system can reduce the workload of interacting with an in-vehicle display by nearly 50 %, therefore, significantly simplifying and facilitating the on-screen target acquisition task via free hand pointing gestures.
Figures 5 and 6 demonstrate the ability of a sequential Monte Carlo method, namely the variable rate particle filter, to remove highly non-linear perturbation-related pointing movements when interacting with a touchscreen via free hand pointing gestures or selecting icons of a GUI displayed on a computer screen using a mechanical mouse, respectively. The raw cursor movement data in Fig. 6 is for a user that suffers from cerebral palsy. The figure exhibits the confidence ellipses obtained from the sequential Monte Carlo filter, which has visibly removed the health-induced-impairment-related jumping behavior of the mouse cursor position and can assist identifying the user’s intended destination (despite the ambiguity of the raw pointing data). On the other hand, unintentional situational-induced-impairment-related pointing finger movements are successfully removed in Fig. 6.
4 Conclusions and Future Work
Consideration of the effectiveness of measures intended for situational impairment, such as when using a touchscreen in a moving vehicle over a badly maintained road (a perturbed environment) have shown that probabilistic predictors can bring significant gains in SIID. This strongly suggests that similar gains can be achieved in a health induced impairment scenario. That is to say that spasm, weakness, tremor and athetosis may be mitigated or eliminated by the predictive approach based on automotive algorithms/applications we describe. In particular, motion impaired users, who may have difficulty pointing and selecting on touchscreens will benefit not only from prediction and automated selection (i.e. auto-selection as in Fig. 4), but also from the reduction of workload reported by the automotive trial participants, reliably measured using NASA TLX scores.
Additionally, from an inclusive design perspective [29] the predictive display technology may greatly benefit those with age related or mild physical or perceptual impairments by enhancing performance in pointing-selection and reducing the associated workload. Mild functional impairments such as physical, visual, hearing reach and stretch and cognitive may be accommodated. The adopted predictive techniques are also applicable to special purpose designs for more extreme impairment and disability. Experimental studies will initially require the same tasks, modified for floor effects from physically impaired participants. However, these will be superseded by trials of the same algorithms and detection technologies with interfaces in mobile displays, walking scenarios, wheelchair use and on public transportation. Finally, it is noted that predictive displays are particularly flexible in terms of incorporating additional sensory data or input modalities when available, e.g. eye-gaze or voice-based commands, via the Bayesian framework described in Sect. 2.
References
Burnett, G.E., Porter, J.M.: Ubiquitous computing within cars: designing controls for non-visual use. Int. J. Hum. Comput. Stud. 55(4), 521–531 (2001)
Harvey, C., Stanton, N.A.: Usability Evaluation for In-vehicle Systems. CRC Press, Boca Raton (2013)
Burnett, G.E., Lawson, G., Millen, L., Pickering, C.: Designing touchpad user-interfaces for vehicles: which tasks are most suitable? Behav. Inf. Technol. 30, 403–414 (2011)
Jaeger, M.G., Skov, M.B., Thomassen, N.G., et al.: You can touch, but you can’t look: interacting with in-vehicle systems. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 1139–1148 (2008)
Klauer, S.G., Dingus, T.A., Neale, V.L., Sudweeks, J.D., Ramsey, D.J.: The impact of driver inattention on near-crash/crash risk: an analysis using the 100-car naturalistic driving study data. National Highway Traffic Safety Administration, DOT HS 810 5942006 (2006)
Ahmad, B.I., Langdon, P.M., Godsill, S.J., Hardy, R., Skrypchuk, L., Donkor, R.: Touchscreen usability and input performance in vehicles under different road conditions: an evaluative study. In: Proceeding of the 7th International Conference on Automotive User Interfaces and Interactive Vehicular Applications (AutomotiveUI 2015), Nottingham, UK, pp. 47–54 (2015)
Ahmad, B.I., Murphy, J., Langdon, P.M., Godsill, S.J., Hardy, R., Skrypchuk, L.: Intent inference for pointing gesture based interactions in vehicles. IEEE Trans. Cybern. 46(4), 878–889 (2015)
Mackenzie, S.: Fitts’ law as a research and design tool in human-computer interaction. J. Hum. Comput. Interact. 7, 91–139 (1992)
Kopper, R., Bowman, D.A., Silva, M.G., McMahan, R.P.: A human motor behavior model for distal pointing tasks. Int. J. Hum Comput Stud. 68, 603–615 (2010)
Meyer, D.E., Smith, J.E., Kornblum, S., Abrams, R.A., Wright, C.E.: Optimality in human motor performance: ideal control of rapid aimed movements. Psychol. Rev. 8, 340–370 (1988)
McGuffin, M.J., Balakrishnan, R.: Fitts’ law and expanding targets: experimental studies and designs for user interfaces. ACM Trans. Comput. Hum. Interact. 12(4), 388–422 (2005)
Murata, A.: Improvement of pointing time by predicting targets in pointing with a PC mouse. IJHCI 10(1), 23–32 (1998)
Wobbrock, J.O., Fogarty, J., Liu, S., Kimuro, S., Harada, S.: The angle mouse: target-agnostic dynamic gain adjustment based on angular deviation. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 1401–1410 (2009)
Asano, T., Sharlin, E., Kitamura, Y., Takashima, K., Kishino, F.: Predictive interaction using the Delphian desktop. In: Proceedings of the ACM Symposium on User Interface Software and Technology. ACM, pp. 133–141 (2005)
Lank, E., Cheng, Y.-C.N., Ruiz, J.: Endpoint prediction using motion kinematics. In: Proceedings of the SIGCHI Conference on Human factors in Computing Systems, pp. 637–646 (2007)
Ziebart, B., Dey, A., Bagnell, J.A.: Probabilistic pointing target prediction via inverse optimal control. In: Proceedings of the ACM International Conference on Intelligent User Interfaces, pp. 1–10 (2012)
Ahmad, B.I., Langdon, P.M., Bunch, P., Godsill, S.J.: Probabilistic intentionality prediction for target selection based on partial cursor tracks. In: Stephanidis, C., Antona, M. (eds.) UAHCI 2014, Part III. LNCS, vol. 8515, pp. 427–438. Springer, Heidelberg (2014)
Langdon, P., Godsill, S., Clarkson, P.J.: Statistical estimation of user’s interactions from motion impaired cursor use data. In: Proceedings of the 6th International Conference on Disability, Virtual Reality and Associated Technologies (ICDVRAT 2006), Esbjerg, Denmark (2006)
Keates, S., Hwang, F., Langdon, P., Clarkson, P.J, Robinson, P.: Cursor measures for motion-impaired computer users. In: Proceedings of the Fifth International ACM Conference on Assistive Technologies – ASSETS, New York, pp. 135–142 (2002)
Wobbrock, J.O., Gajos, K.Z.: Goal crossing with mice and trackballs for people with motor impairments: performance, submovements, and design directions. ACM Trans. Accessible Comput. (TACCESS) 1(4), 1–37 (2008)
Gajos, K.Z., Wobbrock, J.O., Weld, D.S.: Automatically generating user interfaces adapted to users’ motor and vision capabilities. In: ACM symposium on User interface software and technology, pp. 231–240 (2007)
Biswas, P., Langdon, P.: Developing multimodal adaptation algorithm for mobility impaired users by evaluating their hand strength. Int. J. Hum. Comput. Interact. 28, 576–596 (2012)
Sears, A., Lin, M., Jacko, J., Xiao, Y.: When computers fade: pervasive computing and situationally induced impairments and disabilities. In: Proceedings of HCI International, pp. 1298–1302 (2003)
Newell, A.F., Gregor, P.: Human computer interfaces for people with disabilities. In: Handbook of Human Computer Interaction, pp. 813–824 (1997). ISBN: 0-444-1862-6
Langdon, P., Persad, U., Clarkson, P.J.: Developing a model of cognitive interaction for analytical inclusive design evaluation. Interact. Comput. 22, 510–529 (2010)
Ahmad, B.I., Murphy, J.K., Langdon, P.M., Godsill, S.J.: Filtering perturbed in-vehicle pointing gesture trajectories: improving the reliability of intent inference. In: Proceedings of IEEE International Workshop on Machine Learning for Signal Processing (MLSP 2014) (2014)
Ahmad, B.I., Langdon, P.M., Hardy, R., Godsill, S.J.: Intelligent intent-aware touchscreen systems using gesture tracking with endpoint prediction. In: Antona, M., Stephanidis, C. (eds.) UAHCI 2015. LNCS, vol. 9176, pp. 3–14. Springer, Heidelberg (2015)
Godsill, S., Vermaak, J.: Models and algorithms for tracking using trans-dimensional sequential Monte Carlo. In: Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2004), pp. 973–976 (2004)
Langdon, P., Thimbleby, H.: Inclusion and interaction: designing interaction for inclusive populations. Interact. Comput. 22, 439–448 (2010)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2016 Springer International Publishing Switzerland
About this paper
Cite this paper
Ahmad, B.I., Murphy, J.K., Langdon, P.M., Godsill, S.J. (2016). Predictive Pointing from Automotive to Inclusive Design. In: Antona, M., Stephanidis, C. (eds) Universal Access in Human-Computer Interaction. Users and Context Diversity. UAHCI 2016. Lecture Notes in Computer Science(), vol 9739. Springer, Cham. https://doi.org/10.1007/978-3-319-40238-3_50
Download citation
DOI: https://doi.org/10.1007/978-3-319-40238-3_50
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-40237-6
Online ISBN: 978-3-319-40238-3
eBook Packages: Computer ScienceComputer Science (R0)