Position stabilisation and lag reduction with Gaussian processes in sensor fusion system for user performance improvement

  • Shimin FengEmail author
  • Roderick Murray-Smith
  • Andrew Ramsay
Original Article


In this paper we present a novel Gaussian Process (GP) prior model-based sensor fusion approach to dealing with position uncertainty and lag in a system composed of an external position sensing device (Kinect) and inertial sensors embedded in a mobile device for user performance improvement. To test the approach, we conducted two experiments: (1) GPs sensor fusion simulation. Experimental results show that the novel GP sensor fusion helps improve the accuracy of position estimation, and reduce the lag (0.11 s). (2) User study on a trajectory-based target acquisition task in a spatially aware display application. We implemented the real-time sensor fusion system by augmenting the Kinect with a Nokia N9. In the trajectory-based interaction experiment, each user performed target selection tasks following a trajectory in (a) the Kinect system and (b) the sensor fusion system. In comparison with the Kinect time-delay system, our system enables the user to perform the task easier and faster. The MSE of target selection was reduced by 38.3 % and the average task completion time was reduced by 26.7 %.


Gaussian processes Human-computer interaction Sensor fusion Uncertainty User interfaces 



The authors would like to thank all of the experiment participants for their time and valuable feedback, and also thank Dr. Simon Rogers and Dr. John Williamson for their helpful discussions and valuable suggestions. This research has been jointly funded by University of Glasgow and China Scholarship Council. Nokia donated some of the equipment used.


  1. 1.
    Azimi M (2012) Skeletal joint smoothing white paper. Accessed Aug 2014
  2. 2.
    Azuma RT et al (1997) A survey of augmented reality. Presence 6:355–385CrossRefGoogle Scholar
  3. 3.
    Bo A, Hayashibe M, Poignet P et al (2011) Joint angle estimation in rehabilitation with inertial sensors and its integration with Kinect. In: EMBC’11: 33rd annual international conference of the IEEE engineering in medicine and biology society, pp 3479–3483Google Scholar
  4. 4.
    Casiez G, Roussel N, Vogel D (2012) 1 € filter: a simple speed-based low-pass filter for noisy input in interactive systems. In: Proceedings of the 2012 ACM annual conference on human factors in computing systems, ACM, pp 2527–2530)Google Scholar
  5. 5.
    Conner B, Holden L (1997) Providing a low latency user experience in a high latency application. In: Proceedings of the 1997 symposium on Interactive 3D graphics, ACM, pp 45–ffGoogle Scholar
  6. 6.
    Corke P, Lobo J, Dias J (2007) An introduction to inertial and visual sensing. Int J Robot Res 26:519–535 (SAGE Publications)Google Scholar
  7. 7.
    Feng S, Murray-Smith R (2014) Fusing Kinect sensor and inertial sensors with multi-rate Kalman filter. In: IET conference on data fusion target tracking 2014: algorithms and applications (DF TT 2014), pp 1–8Google Scholar
  8. 8.
    Feng S, Murray-Smith R (2016) Transformations of Gaussian Process priors for user matching. Int J Hum Comput Stud 86:32–47 (Elsevier)Google Scholar
  9. 9.
    Fitzmaurice GW (1993) Situated information spaces and spatially aware palmtop computers. Commun ACM 36:39–49 (ACM)Google Scholar
  10. 10.
    Forrester A, Sobester A, Keane A (2008) Engineering design via surrogate modelling: a practical guide, WileyGoogle Scholar
  11. 11.
    Forrester AI, Sóbester A, Keane AJ (2007) Multi-fidelity optimization via surrogate modelling. In: Proceedings of the royal society of london a: mathematical, physical and engineering sciences, vol 463. The Royal Society, pp 3251–3269Google Scholar
  12. 12.
    Girard A, Rasmussen CE, Candela JQ, Murray-Smith R (2003) Gaussian process priors with uncertain inputs—application to multiple-step ahead time series forecasting. In: Becker STS, Obermayer K (eds) Advances in neural information processing systems, vol 15. MIT Press, Cambridge, pp 529–536Google Scholar
  13. 13.
    Goovaerts P (1998) Ordinary cokriging revisited. Math Geol 30:21–42MathSciNetCrossRefGoogle Scholar
  14. 14.
    Hall DL, Llinas J (1997) An introduction to multisensor data fusion. Proc IEEE 85:6–23 (IEEE)Google Scholar
  15. 15.
    Han Z-H, Görtz S, Zimmermann R (2013) Improving variable-fidelity surrogate modeling via gradient-enhanced kriging and a generalized hybrid bridge function. Aerosp Sci Technol 25: 177–189 (Elsevier)Google Scholar
  16. 16.
    Hart SG, Staveland LE (1988) Development of NASA-TLX (task load index): results of empirical and theoretical research. Hum Ment Workload 1:139–183 (Amsterdam, Holland)Google Scholar
  17. 17.
    Hennessey C, Noureddin B, Lawrence P (2008) Fixation precision in high-speed noncontact eye-gaze tracking. Syst Man Cybern Part B Cybern IEEE Trans 38:289–298 (IEEE)Google Scholar
  18. 18.
    Hol J, Schön T, Luinge H, Slycke P, Gustafsson F (2007) Robust real-time tracking by fusing measurements from inertial and vision sensors. J Real Time Image Process 2:149–160 (Springer)Google Scholar
  19. 19.
    Jeon S, Tomizuka M, Katou T (2009) Kinematic Kalman filter (KKF) for robot end-effector sensing. J Dyn Syst Meas Control 131:021010CrossRefGoogle Scholar
  20. 20.
    Khaleghi B, Khamis A, Karray FO, Razavi SN (2011) Multisensor data fusion: a review of the state-of-the-art. Inf Fusion, ElsevierGoogle Scholar
  21. 21.
    Ko J, Fox D (2009) GP-BayesFilters: Bayesian filtering using Gaussian Process prediction and observation models. Auton Robots 27:75–90 (Springer)Google Scholar
  22. 22.
    Ko J, Klein DJ, Fox D, Haehnel D (2007) GP-UKF: unscented Kalman filters with Gaussian Process prediction and observation models. In: Intelligent robots and systems, 2007. IROS 2007. IEEE/RSJ International Conference on, IEEE, pp 1901–1907Google Scholar
  23. 23.
    Krzysztofowicz R, Long D (1990) Fusion of detection probabilities and comparison of multisensor systems. Syst Man Cybern IEEE Trans 20:665–677 (IEEE)Google Scholar
  24. 24.
    Leith DJ, Heidl M, Ringwood JV (2004) Gaussian Process prior models for electrical load forecasting. In: Probabilistic methods applied to power systems, 2004 International Conference on, IEEE, pp 112–117Google Scholar
  25. 25.
    Livingston MA, Sebastian J, Ai Z, Decker JW (2012) Performance measurements for the Microsoft Kinect skeleton. In: Virtual reality short papers and posters (VRW), 2012 IEEE, IEEE, pp 119–120Google Scholar
  26. 26.
    Llinas J, Hall DL, Liggins ME (2009) Handbook of Multisensor data fusion: theory and practice, CRC PressGoogle Scholar
  27. 27.
    Lu Z, Chen X, Li Q, Zhang X, Zhou P (2014) A hand gesture recognition framework and wearable gesture-based interaction prototype for mobile devices. Hum Mach Systems IEEE Trans 44:293–299CrossRefGoogle Scholar
  28. 28.
    Luo RC, Chang CC, Lai CC (2011) Multisensor fusion and integration: theories, applications, and its perspectives. Sens J IEEE 11:3122–3138 (IEEE)Google Scholar
  29. 29.
    MacKay DJ (1998) Introduction to Gaussian processes. NATO ASI Ser F Comput Syst Sci 168:133–166 (Springer Verlag)Google Scholar
  30. 30.
    MacKenzie IS, Ware C (1993) Lag as a determinant of human performance in interactive systems. In: Proceedings of the INTERACT’93 and CHI’93 conference on Human factors in computing systems, ACM, pp 488–493Google Scholar
  31. 31.
    Motion L (2015) Leap motion controller. Accessed Apr 2015
  32. 32.
    Murphy RR (1996) Biological and cognitive foundations of intelligent sensor fusion. Syst Man Cybern Part A Syst Hum IEEE Trans 26:42–51 (IEEE)Google Scholar
  33. 33.
    Murray-Smith R, Pearlmutter BA (2005) Transformations of Gaussian process priors. In: Proceedings of the first international conference on deterministic and statistical methods in machine learning,Springer-Verlag, Berlin, Heidelberg, pp 110–123Google Scholar
  34. 34.
    Norrie L, Koelle M, Murray-Smith R, Kranz M (2013) Putting books back on the shelf: Situated interactions with digital book collections on smartphones. In: Proceedings of the 12th international conference on mobile and ubiquitous multimedia MUM ’13, ACM, pp 44:1–44:2Google Scholar
  35. 35.
    OpenNI (2014) OpenNI. Accessed Jan 2014
  36. 36.
    Pavlovych A, Gutwin C (2012) Assessing target acquisition and tracking performance for complex moving targets in the presence of latency and jitter. In: Proceedings of graphics interface 2012 GI ’12, Canadian Information Processing Society, pp 109–116Google Scholar
  37. 37.
    Pavlovych A, Stuerzlinger W (2009) The tradeoff between spatial jitter and latency in pointing tasks. In: Proceedings of the 1st ACM SIGCHI symposium on Engineering interactive computing systems, ACM, pp 187–196Google Scholar
  38. 38.
    Rasmussen CE, Williams CKI (2005) Gaussian processes for machine learning (adaptive computation and machine learning), The MIT PressGoogle Scholar
  39. 39.
    Rohs M, Oulasvirta A (2008) Target acquisition with camera phones when used as magic lenses. In: Proceedings of the SIGCHI conference on human factors in computing systems, ACM, pp 1409–1418Google Scholar
  40. 40.
    Roweis S, Ghahramani Z (1999) A unifying review of linear Gaussian models. Neural Comput 11:305–345 (MIT Press)Google Scholar
  41. 41.
    Schmidt A (2000) Implicit human computer interaction through context. Pers Technol 4:191–199 (Springer)Google Scholar
  42. 42.
    Shneiderman B, Plaisant C (2005) Designing the user interface, edition4 edn. Pearson Addison Wesley, ReadingGoogle Scholar
  43. 43.
    Smith D, Singh S (2006) Approaches to multisensor data fusion in target tracking: a survey. Knowl Data Eng IEEE Trans 18:1696–1710 (IEEE)Google Scholar
  44. 44.
    Solak E, Murray-Smith R, Leithead WE, Leith DJ, Rasmussen CE (2003) Derivative observations in Gaussian Process models of dynamic systems. In: Becker STS, Obermayer K (eds) Advances in neural information processing systems, vol 15. MIT Press, Cambridge, pp 1033–1040Google Scholar
  45. 45.
    Stanton NA, Walker GH et al (2013) Human factors methods: a practical guide for engineering and design. Ashgate Publishing LtdGoogle Scholar
  46. 46.
    Strachan S, Murray-Smith R (2009) Bearing-based selection in mobile spatial interaction. Personal Ubiquitous Comput 13:265–280 (Springer-Verlag)Google Scholar
  47. 47.
    Susperregi L, Arruti A, Jauregi E, Sierra B, Martínez-Otzeta JM, Lazkano E, Ansuategui A (2013) Fusing multiple image transformations and a thermal sensor with Kinect to improve person detection ability. Eng Appl Artif Intell 26:1980–1991 (Elsevier)Google Scholar
  48. 48.
    Thrun S (2002) Probabilistic robotics. Commun ACM 45:52–57 (ACM)Google Scholar
  49. 49.
    Titterton D, Weston J (2004) Strapdown inertial navigation technology, vol 17. Peter Peregrinus LtdGoogle Scholar
  50. 50.
    Turner RD (2012) Gaussian Processes for state space models and change point detection. PhD thesis, University of CambridgeGoogle Scholar
  51. 51.
    Vasudevan S (2012) Data fusion with Gaussian processes. Robot Auton Syst 60:1528–1544 (Elsevier)Google Scholar
  52. 52.
    Wang JM, Fleet DJ, Hertzmann A (2008) Gaussian Process dynamical models for human motion. Pattern Anal Mach Intell IEEE Trans 30:283–298 (IEEE)Google Scholar
  53. 53.
    Ware C, Balakrishnan R (1994) Reaching for objects in VR displays: lag and frame rate. ACM Trans Comput Hum Interact (TOCHI) 1:331–356 (ACM)Google Scholar
  54. 54.
    Welch G, Bishop G (1995) An introduction to the Kalman filter, vol 7. University of North Carolina at Chapel Hill, Chapel HillGoogle Scholar
  55. 55.
    Welch G, Bishop G (1997) SCAAT: Incremental tracking with incomplete information. In: Proceedings of the 24th annual conference on computer graphics and interactive techniques SIGGRAPH ’97. ACM Press/Addison-Wesley Publishing Co, New York, pp 333–344Google Scholar
  56. 56.
    Williamson J (2006) Continuous uncertain interaction. Phd thesis, University of GlasgowGoogle Scholar
  57. 57.
    Williamson J, Murray-Smith R, Hughes S (2007) Shoogle: excitatory multimodal interaction on mobile devices. In: Proceedings of the SIGCHI conference on Human factors in computing systems CHI ’07, pp 121–124Google Scholar
  58. 58.
    Yee K-P (2003) Peephole displays: pen interaction on spatially aware handheld computers. In: Proceedings of the SIGCHI conference on human factors in computing systems. ACM, pp 1–8Google Scholar
  59. 59.
    Zaraki A, Mazzei D, Giuliani M, De Rossi D (2014) Designing and evaluating a social gaze-control system for a humanoid robot. Hum Mach Syst IEEE Trans 44:157–168CrossRefGoogle Scholar
  60. 60.
    Zhang Z-Q, Ji L-Y, Huang Z-P, Wu J-K (2012) Adaptive information fusion for human upper limb movement estimation. Syst Man Cybern Part A Syst Hum IEEE Trans 42:1100–1108 (IEEE)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2016

Authors and Affiliations

  • Shimin Feng
    • 1
    Email author
  • Roderick Murray-Smith
    • 1
  • Andrew Ramsay
    • 1
  1. 1.School of Computing Science, University of GlasgowGlasgowUK

Personalised recommendations