Intelligent Service Robotics

, Volume 6, Issue 4, pp 169–180 | Cite as

Comparative evaluation of performance measures for human driving skills

  • Reza Haghighi Osgouei
  • Hojin Lee
  • Seungmoon Choi
Original Research


In this paper, we evaluate the adequacy of several performance measures for the evaluation of driving skills between different drivers. This work was motivated by the need for a training system that captures the driving skills of an expert driver and transfers the skills to novice drivers using a haptic-enabled driving simulator. The performance measures examined include traditional task performance measures, e.g., the mean position error, and a stochastic distance between a pair of hidden Markov models (HMMs), each of which is trained for an individual driver. The emphasis of the latter is on the differences between the stochastic somatosensory processes of human driving skills. For the evaluation, we developed a driving simulator and carried out an experiment that collected the driving data of an expert driver whose data were used as a reference for comparison and of many other subjects. The performance measures were computed from the experimental data, and they were compared to each other. We also collected the subjective judgement scores of the driver’s skills made by a highly-experienced external evaluator, and these subjective scores were compared with the objective performance measures. Analysis results showed that the HMM-based distance metric had a moderately high correlation between the subjective scores and it was also consistent with the other task performance measures, indicating the adequacy of the HMM-based metric as an objective performance measure for driving skill learning. The findings of this work can contribute to developing a driving simulator for training with an objective assessment function of driving skills.


Hidden Markov model Driving skill evaluation Driving simulator Objective and subjective evaluation Absolute and relative performance measures 



This work was supported by NRF through an ERC 2011-0030075, a Pioneer program 2011-0027995, a BRL 2012-0008835, and a grant 013R1A2A2A01016907.


  1. 1.
    Irrlicht engine, a free open source 3d engine.
  2. 2.
    Newton game dynamics, open-source physics engine.
  3. 3.
    Berndt H, Emmert J, Dietmayer K (2008) Continuous driver intention recognition with hidden Markov models. In: Proceedings of the International IEEE Conference on Intelligent Transportation Systems, pp 1189–1194. doi: 10.1109/ITSC.2008.4732630
  4. 4.
    Calinon S, Billard A (2005) Recognition and reproduction of gestures using a probabilistic framework combining pca, ica and hmm. In: Proceedings of the 22nd international conference on Machine learning, ICML ’05, pp 105–112. ACM, New York, NY, USA. doi: 10.1145/1102351.1102365.
  5. 5.
    Cox TF, Cox MAA (1994) Multidimensional Scaling. Chapman & Hall, LondonMATHGoogle Scholar
  6. 6.
    Fraser AM (2008) Hidden Markov models and dynamical systems. SIAM (Society for Industrial and Applied Mathematics)Google Scholar
  7. 7.
    Juang BH, Rabiner LR (1985) A probabilistic distance measure for hidden Markov models. AT &T Tech J 64(2):391–408MathSciNetCrossRefGoogle Scholar
  8. 8.
    Kim S, Park G, Yim S, Choi S, Choi S (2009) Gesture-recognizing hand-held interface with vibrotactile feedback for 3d interaction. IEEE Trans Consumer Electron 55(3):1169–1177. doi: 10.1109/TCE.2009.5277972 CrossRefGoogle Scholar
  9. 9.
    Kruskal J (1964) Multidimensional scaling by optimizing goodness of fit to a nonmetric hypothesis. Psychometrika 29(1):1–27MathSciNetCrossRefMATHGoogle Scholar
  10. 10.
    Lee J, Choi S (2010) Effects of haptic guidance and disturbance on motor learning: potential advantage of haptic disturbance. In: Proceedings of the Haptics Symposium, IEEE, pp 335–342Google Scholar
  11. 11.
    Meng X, Lee KK, Xu Y (2006) Human driving behavior recognition based on hidden Markov models. In: Proceedings of the IEEE International Conference on Robotics and Biomimetics, pp 274–279. doi: 10.1109/ROBIO.2006.340166
  12. 12.
    Mitrovic D (2005) Reliable method for driving events recognition. IEEE Trans Intel Transp Syst 6(2):198–205. doi: 10.1109/TITS.2005.848367 CrossRefGoogle Scholar
  13. 13.
    Murphy K (2005) Hidden markov model (hmm) toolbox for matlab. (1998). Last updated: June 2005
  14. 14.
    Osgouei RH, Choi S (2012) Evaluation of driving skills using an hmm-based distance measure. In: Proceedings of the IEEE International Symposium on Haptic Audio Visual Environments and Games, pp. 50–55. (\(\copyright \) 2012 IEEE. Reprinted, with permission, from Evaluation of Driving Skills Using an HMM-based Distance Measure by Reza Haghighi Osgouei and Seungmoon Choi. In: Proceedings of the IEEE International Symposium on Haptic Audio Visual Environments and Games) Google Scholar
  15. 15.
    Porikli F (2004) Trajectory distance metric using hidden Markov model based representation. In: IEEE European Conference on Computer Vision, PETS, Workshop, vol. 3Google Scholar
  16. 16.
    Rabiner LR (1989) A tutorial on hidden Markov models and selected applications in speech recognition. Proc IEEE 77:257–286CrossRefGoogle Scholar
  17. 17.
    Rosen J, Solazzo M, Hannaford B, Sinanan M (2002) Task decomposition of laparoscopic surgery for objective evaluation of surgical residents’ learning curve using hidden Markov model. Comput Aided Surg 7:49–61CrossRefGoogle Scholar
  18. 18.
    Takano W, Matsushita A, Iwao K, Nakamura Y (2008) Recognition of human driving behaviors based on stochastic symbolization of time series signal. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, pp 167–172. doi: 10.1109/IROS.2008.4650671
  19. 19.
    Torkkola K, Venkatesan S, Liu H (2005) Sensor sequence modeling for driving. In: Proceedings of the International Florida Artificial Intelligence Research Society ConferenceGoogle Scholar
  20. 20.
    Yang J, Xu Y, Chen C (1993) Hidden markov model approach to skill learning and its application in telerobotics. In: Proceedings of the IEEE International Conference on Robotics and Automation, vol. 1, pp 396–402. doi: 10.1109/ROBOT.1993.292013
  21. 21.
    Zucchini W (2009) Hidden Markov models for time series: an introduction using R. CRC Press, Boca RatonCrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Reza Haghighi Osgouei
    • 1
  • Hojin Lee
    • 2
  • Seungmoon Choi
    • 2
  1. 1.Pohang University of Science and Technology (POSTECH)PohangRepublic of Korea
  2. 2.Pohang University of Science and Technology (POSTECH)PohangRepublic of Korea

Personalised recommendations