Smartphone-Based Driver Support in Vehicle Cabin: Human-Computer Interaction Interface

  • Alexey KashevnikEmail author
  • Igor Lashkov
  • Dmitry Ryumin
  • Alexey Karpov
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11659)


The paper proposes an approach to driver support in vehicle cabin oriented to dangerous states determination and recommendation generation. To determine dangerous states, we propose to analyze images from smartphone front-facing camera as well as analyze information from accessible sensors. We identified two main dangerous states that are most important to identify to prevent the possible accidents in the roads: drowsiness and distraction. In scope of the approach determined dangerous states are used to generate recommendations for the driver to notify him/her about drowsiness and distraction. Since the attention of the driver should be in the road during the driver, we propose the human-computer interaction interface that is based on speech recognition to interact the driver. Using the interface, the driver interacts with the driver support system to increase the quality of the dangerous state determination and recommendation generation.


Driver support Dangerous driving Human-computer interaction Vehicle Smartphone 



The work has been partially financially supported by grants: #17-29-03284, #16-07-00462, and #16-37-60100 of Russian Foundation for Basic Research; by Government of Russian Federation, Grant #08-08; and by the Russian State Research #0073-2019-0005.


  1. 1.
    Markovnikov, N., Kipyatkova, I.: An analytic survey of end-to-end speech recognition systems. SPIIRAS Proc. 3(58), 77–110 (2018)CrossRefGoogle Scholar
  2. 2.
    Schmidt, J., Laarousi, R., Stolzmann, W., Karrer, K.: Eye blink detection for different driver states in conditionally automated driving and manual driving using EOG and a driver camera. Behav. Res. Meth. 50(3), 1088–1101 (2018)CrossRefGoogle Scholar
  3. 3.
    Galarza, E.E., Egas, F.D., Silva, F.M., Velasco, P.M., Galarza, E.D.: Real time driver drowsiness detection based on driver’s face image behavior using a system of human computer interaction implemented in a smartphone. In: Rocha, Á., Guarda, T. (eds.) ICITS 2018. AISC, vol. 721, pp. 563–572. Springer, Cham (2018). Scholar
  4. 4.
    Mohammad, F., Mahadas, K., Hung, G.K.: Drowsy driver mobile application: development of a novel scleral-area detection method. Comput. Biol. Med. 89, 76–83 (2017)CrossRefGoogle Scholar
  5. 5.
    Bradski, G., Kaehler, A.: Learning OpenCV: Computer Vision in C ++ with the OpenCV Library, 2nd edn. O’Reilly Media Inc, Sebastopol (2013)Google Scholar
  6. 6.
    Dasgupta, A., Rahman, D., Routray, A.: A smartphone-based drowsiness detection and warning system for automotive drivers. In: IEEE Transactions on Intelligent Transportation Systems, pp. 1–10 (2018)Google Scholar
  7. 7.
    García-García, M., Caplier, A., Rombaut, M.: Sleep deprivation detection for real-time driver monitoring using deep learning. In: Campilho, A., Karray, F., ter Haar Romeny, B. (eds.) ICIAR 2018. LNCS, vol. 10882, pp. 435–442. Springer, Cham (2018). Scholar
  8. 8.
    Ramachandran, M., Chandrakala, S.: Android OpenCV based effective driver fatigue and distraction monitoring system. In: 2015 International Conference on Computing and Communications Technologies (ICCCT), pp. 262–266 (2015)Google Scholar
  9. 9.
    Abulkhair, M., et al.: Mobile platform detect and alerts system for driver fatigue. Procedia Comput. Sci. 62, 555–564 (2015)CrossRefGoogle Scholar
  10. 10.
    García-García, M., Caplier, A., Rombaut, M.: Driver head movements while using a smartphone in a naturalistic context. In: 6th International Symposium on Naturalistic Driving Research, pp. 1–5 (2017)Google Scholar
  11. 11.
    Qiao, Y., Zeng, K., Xu, L., Yin, X.: A smartphone-based driver fatigue detection using fusion of multiple real-time facial features. In: 2016 13th IEEE Annual Consumer Communications & Networking Conference (CCNC), pp. 230–235 (2016)Google Scholar
  12. 12.
    Kong, W., et al.: A system of driving fatigue detection based on machine vision and its application on smart device. J. Sens. 2015, 1–11 (2015)CrossRefGoogle Scholar
  13. 13.
    Nambi, A.U., et al.: HAMS: Driver and driving monitoring using a Smartphone. In: Proceedings of the 24th Annual International Conference on Mobile Computing and Networking. MobiCom 2018, pp. 840–842. ACM (2018)Google Scholar
  14. 14.
    Redmon, J., Farhadi, A.: YOLO9000: better, faster, stronger. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6517–6525 (2017)Google Scholar
  15. 15.
    Smirnov, A., Kashevnik, A., Lashkov, I., Parfenov, V.: Smartphone-based identification of dangerous driving situations: algorithms and implementation. In: Proceedings of the 18th Conference of Open Innovations Association FRUCT, St.Petersburg, pp. 306–313 (2016)Google Scholar
  16. 16.
    Smirnov, A., Kashevnik, A., Lashkov, I.: Human-smartphone interaction for dangerous situation detection and recommendation generation while driving. In: Ronzhin, A., Potapova, R., Németh, G. (eds.) SPECOM 2016. LNCS (LNAI), vol. 9811, pp. 346–353. Springer, Cham (2016). Scholar
  17. 17.
    Kopinski, T., Geisler, S., Handmann, U.: Contactless interaction for automotive applications. In: Mensch & Computer Workshopband, pp. 87–94 (2013)Google Scholar
  18. 18.
    Huang, Z., Huang, X.: A study on the application of voice interaction in automotive human machine interface experience design. In: AIP Conference Proceedings, vol. 1955, no. 1, p. 040074 (2018)Google Scholar
  19. 19.
    Hua, Z., Ng, W.L.: Speech recognition interface design for in-vehicle system. In: Proceedings of the 2nd International Conference on Automotive User Interfaces and Interactive Vehicular Applications, pp. 29–33 (2010)Google Scholar
  20. 20.
    Nass, C., et al.: Improving automotive safety by pairing driver emotion and car voice emotion. In: CHI 2005 Extended Abstracts on Human Factors In Computing Systems, pp. 1973–1976 (2005)Google Scholar
  21. 21.
    Peissner, M., Doebler, V., Metze, F.: Can voice interaction help reducing the level of distraction and prevent accidents. In: Meta-Study Driver Distraction Voice Interaction, p. 24 (2011)Google Scholar
  22. 22.
    Owens, J.M., McLaughlin, S.B., Sudweeks, J.: On-road comparison of driving performance measures when using handheld and voice-control interfaces for mobile phones and portable music players. SAE Int. J. Passeng. Cars-Mech. Syst. 3, 734–743 (2010)CrossRefGoogle Scholar
  23. 23.
    Meng, L.J., Wang, Z.Z.: Design and implementation of wireless voice controlled intelligent obstacle-avoiding toy car system. In: 2011 International Conference on Electronics, Communications and Control (ICECC), pp. 1982–1984 (2011)Google Scholar
  24. 24.
    Kashevnik, A., Lashkov, I.: Decision support system for drivers & passengers: Smartphone-based reference model and evaluation. In: Proceedings of the 23rd Conference of Open Innovations Association FRUCT, pp. 166–171 (2018)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Alexey Kashevnik
    • 1
    • 2
    Email author
  • Igor Lashkov
    • 1
  • Dmitry Ryumin
    • 1
  • Alexey Karpov
    • 1
  1. 1.SPIIRASSt. PetersburgRussia
  2. 2.ITMO UniversitySt. PetersburgRussia

Personalised recommendations