Abstract
This paper aims to investigate the physio-emotional state of the driver in the vehicle cabin using a multimodal approach, comprising context, motion, visual, and audio data, collected beforehand. Driver behavior monitoring is implemented upon the data gained from different types of sensors, including accelerometer, magnetometer, gyroscope, GPS, front-facing camera, microphone, and information retrieved from third-party services. This data is intended to fully describe driver behavior and aid advanced driver assistant systems to fully classify and recognize dangerous driving behavior, and generate alerts on how to eliminate emergency situations. The emotional state of the driver is determined as six basic emotions, including sadness, fear, disgust, anger, surprise, and happiness. This work eventually presents driving style classification, dividing drivers into three groups: normal, ecological, urban, risky, and aggressive driving. This classification may potentially recognize aggressive vehicle drivers on public roads, and, therefore, undertake measures to reduce the risk of traffic accident occurrence.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Examination of the Traffic Safety Environment During the Second Quarter Of 2020 October 2020, DOT HS 813 011. https://rosap.ntl.bts.gov/view/dot/50940/dot_50940_DS1.pdf. Accessed 28 Jan 2021
Haas, R.E., Bhattacharjee, S., Möller, D.P.F.: Advanced driver assistance systems. In: Akhilesh, K.B., Möller, D.P.F. (eds.) Smart Technologies, pp. 345–371. Springer, Singapore (2020). https://doi.org/10.1007/978-981-13-7139-4_27
Rebecca, S., Vahabaghaie, A., Murakhovsky, D., Bahouth, G., Drayer, B., St Lawrence, S.: Effectiveness of Advanced Driver Assistance Systems in Preventing System-Relevant Crashes. No. 2021–01–0869. SAE Technical Paper (2021)
Morando, A., Gershon, P., Mehler, B., Reimer, B.: Driver-initiated tesla autopilot disengagements in naturalistic driving. In: 12th International Conference on Automotive User Interfaces and Interactive Vehicular Applications (AutomotiveUI '20). Association for Computing Machinery, New York, NY, USA, pp. 57–65 (2020)
Webb, N., et al.: Waymo's safety methodologies and safety readiness determinations. arXiv preprint arXiv:2011.00054 (2020)
Islam, M., Mannering, F.: A temporal analysis of driver-injury severities in crashes involving aggressive and non-aggressive driving. Anal. Meth. Accid.t Res. 27, 100128 (2020)
Kashevnik, A., Lashkov, I., Gurtov, A.: Methodology and mobile application for driver behavior analysis and accident prevention. IEEE Trans. Intell. Transp. Syst. IEEE 21(6), 2427–2436 (2019)
Kashevnik, A., Lashkov, I., Ponomarev, A., Teslya, N., Gurtov, A.: Cloud-based driver monitoring system using a smartphone. IEEE Sens. IEEE. 20(12), 6701–6715 (2020)
Kashevnik, A., et al.: Multimodal Corpus Design for Audio-Visual Speech Recognition in Vehicle Cabin. IEEE Access (2021)
Jardin, P., Moisidis, I., Zetina, S.S., Rinderknecht, S.: Rule-based driving style classification using acceleration data profiles. In: 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC), Rhodes, Greece, 2020, pp. 1–6 (2020). https://doi.org/10.1109/ITSC45102.2020.9294611
Wang, Q., Zhang, R., Wang, Y., Lv, S.: Machine Learning-Based Driving Style Identification of Truck Drivers in Open-Pit Mines. Electronics 9(1), 1–23 (2020). https://doi.org/10.3390/electronics9010019
Mohammadnazar, A., Arvin, R., Khattak, A.J.: Classifying travelers’ driving style using basic safety messages generated by connected vehicles: Application of unsupervised machine learning. Transp. Res. Part C: Emer. Technol. 122, 102917 (2021). https://doi.org/10.1016/j.trc.2020.102917
Zhang, Y., Xu, Q., Wang, J., Wu, K., Zheng, Z., Lu, K.: A learning-based discretionary lane-change decision-making model with driving style awareness (2020)
Huang, C., Wang, X., Cao, J., Wang, S., Zhang, Y.: HCF: a hybrid CNN framework for behavior detection of distracted drivers. IEEE Access 8, 109335–109349 (2020). https://doi.org/10.1109/ACCESS.2020.3001159
Mafeni Mase, J., Chapman, P., Figueredo, G.P., Torres Torres, M.: A hybrid deep learning approach for driver distraction detection. In: 2020 International Conference on Information and Communication Technology Convergence (ICTC), Jeju Island, Korea (South), pp. 1–6 (2020). https://doi.org/10.1109/ICTC49870.2020.9289588
Lee, H., Lee, J., Shin, M.: Using wearable ECG/PPG sensors for driver drowsiness detection based on distinguishable pattern of recurrence plots. Electronics 8(2), 192 (2019). https://doi.org/10.3390/electronics8020192
Naqvi, R.A., Arsalan, M., Rehman, A., Rehman, A.U., Loh, W.-K., Paul, A.: Deep learning-based drivers emotion classification system in time series data for remote applications. Remote Sens. 12(3), 587 (2020). https://doi.org/10.3390/rs12030587
Choi, D.Y., Kim, D.-H., Song, B.C.: Multimodal attention network for continuous-time emotion recognition using video and EEG signals. IEEE Access 8, 203814–203826 (2020). https://doi.org/10.1109/ACCESS.2020.3036877
Cordero, J., Aguilar, J., Aguilar, K., Chávez, D., Puerto, E.: Recognition of the driving style in vehicle drivers. Sensors (Basel) 20(9), 2597 (2020). https://doi.org/10.3390/s20092597.PMID:32370223;PMCID:PMC7249129
Puerto, E., Aguilar, J., Vargas, R., Reyes, J.: An Ar2p deep learning architecture for the discovery and the selection of features. Neural Process. Lett. 50(1), 623–643 (2019). https://doi.org/10.1007/s11063-019-10062-4
Huang, Y., Yang, J., Liu, S., Pan, J.: Combining facial expressions and electroencephalography to enhance emotion recognition. Future Internet 11(5), 105 (2019). https://doi.org/10.3390/fi11050105
An, S., Ji, L.J., Marks, M., Zhang, Z.: Two sides of emotion: exploring positivity and negativity in six basic emotions across cultures. Front. Psychol. 8, 610 (2017)
Niu, S.F., Liu, Y.J., Wang, L., Li, H.Q.: Effects of different intervention methods on novice drivers’ speeding. Sustainability 11(4), 1168 (2019)
Mumcuoglu, M.E., et al.: Driver evaluation in heavy duty vehicles based on acceleration and braking behaviors. In: IECON 2020 The 46th Annual Conference of the IEEE Industrial Electronics Society, pp. 447–452. IEEE, October 2020
Minhad, K.N., Ali, S.H.M., Reaz, M.B.I.: Happy-anger emotions classifications from electrocardiogram signal for automobile driving safety and awareness. J. Transp. Health 7, 75–89 (2017)
Google Play – Drive Safely. https://play.google.com/store/apps/details?id=ru.igla.drivesafely. Accessed 01 Feb 2021
af Wåhlberg, A., Dorn, L., Kline, T.: The Manchester Driver Behaviour Questionnaire as a predictor of road traffic accidents. Theoret. Issues Ergon. Sci. 12(1), 66–86 (2011)
Spielberger, C.D.: State‐Trait anxiety inventory. Corsini Encyclopedia Psychol. 1 (2010)
Acknowledgments
The research has been supported by the Russian Foundation for Basic Research project # 19–29-09081. The prototype (discussed in Sect. 6) has been supported by the Russian State Research # 0073–2019-0005.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Lashkov, I., Kashevnik, A. (2022). A Multimodal Approach to Psycho-Emotional State Detection of a Vehicle Driver. In: Arai, K. (eds) Intelligent Systems and Applications. IntelliSys 2021. Lecture Notes in Networks and Systems, vol 295. Springer, Cham. https://doi.org/10.1007/978-3-030-82196-8_42
Download citation
DOI: https://doi.org/10.1007/978-3-030-82196-8_42
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-82195-1
Online ISBN: 978-3-030-82196-8
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)