Skip to main content
SpringerLink
Log in

Multi-feature fusion prediction of fatigue driving based on improved optical flow algorithm

  • Original Paper
  • Published:
Signal, Image and Video Processing Aims and scope Submit manuscript

Abstract

To predict whether a driver is fatigued, a fatigue prediction algorithm based on the fusion of improved optical flow features and microfeatures was proposed. By improving the Main Directional Mean Optical-flow (MDMO) algorithm to extract facial microfeatures, the spatial and temporal features of facial micro-expressions were obtained without loss of feature information. Simultaneously, a remote photoplethysmography-based physiological feature detection algorithm was applied to extract the facial region of interest of the driver. The algorithm improves the convenience and accuracy of heart rate variability signal extraction. Then, the physiological characteristics and micro-expression characteristics were input to the long short-term memory to obtain the corresponding timing characteristics. Finally, the two time-series features were fused to predict the fatigue state of the driver. The experiment used the CASME II video face database which contains a large number of spontaneous and dynamic micro-expressions, the self-built 20 face video database, and the actual detection items in a fatigue state for 1 min each. The experimental results show that the improved MDMO optical flow algorithm is accurate for micro-expression recognition, with a rate of 75.2%, which is 7.83% higher than that of the traditional optical flow algorithm. In the case of microfeature fusion, the accuracy of driver fatigue prediction is as high as 95.24%, showing acceptable results in the prediction of driver fatigue.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  1. Li, K., Gong, Y., Ren, Z.: A fatigue driving detection algorithm based on facial multi-feature fusion. IEEE Access 8, 101244–101259 (2020). https://doi.org/10.1109/ACCESS.2020.2998363

    Article  Google Scholar 

  2. Li, Z., Nianqiang, L.: Fatigue Driving Detection System Based on Face Feature. In: 2019 IEEE 2nd International Conference on Electronics Technology (ICET), 2019, pp. 525–529. https://doi.org/10.1109/ELTECH.2019.8839479.

  3. Lei, J. et al.: Study on driving fatigue evaluation system based on short time period ECG Signal. In: 2018 21st International Conference on Intelligent Transportation Systems (ITSC), 2018, pp. 2466–2470. https://doi.org/10.1109/ITSC.2018.8569409.

  4. Ma, Y., Tian, F., Zhao, Q., Hu, B.: Design and application of mental fatigue detection system using non-contact ECG and BCG measurement. IEEE International Conference on Bioinformatics and Biomedicine (BIBM) 2018, 1508–1513 (2018). https://doi.org/10.1109/BIBM.2018.8621578

    Article  Google Scholar 

  5. Luo, X., Wang, H.: EEG-Based Recognition of Fatigue Driving on Highway. In: 2020 International Conference on Communications, Information System and Computer Engineering (CISCE), pp. 241–244 (2020). https://doi.org/10.1109/CISCE50729.2020.00054.

  6. Attarodi, G., Matla Nikooei, S., Jafarnia Dabanloo N., Pourmasoumi, P., Tareh, A.: Detection of Driver’s Drowsiness Using New Features Extracted from HRV Signal, 2018 Computing in Cardiology Conference (CinC), 2018, pp. 1–4. https://doi.org/10.22489/CinC.2018.014.

  7. Li, P., Benezeth, Y., Nakamura, K., Gomez, R., Li, C., Yang, F.: An Improvement for Video-based Heart Rate Variability Measurement. In: 2019 IEEE 4th International Conference on Signal and Image Processing (ICSIP), pp. 435–439 (2019). https://doi.org/10.1109/SIPROCESS.2019.8868712.

  8. Sahayadhas, A., Sundaraj, K., Murugappan, M.: Detecting driver Drowsiness based on sensors: a review. Sensors 12, 16937–16953 (2012). https://doi.org/10.3390/s121216937

    Article  Google Scholar 

  9. Wang, W., den Brinker, A.C., Stuijk, S., de Haan, G.: Algorithmic principles of remote PPG. IEEE Trans. Biomed. Eng. 64(7), 1479–1491 (2017). https://doi.org/10.1109/TBME.2016.2609282

    Article  Google Scholar 

  10. Zhao, Y., Xie, K., Zou, Z., He, J.-B.: Intelligent Recognition of Fatigue and Sleepiness Based on InceptionV3-LSTM via Multi-Feature Fusion. IEEE Access 8, 144205–144217 (2020). https://doi.org/10.1109/ACCESS.2020.3014508

    Article  Google Scholar 

  11. Liu, Y., Zhang, J., Yan, W., Wang, S., Zhao, G., Fu, X.: A main directional mean optical flow feature for spontaneous micro-expression recognition. IEEE Trans. Affect. Comput. 7(4), 299–310 (2016). https://doi.org/10.1109/TAFFC.2015.2485205

    Article  Google Scholar 

  12. Guojiang, W., Guoliang, Y.: A Modified optical flow algorithm and its application in facial expression recognition. In: 3rd IEEE International Conference on Computer and Communications (ICCC), pp. 1601–1605 (2017). https://doi.org/10.1109/CompComm.2017.8322810.

  13. Verburg, M., Menkovski, V.: Micro-expression detection in long videos using optical flow and recurrent neural networks. In: 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019), 2019, pp. 1–6, doi: https://doi.org/10.1109/FG.2019.8756588.

  14. Ma, H., An, G., Ruan, Q.: Micro-expression recognition described by histogram of average optical flow direction. Signal Processing 34(223): 279–288 (2018). https://doi.org/10.16798/j.issn.1003-0530.2018.03.004.

  15. Ma, H., An, G., Wu, S., Yang, F.: A region histogram of oriented optical flow (RHOOF) feature for apex frame spotting in micro-expression. International Symposium on Intelligent Signal Processing and Communication Systems (ISPACS) 2017, 281–286 (2017). https://doi.org/10.1109/ISPACS.2017.8266489

    Article  Google Scholar 

  16. Luguev, T., Seuß, D., Garbas, J.: Deep Learning Based Affective Sensing With Remote Photoplethysmography. In: 2020 54th Annual Conference on Information Sciences and Systems (CISS), pp. 1–4 (2020). https://doi.org/10.1109/CISS48834.2020.1570617362.

  17. Zhao, C., Han, W., Chen, Z., Li, Y., Feng, Y.: Remote estimation of heart rate based on multi-scale facial ROIs. IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) 2020, 1120–1129 (2020). https://doi.org/10.1109/CVPRW50498.2020.00147

    Article  Google Scholar 

  18. Guo, Y., Lu, Y., Liu, R.W., Yang, M., Chui, K.T.: Low-light image enhancement with regularized illumination optimization and deep noise suppression. IEEE Access 8, 145297–145315 (2020). https://doi.org/10.1109/ACCESS.2020.3015217

    Article  Google Scholar 

  19. Liu, H., Sun, X., Han, H., Cao, W.: Low-light video image enhancement based on multiscale retinex-like algorithm. Chin. Control Decis. Conf. (CCDC) 2016, 3712–3715 (2016). https://doi.org/10.1109/CCDC.2016.7531629

    Article  Google Scholar 

  20. Ben, X et al.: Video-Based Facial Micro-Expression Analysis: A Survey of Datasets, Features and Algorithms. In: IEEE Transactions on Pattern Analysis and Machine Intelligence (2021). https://doi.org/10.1109/TPAMI.2021.3067464.

  21. Wu, H.-Y., Rubinstein, M., Shih, E., Guttag, J., Durand, F., Freeman, W.: Eulerian video magnification for revealing subtle changes in the world. ACM Trans. Graph. 31(4), 8 (2012). https://doi.org/10.1145/2185520.2185561

    Article  Google Scholar 

  22. Rong, C., Xuanxuan, G.: Histogram-Based Self-Adaptive Segmented Linear Method of Grayscale Transformation for Infrared Images. In: 7th International Conference on Information Science and Control Engineering (ICISCE), pp. 679–681 (2020). https://doi.org/10.1109/ICISCE50968.2020.00145.

  23. Baltrušaitis, T., Robinson, P., Morency, L.: OpenFace: an open source facial behavior analysis toolkit. IEEE Winter Conf. Appl. Comput. Vis. (WACV) 2016, 1–10 (2016). https://doi.org/10.1109/WACV.2016.7477553

    Article  Google Scholar 

  24. Baltrusaitis, T., Zadeh, A., Lim, Y. C., Morency, L.: OpenFace 2.0: Facial Behavior Analysis Toolkit. In: 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), pp. 59–66 (2018). https://doi.org/10.1109/FG.2018.00019.

  25. Baltrusaitis, T., Robinson, P., Morency, L.: Constrained local neural fields for robust facial landmark detection in the wild. IEEE Int. Conf. Comput. Vis. Workshops 2013, 354–361 (2013). https://doi.org/10.1109/ICCVW.2013.54

    Article  Google Scholar 

  26. Baltrušaitis, T., Robinson, P., Morency, L.-P.: 3D Constrained Local Model for rigid and non-rigid facial tracking. IEEE Conf. Comput. Vis. Pattern Recogn. 2012, 2610–2617 (2012). https://doi.org/10.1109/CVPR.2012.6247980

    Article  Google Scholar 

  27. Asthana, A., Zafeiriou, S., Cheng, S., Pantic, M.: Robust discriminative response map fitting with constrained local models. IEEE Conf. Comput. Vis. Pattern Recognit. 2013, 3444–3451 (2013). https://doi.org/10.1109/CVPR.2013.442

    Article  Google Scholar 

  28. Zhang, H., Xiao, L., Xu, G.: A novel tracking method based on improved FAST corner detection and pyramid LK optical flow. Chin. Control Decis. Conf. (CCDC) 2020, 1871–1876 (2020). https://doi.org/10.1109/CCDC49329.2020.9164332

    Article  Google Scholar 

  29. Kwon, S., Kim, J., Lee, D., Park, K.: ROI analysis for remote photoplethysmography on facial video. In: 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 4938–4941 (2015). https://doi.org/10.1109/EMBC.2015.7319499.

  30. Poh, M.Z., McDuff, D.J., Picard, R.W.: Advancements in noncontact, multiparameter physiological measurements using a webcam, IEEE transactions on bio-medical. Engineering 58(1), 7–11 (2011)

    Google Scholar 

Download references

Acknowledgements

This work was supported in part by the Natural Science Foundation of Xinjiang Uygur Autonomous Region (2020D01A131), the Fund of Hubei Ministry of Education (B2019039), the Graduate Teaching and Research Fund of Yangtze University (YJY201909), the Teaching and Research Fund of Yangtze University (JY2019011), the Undergraduate Training Programs for Innovation and Entrepreneurship of Yangtze University under Grant Yz2020057, Yz2020059, Yz2020156, and the National College Student Innovation and Entrepreneurship Training Program (202110489003).

Author information

Author notes

  1. Kai Tao and Kai Xie have been contributed equally to this work.

Authors and Affiliations

  1. School of Electronic Information, Yangtze University, Jingzhou, 434023, China

    Kai Tao & Kai Xie

  2. National Demonstration Center for Experimental Electrical and Electronic Education, Yangtze University, Jingzhou, 434023, China

    Kai Tao & Kai Xie

  3. Western Institute of Yangtze University, Karamay, 834000, China

    Kai Tao, Kai Xie & Chang Wen

  4. School of Computer Science, Yangtze University, Jingzhou, 434023, China

    Chang Wen

  5. School of Computer Science and Engineering, Central South University, Changsha, 410083, China

    Jian-Biao He

Authors
  1. Kai Tao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kai Xie
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chang Wen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jian-Biao He
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kai Xie.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Tao, K., Xie, K., Wen, C. et al. Multi-feature fusion prediction of fatigue driving based on improved optical flow algorithm. SIViP 17, 371–379 (2023). https://doi.org/10.1007/s11760-022-02242-y

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11760-022-02242-y

Keywords

Search

Navigation