Skip to main content

Multimodal focus attention and stress detection and feedback in an augmented driver simulator

Abstract

This paper presents a driver simulator, which takes into account the information about the user’s state of mind (level of attention, fatigue state, stress state). The user’s state of mind analysis is based on video data and biological signals. Facial movements such as eyes blinking, yawning, head rotations, etc., are detected on video data: they are used in order to evaluate the fatigue and the attention level of the driver. The user’s electrocardiogram and galvanic skin response are recorded and analyzed in order to evaluate the stress level of the driver. A driver simulator software is modified so that the system is able to appropriately react to these critical situations of fatigue and stress: some audio and visual messages are sent to the driver, wheel vibrations are generated and the driver is supposed to react to the alert messages. A multi-threaded system is proposed to support multi-messages sent by the different modalities. Strategies for data fusion and fission are also provided. Some of these components are integrated within the first prototype of OpenInterface: the multimodal similar platform.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

References

  1. Benoit A, Caplier A (2005) Motion estimator inspired from biological model for head motion interpretation—WIAMIS05, Montreux, Suisse, April 2005

  2. Torralba AB, Herault J (1999) An efficient neuromorphic analog network for motion estimation. IEEE Trans Circuits Syst-I: Special issue on bio-inspired processors and CNNs for vision 46(2)

  3. Benoit A, Caplier A (2005) Head nods analysis : interpretation of non verbal communication gestures. IEEE ICIP, Genova

    Google Scholar 

  4. Benoit A, Caplier A (2005) Hypovigilence analysis: open or closed eye or mouth? Blinking or yawning frequency? In: IEEE AVSS, Como, Italy

  5. Machine Perception Toolbox (MPT). http://mplab.ucsd.edu/grants/project1/freesoftware/MPTWebSite/API/

  6. Bayes Net Toolbox for MatLab. http://www.cs.ubc.ca/∼murphyk/Software/BNT/bnt.html

  7. Qiang Ji, Zhiwei Zhu, Peilin Lan (2004) Real-time nonintrusive monitoring and prediction of driver fatigue. IEEE Trans Vehicular Technol 53(4):1052–1068

    Google Scholar 

  8. Force Feedback Direct Input Library. http://courses.washington.edu/css450/Fall2003/web_contents/direct_input_lib/DirectInput.html

  9. TORCS Driver Simulator. http://torcs.sourceforge.net/

  10. Similar Network of Excellence. http://www.similar.cc/

  11. Lal SK, Craig A (2002) Driver fatigue: electroencephalography and psychological assessment. Psychophysiology 39(3 May):313–321

    Google Scholar 

  12. Healey J, Seger J, Picard R Quantifying driver stress: developing a system for collecting and processing bio-metric signals in natural situation–MIT technical report. No. 483

  13. OpenCV library. http://www.intel.com/technology/computing/opencv

Download references

Acknowledgment

This work has been supported by the Simlar Network of Excellence.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Alice Caplier.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Benoit, A., Bonnaud, L., Caplier, A. et al. Multimodal focus attention and stress detection and feedback in an augmented driver simulator. Pers Ubiquit Comput 13, 33–41 (2009). https://doi.org/10.1007/s00779-007-0173-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00779-007-0173-0

Keywords

  • Driver simulator
  • Facial motion analysis
  • Physiological signals
  • Stress
  • Attention level
  • Data fusion
  • Data fission
  • OpenInterface