Skip to main content

Advertisement

SpringerLink
Log in

A software framework for real-time multi-modal detection of microsleeps

  • Technical Paper
  • Published:
Australasian Physical & Engineering Sciences in Medicine Aims and scope Submit manuscript

Abstract

A software framework is described which was designed to process EEG, video of one eye, and head movement in real time, towards achieving early detection of microsleeps for prevention of fatal accidents, particularly in transport sectors. The framework is based around a pipeline structure with user-replaceable signal processing modules. This structure can encapsulate a wide variety of feature extraction and classification techniques and can be applied to detecting a variety of aspects of cognitive state. Users of the framework can implement signal processing plugins in C++ or Python. The framework also provides a graphical user interface and the ability to save and load data to and from arbitrary file formats. Two small studies are reported which demonstrate the capabilities of the framework in typical applications: monitoring eye closure and detecting simulated microsleeps. While specifically designed for microsleep detection/prediction, the software framework can be just as appropriately applied to (i) other measures of cognitive state and (ii) development of biomedical instruments for multi-modal real-time physiological monitoring and event detection in intensive care, anaesthesiology, cardiology, neurosurgery, etc. The software framework has been made freely available for researchers to use and modify under an open source licence.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Borghini G, Astolfi L, Vecchiato G, Mattia D, Babiloni F (2014) Measuring neurophysiological signals in aircraft pilots and car drivers for the assessment of mental workload, fatigue and drowsiness. Neurosci Biobehav Rev 44:58–75. doi:10.1016/j.neubiorev.2012.10.003

    Article  PubMed  Google Scholar 

  2. Zander TO, Kothe C (2011) Towards passive brain–computer interfaces: applying brain–computer interface technology to human–machine systems in general. J Neural Eng 8:25005. doi:10.1088/1741-2560/8/2/025005

    Article  Google Scholar 

  3. Reeves LM, Schmorrow DD, Stanney KM (2007) Augmented cognition and cognitive state assessment technology – near-term, mid-term, and long-term research objectives. In: Foundations of augmented cognition. Springer, Berlin, pp. 220–228. doi:10.1007/978-3-540-73216-7_25

    Chapter  Google Scholar 

  4. Jones RD, Poudel GR, Innes CRH, Davidson PR, Peiris MTR, Malla AM, Signal TL, Carroll GJ, Watts R, Bones PJ (2010) Lapses of responsiveness: characteristics, detection, and underlying mechanisms. In: Proceedings of 32nd IEEE Conference on Engineering in Medicine and Biology Society. pp 1788–1791. doi:10.1109/IEMBS.2010.5626385

  5. Davidson PR, Jones RD, Peiris MTR (2007) EEG-based lapse detection with high temporal resolution. IEEE Trans Biomed Eng 54:832–839. doi:10.1109/TBME.2007.893452

    Article  PubMed  Google Scholar 

  6. Peiris MTR, Davidson PR, Bones PJ, Jones RD (2011) Detection of lapses in responsiveness from the EEG. J Neural Eng 8:16003. doi:10.1088/1741-2560/8/1/016003

    Article  Google Scholar 

  7. Golz M, Sommer D, Chen M, Trutschel U, Mandic D (2007) Feature fusion for the detection of microsleep events. J VLSI Signal Process 49:329–342. doi:10.1007/s11265-007-0083-4

    Article  Google Scholar 

  8. Knopp SJ (2015) A multi-modal device for application in microsleep detection. PhD thesis, University of Canterbury. http://hdl.handle.net/10092/10408. Accessed 04 Oct 2015

  9. da Silva HP, Lourenço A, Fred A, Martins R (2014) BIT: biosignal igniter toolkit. Comput Methods Progr Biomed 115:20–32. doi:10.1016/j.cmpb.2014.03.002

    Article  Google Scholar 

  10. BITalino DIY biosignals. http://bitalino.com/. Accessed 04 Oct 2015

  11. Lourenço A, da Silva HP, Carreiras C, Alves AP, Fred A (2014) A web-based platform for biosignal visualization and annotation. Multimed Tools Appl 70:433–460. doi:10.1007/s11042-013-1397-9

    Article  Google Scholar 

  12. Heger D, Putze F, Amma C, Wand M, Plotkin I, Wielatt T, Schultz T (2010) BiosignalsStudio: a flexible framework for biosignal capturing and processing. In: Dillmann R, Beyerer J, Hanebeck UD, Schultz T (eds) Annual conference on artificial intelligence. Springer, Berlin/Heidelberg, pp 33–39. doi:10.1007/978-3-642-16111-7_3

    Google Scholar 

  13. Guger Technologies. http://www.gtec.at/Products/. Accessed 04 Oct 2015

  14. GStreamer: open-source multimedia framework. http://gstreamer.freedesktop.org/. Accessed 04 Oct 2015

  15. Qt. http://qt.io/. Accessed 04 Oct 2015

  16. Gamma E, Helm R, Johnson R, Vlissides J (1994) Design patterns: elements of reusable object-oriented software. Pearson Education, Upper Saddle River

    Google Scholar 

  17. Pree W (1994) Meta patterns—a Means for capturing the essentials of reusable object-oriented design. In: Object-oriented program. Springer, Berlin, pp 150–162

    Chapter  Google Scholar 

  18. Jones E, Oliphant T, Peterson P et al (2001) SciPy: open source scientific tools for Python. http://www.scipy.org/. Accessed 04 Oct 2015

  19. Santamaria J, Chiappa KH (1987) The EEG of drowsiness in normal adults. J Clin Neurophysiol 4:327–382. doi:10.1097/00004691-198710000-00002

    Article  CAS  PubMed  Google Scholar 

  20. Knopp SJ, Bones PJ, Weddell SJ, Innes CRH, Jones RD (2013) A wearable device for measuring eye dynamics in real-world conditions. In: Proceedings of 35th IEEE Conference on Engineering in Medicine and Biology Society, pp 6615–6618. doi:10.1109/EMBC.2013.6611072

  21. Wierwille WW, Ellsworth LA (1994) Evaluation of driver drowsiness by trained raters. Accid Anal Prev 26:571–581. doi:10.1016/0001-4575(94)90019-1

    Article  CAS  PubMed  Google Scholar 

  22. Boulton RJ, Walthinsen E, Baker S, Johnson L, Bultje RS, Kost S, Müller T-P, Taymans W (2015) GStreamer plugin writer’s guide, Ch 15: memory management. http://gstreamer.freedesktop.org/data/doc/gstreamer/head/pwg/html/chapter-allocation.html. Accessed 04 Oct 2015

  23. Ayyagari SSDP, Jones RD, Weddell S (2015). Optimized echo state networks with leaky integrator neurons for EEG-based microsleep detection. In: Proceedings of 37th IEEE Conference on Engineering in Medicine and Biology Society, pp 3775–3778. doi:10.1109/EMBC.2015.7319215

  24. Shoorangiz R, Weddell S, Jones RD (2016). Prediction of microsleeps from EEG: preliminary results. In: Proceedings of 38th IEEE Conference on Engineering in Medicine and Biology Society, pp 4650–4653

Download references

Acknowledgements

Simon Knopp was the recipient of a University of Canterbury Doctoral Scholarship and the work reported formed part of his doctoral study.

Author information

Authors and Affiliations

  1. Department of Electrical and Computer Engineering, University of Canterbury, Christchurch, New Zealand

    Simon J. Knopp, Philip J. Bones, Stephen J. Weddell & Richard D. Jones

  2. New Zealand Brain Research Institute, Christchurch, New Zealand

    Simon J. Knopp & Richard D. Jones

Authors
  1. Simon J. Knopp
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Philip J. Bones
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Stephen J. Weddell
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Richard D. Jones
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Simon J. Knopp.

Ethics declarations

Conflict of interest

The authors declare that they have no financial or personal relationships with other people or organisations that could have inappropriately influenced this work.

Ethical approval

Ethical approval was not required due to the small scale and non-invasive nature of the experiments.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Knopp, S.J., Bones, P.J., Weddell, S.J. et al. A software framework for real-time multi-modal detection of microsleeps. Australas Phys Eng Sci Med 40, 739–749 (2017). https://doi.org/10.1007/s13246-017-0559-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13246-017-0559-x

Keywords

Search

Navigation