A software framework for real-time multi-modal detection of microsleeps

  • Simon J. Knopp
  • Philip J. Bones
  • Stephen J. Weddell
  • Richard D. Jones
Technical Paper


A software framework is described which was designed to process EEG, video of one eye, and head movement in real time, towards achieving early detection of microsleeps for prevention of fatal accidents, particularly in transport sectors. The framework is based around a pipeline structure with user-replaceable signal processing modules. This structure can encapsulate a wide variety of feature extraction and classification techniques and can be applied to detecting a variety of aspects of cognitive state. Users of the framework can implement signal processing plugins in C++ or Python. The framework also provides a graphical user interface and the ability to save and load data to and from arbitrary file formats. Two small studies are reported which demonstrate the capabilities of the framework in typical applications: monitoring eye closure and detecting simulated microsleeps. While specifically designed for microsleep detection/prediction, the software framework can be just as appropriately applied to (i) other measures of cognitive state and (ii) development of biomedical instruments for multi-modal real-time physiological monitoring and event detection in intensive care, anaesthesiology, cardiology, neurosurgery, etc. The software framework has been made freely available for researchers to use and modify under an open source licence.


Biosignals Real-time Multi-modal Cognitive monitoring Software framework 



Simon Knopp was the recipient of a University of Canterbury Doctoral Scholarship and the work reported formed part of his doctoral study.

Compliance with ethical standards

Conflict of interest

The authors declare that they have no financial or personal relationships with other people or organisations that could have inappropriately influenced this work.

Ethical approval

Ethical approval was not required due to the small scale and non-invasive nature of the experiments.


  1. 1.
    Borghini G, Astolfi L, Vecchiato G, Mattia D, Babiloni F (2014) Measuring neurophysiological signals in aircraft pilots and car drivers for the assessment of mental workload, fatigue and drowsiness. Neurosci Biobehav Rev 44:58–75. doi: 10.1016/j.neubiorev.2012.10.003 CrossRefPubMedGoogle Scholar
  2. 2.
    Zander TO, Kothe C (2011) Towards passive brain–computer interfaces: applying brain–computer interface technology to human–machine systems in general. J Neural Eng 8:25005. doi: 10.1088/1741-2560/8/2/025005 CrossRefGoogle Scholar
  3. 3.
    Reeves LM, Schmorrow DD, Stanney KM (2007) Augmented cognition and cognitive state assessment technology – near-term, mid-term, and long-term research objectives. In: Foundations of augmented cognition. Springer, Berlin, pp. 220–228. doi: 10.1007/978-3-540-73216-7_25 CrossRefGoogle Scholar
  4. 4.
    Jones RD, Poudel GR, Innes CRH, Davidson PR, Peiris MTR, Malla AM, Signal TL, Carroll GJ, Watts R, Bones PJ (2010) Lapses of responsiveness: characteristics, detection, and underlying mechanisms. In: Proceedings of 32nd IEEE Conference on Engineering in Medicine and Biology Society. pp 1788–1791. doi: 10.1109/IEMBS.2010.5626385
  5. 5.
    Davidson PR, Jones RD, Peiris MTR (2007) EEG-based lapse detection with high temporal resolution. IEEE Trans Biomed Eng 54:832–839. doi: 10.1109/TBME.2007.893452 CrossRefPubMedGoogle Scholar
  6. 6.
    Peiris MTR, Davidson PR, Bones PJ, Jones RD (2011) Detection of lapses in responsiveness from the EEG. J Neural Eng 8:16003. doi: 10.1088/1741-2560/8/1/016003 CrossRefGoogle Scholar
  7. 7.
    Golz M, Sommer D, Chen M, Trutschel U, Mandic D (2007) Feature fusion for the detection of microsleep events. J VLSI Signal Process 49:329–342. doi: 10.1007/s11265-007-0083-4 CrossRefGoogle Scholar
  8. 8.
    Knopp SJ (2015) A multi-modal device for application in microsleep detection. PhD thesis, University of Canterbury. Accessed 04 Oct 2015
  9. 9.
    da Silva HP, Lourenço A, Fred A, Martins R (2014) BIT: biosignal igniter toolkit. Comput Methods Progr Biomed 115:20–32. doi: 10.1016/j.cmpb.2014.03.002 CrossRefGoogle Scholar
  10. 10.
    BITalino DIY biosignals. Accessed 04 Oct 2015
  11. 11.
    Lourenço A, da Silva HP, Carreiras C, Alves AP, Fred A (2014) A web-based platform for biosignal visualization and annotation. Multimed Tools Appl 70:433–460. doi: 10.1007/s11042-013-1397-9 CrossRefGoogle Scholar
  12. 12.
    Heger D, Putze F, Amma C, Wand M, Plotkin I, Wielatt T, Schultz T (2010) BiosignalsStudio: a flexible framework for biosignal capturing and processing. In: Dillmann R, Beyerer J, Hanebeck UD, Schultz T (eds) Annual conference on artificial intelligence. Springer, Berlin/Heidelberg, pp 33–39. doi: 10.1007/978-3-642-16111-7_3 Google Scholar
  13. 13.
    Guger Technologies. Accessed 04 Oct 2015
  14. 14.
    GStreamer: open-source multimedia framework. Accessed 04 Oct 2015
  15. 15.
    Qt. Accessed 04 Oct 2015
  16. 16.
    Gamma E, Helm R, Johnson R, Vlissides J (1994) Design patterns: elements of reusable object-oriented software. Pearson Education, Upper Saddle RiverGoogle Scholar
  17. 17.
    Pree W (1994) Meta patterns—a Means for capturing the essentials of reusable object-oriented design. In: Object-oriented program. Springer, Berlin, pp 150–162CrossRefGoogle Scholar
  18. 18.
    Jones E, Oliphant T, Peterson P et al (2001) SciPy: open source scientific tools for Python. Accessed 04 Oct 2015
  19. 19.
    Santamaria J, Chiappa KH (1987) The EEG of drowsiness in normal adults. J Clin Neurophysiol 4:327–382. doi: 10.1097/00004691-198710000-00002 CrossRefPubMedGoogle Scholar
  20. 20.
    Knopp SJ, Bones PJ, Weddell SJ, Innes CRH, Jones RD (2013) A wearable device for measuring eye dynamics in real-world conditions. In: Proceedings of 35th IEEE Conference on Engineering in Medicine and Biology Society, pp 6615–6618. doi: 10.1109/EMBC.2013.6611072
  21. 21.
    Wierwille WW, Ellsworth LA (1994) Evaluation of driver drowsiness by trained raters. Accid Anal Prev 26:571–581. doi: 10.1016/0001-4575(94)90019-1 CrossRefPubMedGoogle Scholar
  22. 22.
    Boulton RJ, Walthinsen E, Baker S, Johnson L, Bultje RS, Kost S, Müller T-P, Taymans W (2015) GStreamer plugin writer’s guide, Ch 15: memory management. Accessed 04 Oct 2015
  23. 23.
    Ayyagari SSDP, Jones RD, Weddell S (2015). Optimized echo state networks with leaky integrator neurons for EEG-based microsleep detection. In: Proceedings of 37th IEEE Conference on Engineering in Medicine and Biology Society, pp 3775–3778. doi: 10.1109/EMBC.2015.7319215
  24. 24.
    Shoorangiz R, Weddell S, Jones RD (2016). Prediction of microsleeps from EEG: preliminary results. In: Proceedings of 38th IEEE Conference on Engineering in Medicine and Biology Society, pp 4650–4653Google Scholar

Copyright information

© Australasian College of Physical Scientists and Engineers in Medicine 2017

Authors and Affiliations

  • Simon J. Knopp
    • 1
    • 2
  • Philip J. Bones
    • 1
  • Stephen J. Weddell
    • 1
  • Richard D. Jones
    • 1
    • 2
  1. 1.Department of Electrical and Computer EngineeringUniversity of CanterburyChristchurchNew Zealand
  2. 2.New Zealand Brain Research InstituteChristchurchNew Zealand

Personalised recommendations