A software framework for real-time multi-modal detection of microsleeps
- 149 Downloads
A software framework is described which was designed to process EEG, video of one eye, and head movement in real time, towards achieving early detection of microsleeps for prevention of fatal accidents, particularly in transport sectors. The framework is based around a pipeline structure with user-replaceable signal processing modules. This structure can encapsulate a wide variety of feature extraction and classification techniques and can be applied to detecting a variety of aspects of cognitive state. Users of the framework can implement signal processing plugins in C++ or Python. The framework also provides a graphical user interface and the ability to save and load data to and from arbitrary file formats. Two small studies are reported which demonstrate the capabilities of the framework in typical applications: monitoring eye closure and detecting simulated microsleeps. While specifically designed for microsleep detection/prediction, the software framework can be just as appropriately applied to (i) other measures of cognitive state and (ii) development of biomedical instruments for multi-modal real-time physiological monitoring and event detection in intensive care, anaesthesiology, cardiology, neurosurgery, etc. The software framework has been made freely available for researchers to use and modify under an open source licence.
KeywordsBiosignals Real-time Multi-modal Cognitive monitoring Software framework
Simon Knopp was the recipient of a University of Canterbury Doctoral Scholarship and the work reported formed part of his doctoral study.
Compliance with ethical standards
Conflict of interest
The authors declare that they have no financial or personal relationships with other people or organisations that could have inappropriately influenced this work.
Ethical approval was not required due to the small scale and non-invasive nature of the experiments.
- 4.Jones RD, Poudel GR, Innes CRH, Davidson PR, Peiris MTR, Malla AM, Signal TL, Carroll GJ, Watts R, Bones PJ (2010) Lapses of responsiveness: characteristics, detection, and underlying mechanisms. In: Proceedings of 32nd IEEE Conference on Engineering in Medicine and Biology Society. pp 1788–1791. doi: 10.1109/IEMBS.2010.5626385
- 8.Knopp SJ (2015) A multi-modal device for application in microsleep detection. PhD thesis, University of Canterbury. http://hdl.handle.net/10092/10408. Accessed 04 Oct 2015
- 10.BITalino DIY biosignals. http://bitalino.com/. Accessed 04 Oct 2015
- 12.Heger D, Putze F, Amma C, Wand M, Plotkin I, Wielatt T, Schultz T (2010) BiosignalsStudio: a flexible framework for biosignal capturing and processing. In: Dillmann R, Beyerer J, Hanebeck UD, Schultz T (eds) Annual conference on artificial intelligence. Springer, Berlin/Heidelberg, pp 33–39. doi: 10.1007/978-3-642-16111-7_3 Google Scholar
- 13.Guger Technologies. http://www.gtec.at/Products/. Accessed 04 Oct 2015
- 14.GStreamer: open-source multimedia framework. http://gstreamer.freedesktop.org/. Accessed 04 Oct 2015
- 15.Qt. http://qt.io/. Accessed 04 Oct 2015
- 16.Gamma E, Helm R, Johnson R, Vlissides J (1994) Design patterns: elements of reusable object-oriented software. Pearson Education, Upper Saddle RiverGoogle Scholar
- 18.Jones E, Oliphant T, Peterson P et al (2001) SciPy: open source scientific tools for Python. http://www.scipy.org/. Accessed 04 Oct 2015
- 20.Knopp SJ, Bones PJ, Weddell SJ, Innes CRH, Jones RD (2013) A wearable device for measuring eye dynamics in real-world conditions. In: Proceedings of 35th IEEE Conference on Engineering in Medicine and Biology Society, pp 6615–6618. doi: 10.1109/EMBC.2013.6611072
- 22.Boulton RJ, Walthinsen E, Baker S, Johnson L, Bultje RS, Kost S, Müller T-P, Taymans W (2015) GStreamer plugin writer’s guide, Ch 15: memory management. http://gstreamer.freedesktop.org/data/doc/gstreamer/head/pwg/html/chapter-allocation.html. Accessed 04 Oct 2015
- 23.Ayyagari SSDP, Jones RD, Weddell S (2015). Optimized echo state networks with leaky integrator neurons for EEG-based microsleep detection. In: Proceedings of 37th IEEE Conference on Engineering in Medicine and Biology Society, pp 3775–3778. doi: 10.1109/EMBC.2015.7319215
- 24.Shoorangiz R, Weddell S, Jones RD (2016). Prediction of microsleeps from EEG: preliminary results. In: Proceedings of 38th IEEE Conference on Engineering in Medicine and Biology Society, pp 4650–4653Google Scholar