Abstract
Detailed analysis of seizure semiology, the symptoms and signs which occur during a seizure, is critical for management of epilepsy patients. Inter-rater reliability using qualitative visual analysis is often poor for semiological features. Therefore, automatic and quantitative analysis of video-recorded seizures is needed for objective assessment. We present GESTURES, a novel architecture combining convolutional neural networks (CNNs) and recurrent neural networks (RNNs) to learn deep representations of arbitrarily long videos of epileptic seizures. We use a spatiotemporal CNN (STCNN) pre-trained on large human action recognition (HAR) datasets to extract features from short snippets (\({\approx }\)0.5 s) sampled from seizure videos. We then train an RNN to learn seizure-level representations from the sequence of features. We curated a dataset of seizure videos from 68 patients and evaluated GESTURES on its ability to classify seizures into focal onset seizures (FOSs) (\(N = 106\)) vs. focal to bilateral tonic-clonic seizures (TCSs) (\(N = 77\)), obtaining an accuracy of 98.9% using bidirectional long short-term memory (BLSTM) units. We demonstrate that an STCNN trained on a HAR dataset can be used in combination with an RNN to accurately represent arbitrarily long videos of seizures. GESTURES can provide accurate seizure classification by modeling sequences of semiologies. The code, models and features dataset are available at https://github.com/fepegar/gestures-miccai-2021.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Change history
21 September 2021
In the first line of Sect. 2.2, “A neurophysiologist (A.A.)” has been replaced by “A neurophysiologist (C.S.)”. In addition, Equation (1) and one mathematical expression two paragraphs above this had been rasterized into images. This has been remedied.
References
Ahmedt-Aristizabal, D., Nguyen, K., Denman, S., Sridharan, S., Dionisio, S., Fookes, C.: Deep motion analysis for epileptic seizure classification. In: 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 3578–3581 (July 2018). ISSN 1558-4615. https://doi.org/10.1109/EMBC.2018.8513031
Ahmedt-Aristizabal, D., et al.: A hierarchical multimodal system for motion analysis in patients with epilepsy. Epilepsy Behav. 87, 46–58 (2018). https://doi.org/10.1016/j.yebeh.2018.07.028
Ahmedt-Aristizabal, D., et al.: Automated analysis of seizure semiology and brain electrical activity in presurgery evaluation of epilepsy: a focused survey. Epilepsia 58(11), 1817–1831 (2017). https://doi.org/10.1111/epi.13907
Ahmedt-Aristizabal, D., et al.: Deep facial analysis: a new phase I epilepsy evaluation using computer vision. Epilepsy Behav. 82, 17–24 (2018). https://doi.org/10.1016/j.yebeh.2018.02.010
Carreira, J., Zisserman, A.: Quo Vadis, action recognition? A new model and the kinetics dataset. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4724–4733 (July 2017). ISSN 1063-6919. https://doi.org/10.1109/CVPR.2017.502
Cunha, J.P.S., Vollmar, C., Li, Z., Fernandes, J., Feddersen, B., Noachtar, S.: Movement quantification during epileptic seizures: a new technical contribution to the evaluation of seizure semiology. In: Proceedings of the 25th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (IEEE Cat. No.03CH37439), vol. 1, pp. 671–673 (September 2003). ISSN 1094-687X. https://doi.org/10.1109/IEMBS.2003.1279851
Cunha, J.P.S., et al.: NeuroKinect: a novel low-cost 3Dvideo-EEG system for epileptic seizure motion quantification. PLOS ONE 11(1), e0145669 (2016). https://doi.org/10.1371/journal.pone.0145669
Fiest, K.M., et al.: Prevalence and incidence of epilepsy. Neurology 88(3), 296–303 (2017). https://doi.org/10.1212/WNL.0000000000003509
Fisher, R.S., et al.: Operational classification of seizure types by the international league against Epilepsy: Position Paper of the ILAE commission for classification and terminology. Epilepsia 58(4), 522–530 (2017). https://doi.org/10.1111/epi.13670
Ghadiyaram, D., Feiszli, M., Tran, D., Yan, X., Wang, H., Mahajan, D.: Large-scale weakly-supervised pre-training for video action recognition. arXiv arXiv:1905.00561 [cs] (May 2019)
Hutchinson, M., et al.: Accuracy and performance comparison of video action recognition approaches. In: 2020 IEEE High Performance Extreme Computing Conference (HPEC), pp. 1–8 (September 2020). ISSN 2643-1971. https://doi.org/10.1109/HPEC43674.2020.9286249
Jenssen, S., Gracely, E.J., Sperling, M.R.: How long do most seizures last? A systematic comparison of seizures recorded in the epilepsy monitoring unit. Epilepsia 47(9), 1499–1503 (2006). https://doi.org/10.1111/j.1528-1167.2006.00622.x
Karácsony, T., Loesch-Biffar, A.M., Vollmar, C., Noachtar, S., Cunha, J.P.S.: A deep learning architecture for epileptic seizure classification based on object and action recognition. In: 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), ICASSP 2020, pp. 4117–4121 (May 2020). ISSN 2379-190X. https://doi.org/10.1109/ICASSP40776.2020.9054649
Li, Z., Silva, A.M., Cunha, J.P.S.: Movement quantification in epileptic seizures: a new approach to video-EEG analysis. IEEE Trans. Biomed. Eng. 49(6), 565–573 (2002). Conference Name: IEEE Transactions on Biomedical Engineering. https://doi.org/10.1109/TBME.2002.1001971
Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. arXiv arXiv:1711.05101 [cs, math] (January 2019)
Maia, P., Hartl, E., Vollmar, C., Noachtar, S., Cunha, J.P.S.: Epileptic seizure classification using the NeuroMov database. In: 2019 IEEE 6th Portuguese Meeting on Bioengineering (ENBENG), pp. 1–4 (February 2019). https://doi.org/10.1109/ENBENG.2019.8692465
Nashef, L., So, E.L., Ryvlin, P., Tomson, T.: Unifying the definitions of sudden unexpected death in epilepsy. Epilepsia 53(2), 227–233 (2012). https://doi.org/10.1111/j.1528-1167.2011.03358.x
O’Dwyer, R., et al.: Lateralizing significance of quantitative analysis of head movements before secondary generalization of seizures of patients with temporal lobe epilepsy. Epilepsia 48(3), 524–530 (2007). https://doi.org/10.1111/j.1528-1167.2006.00967.x
Pérez-García, F., Scott, C., Sparks, R., Diehl, B., Ourselin, S.: Data to support the paper “transfer learning of deep spatiotemporal networks to model arbitrarily long videos of seizures” (July 2021). Publisher: University College London Type: dataset. https://doi.org/10.5522/04/14781771.v1
Ryvlin, P., et al.: Incidence and mechanisms of cardiorespiratory arrests in epilepsy monitoring units (MORTEMUS): a retrospective study. Lancet Neurol. 12(10), 966–977 (2013). https://doi.org/10.1016/S1474-4422(13)70214-X
Simonyan, K., Zisserman, A.: Two-stream convolutional networks for action recognition in videos. In: Proceedings of the 27th International Conference on Neural Information Processing Systems, NIPS 2014, vol. 1, pp. 568–576. MIT Press, Cambridge (December 2014)
Tufenkjian, K., Lüders, H.O.: Seizure semiology: its value and limitations in localizing the epileptogenic zone. J. Clin. Neurol. (Seoul, Korea) 8(4), 243–250 (2012). https://doi.org/10.3988/jcn.2012.8.4.243
Wang, L., et al.: Temporal segment networks for action recognition in videos. IEEE Trans. Pattern Anal. Mach. Intell. 41(11), 2740–2755 (2019). Conference Name: IEEE Transactions on Pattern Analysis and Machine Intelligence. https://doi.org/10.1109/TPAMI.2018.2868668
Zagoruyko, S., Komodakis, N.: Wide residual networks. arXiv arXiv:1605.07146 [cs] (June 2017)
Acknowledgments
This work is supported by the Engineering and Physical Sciences Research Council (EPSRC) [EP/R512400/1]. This work is additionally supported by the EPSRC-funded UCL Centre for Doctoral Training in Intelligent, Integrated Imaging in Healthcare (i4health) [EP/S021930/1] and the Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS, UCL) [203145Z/16/Z]. The data acquisition was supported by the National Institute of Neurological Disorders and Stroke [U01-NS090407].
This publication represents, in part, independent research commissioned by the Wellcome Innovator Award [218380/Z/19/Z/]. The views expressed in this publication are those of the authors and not necessarily those of the Wellcome Trust.
The weights for the 2D and 3D models were downloaded from TorchVision and https://github.com/moabitcoin/ig65m-pytorch, respectively.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Supplementary material 1 (mp4 29922 KB)
Supplementary material 2 (mp4 7218 KB)
Supplementary material 3 (mp4 27807 KB)
Supplementary material 4 (mp4 1372 KB)
Supplementary material 5 (mp4 12153 KB)
Supplementary material 6 (mp4 35553 KB)
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Pérez-García, F., Scott, C., Sparks, R., Diehl, B., Ourselin, S. (2021). Transfer Learning of Deep Spatiotemporal Networks to Model Arbitrarily Long Videos of Seizures. In: de Bruijne, M., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2021. MICCAI 2021. Lecture Notes in Computer Science(), vol 12905. Springer, Cham. https://doi.org/10.1007/978-3-030-87240-3_32
Download citation
DOI: https://doi.org/10.1007/978-3-030-87240-3_32
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-87239-7
Online ISBN: 978-3-030-87240-3
eBook Packages: Computer ScienceComputer Science (R0)