Abstract
Stress is a significant and growing phenomenon in the modern world that leads to numerous health problems. Robust and non-invasive method developments for early and accurate stress detection are crucial in enhancing people’s quality of life. Previous researches show that using machine learning approaches on physiological signals is a reliable stress predictor by achieving significant results. However, it requires determining features by hand. Such a selection is a challenge in this context since stress determines nonspecific human responses. This work overcomes such limitations by considering STREDWES, an approach for Stress Detection from Wearable Sensors Data. STREDWES encodes signal fragments of physiological signals into images and classifies them by a Convolutional Neural Network (CNN). This study aims to study several encoding methods, including the Gramian Angular Summation/Difference Field method and Markov Transition Field, to evaluate the best way to encode signals into images in this domain. Such a study is performed on the NEURO dataset. Moreover, we investigate the usefulness of STREDWES in real scenarios by considering the SWELL dataset and a personalized approach. Finally, we compare the proposed approach with its competitors by considering the WESAD dataset. It outperforms the others.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data availability and materials
The data used in the publication is publicly available.
Code availability
The developed code for experiments related to SWELL-WK dataset are available for download at: https://github.com/ggerardlatek/STREDWES-SWELL, whereas the developed code for experiments related to WESAD and NEURO datasets can be provided by the Corresponding Author on reasonable request.
Notes
the tuning algorithm chooses the best scale for the hyperparameter exploration among linear, logarithmic, and reverse logarithmic.
the tuning algorithm chooses the best scale for the hyperparameter exploration among linear, logarithmic, and reverse logarithmic.
References
Birjandtalab, J., Cogan, D., Pouyan, M.B., Nourani, M. (2016). A non-eeg biosignals dataset for assessment and visualization of neurological status. In 2016 IEEE International Workshop on Signal Processing Systems (SiPS) (pp. 110–114). IEEE.
Chollet, F., et al. (2015). Keras. https://keras.io.
de Souza, A., Melchiades, M.B., Rigo, S.J., & Ramos, G.d.O. (2022). Mostress: A sequence model for stress classification. In 2022 International Joint Conference on Neural Networks (IJCNN) (pp. 1–8). IEEE.
Faouzi, J., & Janati, H. (2020). pyts: A python package for time series classification. The Journal of Machine Learning Research, 21, 1720–1725.
Garcia, G. R., Michau, G., Ducoffe, M., Gupta, J. S., & Fink, O. (2022). Temporal signals to images: Monitoring the condition of industrial assets with deep learning image processing algorithms. Proceedings of the Institution of Mechanical Engineers, Part O: Journal of Risk and Reliability, 236(4), 617–627.
Giannakakis, G., Grigoriadis, D., Giannakaki, K., Simantiraki, O., Roniotis, A., & Tsiknakis, M. (2019). Review on psychological stress detection using biosignals. IEEE Transactions on Affective Computing, 13, 440–460.
Girija, S.S. (2016). Tensorflow: Large-scale machine learning on heterogeneous distributed systems. Software available from https://tensorflow.org/39(9).
Gjoreski, M., Luštrek, M., Gams, M., & Gjoreski, H. (2017). Monitoring stress with a wrist device using context. Journal of Biomedical Informatics, 73, 159–170.
Healey, J. A., & Picard, R. W. (2005). Detecting stress during real-world driving tasks using physiological sensors. IEEE Transactions on Intelligent Transportation Systems, 6(2), 156–166.
Health and Safety Executive (2021). HSE on work-related stress. http://www.hse.gov.uk/statistics/causdis/-ffstress/index.htm. Accessed on March 7, 2022.
Jaiswal, M., Bara, C.P., Luo, Y., Burzo, M., Mihalcea, R., & Provost, E.M. (2020). Muse: a multimodal dataset of stressed emotion. In Proceedings of the Twelfth Language Resources and Evaluation Conference (pp. 1499–1510).
Kirschbaum, C., Pirke, K. M., & Hellhammer, D. H. (1993). The trier social stress test A tool for investigating psychobiological stress responses in a laboratory setting. Neuropsychobiology,28(1–2), 76–81.
Koldijk, S., Sappelli, M., Verberne, S., Neerincx, M.A., & Kraaij, W. (2014). The swell knowledge work dataset for stress and user modeling research. In Proceedings of the 16th international conference on multimodal interaction (pp. 291–298).
Lee, E. H. (2012). Review of the psychometric evidence of the perceived stress scale. Asian Nursing Research, 6(4), 121–127.
Li, R., & Liu, Z. (2020). Stress detection using deep neural networks. BMC Medical Informatics and Decision Making, 20, 1–10.
Lin, J., Pan, S., Lee, C.S., & Oviatt, S. (2019). An explainable deep fusion network for affect recognition using physiological signals. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management (pp. 2069–2072).
Lundberg, U., Kadefors, R., Melin, B., Palmerud, G., Hassmén, P., Engström, M., & Elfsberg Dohns, I. (1994). Psychophysiological stress and EMG activity of the trapezius muscle. International Journal of Behavioral Medicine, 1(4), 354–370.
Marwan, N., Romano, M. C., Thiel, M., & Kurths, J. (2007). Recurrence plots for the analysis of complex systems. Physics Reports, 438(5–6), 237–329.
McEwen, B. S. (1998). Protective and damaging effects of stress mediators. New England Journal of Medicine, 338(3), 171–179.
Oskooei, A., Chau, S.M., Weiss, J., Sridhar, A., Martínez, M.R., & Michel, B. (2021). Destress: deep learning for unsupervised identification of mental stress in firefighters from heart-rate variability (HRV) data. Explainable AI in Healthcare and Medicine: Building a Culture of Transparency and Accountability (pp. 93–105).
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019). Pytorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems, 32, 8026.
Piangerelli, M., Maestri, S., & Merelli, E. (2020). Visualising 2-simplex formation in metabolic reactions. Journal of Molecular Graphics and Modelling, 97, 107576.
Quadrini, M., Cavallin, M., Daberdaku, S., & Ferrari, C. (2021). Prosps: Protein sites prediction based on sequence fragments. In International Conference on Machine Learning, Optimization, and Data Science (pp. 568–580). Springer.
Quadrini, M., Daberdaku, S., & Ferrari, C. (2020). Hierarchical representation and graph convolutional networks for the prediction of protein–protein interaction sites. In International conference on machine learning, optimization, and data science (pp. 409–420). Springer.
Quadrini, M., Daberdaku, S., Blanda, A., Capuccio, A., Bellanova, L., & Gerard, G. (2022). Stress detection from wearable sensor data using gramian angular fields and CNN. In International Conference on Discovery Science (pp. 173–183). Springer.
Quadrini, M., Merelli, E., & Piergallini, R. (2019). Loop grammars to identify RNA structural patterns. In 10th international conference on bioinformatics models, methods and algorithms, BIOINFORMATICS 2019 - Part of 12th international joint conference on biomedical engineering systems and technologies, BIOSTEC 2019 (pp. 302–309).
Quadrini, M., Daberdaku, S., & Ferrari, C. (2022). Hierarchical representation for PPI sites prediction. BMC Bioinformatics, 23(1), 1–34.
Quadrini, M., Tesei, L., & Merelli, E. (2020). Aspralign: a tool for the alignment of RNA secondary structures with arbitrary pseudoknots. Bioinformatics, 36(11), 3578–3579.
Rastgoo, M. N., Nakisa, B., Maire, F., Rakotonirainy, A., & Chandran, V. (2019). Automatic driver stress level classification using multimodal deep learning. Expert Systems with Applications, 138, 112793.
Sabour, R.M., Benezeth, Y., De Oliveira, P., Chappe, J., & Yang, F. (2021). Ubfc-phys: A multimodal database for psychophysiological studies of social stress. IEEE Transactions on Affective Computing.
Sano, A., & Picard, R.W. (2013). Stress recognition using wearable sensors and mobile phones. In 2013 Humaine association conference on affective computing and intelligent interaction (pp. 671–676). IEEE.
Sasirekha, K., & Thangavel, K. (2020). A novel biometric image enhancement approach with the hybridization of undecimated wavelet transform and deep autoencoder. In Handbook of research on machine and deep learning applications for cyber security (pp. 245–269). IGI Global.
Schmidt, P., Reiss, A., Duerichen, R., Marberger, C., & Van Laerhoven, K. (2018). Introducing WESAD, a multimodal dataset for wearable stress and affect detection. In Proceedings of the 20th ACM international conference on multimodal interaction (pp. 400–408).
Sharma, K., Castellini, C., van den Broek, E. L., Albu-Schaeffer, A., & Schwenker, F. (2019). A dataset of continuous affect annotations and physiological signals for emotion analysis. Scientific Data, 6(1), 196.
Šikić, M., Tomić, S., & Vlahoviček, K. (2009). Prediction of protein-protein interaction sites in sequences and 3D structures by random forests. PLoS Computational Biology, 5(1), e1000278.
The American Institute of Stress. https://www.stress.org/daily-life. Accessed: 2023-02-15.
Verstraete, D., Ferrada, A., Droguett, E.L., Meruane, V., & Modarres, M. (2017). Deep learning enabled fault diagnosis using time-frequency image analysis of rolling element bearings. Shock and Vibration2017.
Wang, Z., & Oates, T. (2015a). Encoding time series as images for visual inspection and classification using tiled convolutional neural networks. In Workshops at the twenty-ninth AAAI conference on artificial intelligence.
Wang, Z., & Oates, T. (2015b). Imaging time-series to improve classification and imputation. In Twenty-Fourth International Joint Conference on Artificial Intelligence.
Xu, G., Liu, M., Jiang, Z., Shen, W., & Huang, C. (2019). Online fault diagnosis method based on transfer convolutional neural networks. IEEE Transactions on Instrumentation and Measurement, 69(2), 509–520.
Zeng, M., Zou, B., Wei, F., Liu, X., & Wang, L. (2016). Effective prediction of three common diseases by combining smote with tomek links technique for imbalanced medical data. In 2016 IEEE International Conference of Online Analysis and Computing Science (ICOACS) (pp. 225–228). IEEE.
Funding
This work has been funded by the European Union - NextGenerationEU under the Italian Ministry of University and Research (MUR) National Innovation Ecosystem grant ECS\(\_\)00000041 VITALITY - CUP J13C22000430001.
Author information
Authors and Affiliations
Contributions
All authors conceived the approach, STREDWES. MQ wrote the manuscript. AC, SD, AB and LB have implemented the draft of code. AC finalized the implementation. DF and AC performed the experiments. MQ and GG supervised the definition of approach and the experiments. MQ and GG reviewed the final version of the manuscript. All authors have read and approved the final manuscript.
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no competing interests.
Consent to participate
The authors provide the consent to publish the images in the manuscript. The data used in the publication is publicly available. We provide respective citations for each of the data sources.
Additional information
Editors: Dino Ienco, Roberto Interdonato, Pascal Poncelet.
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Appendices
Appendix A: Analysis of hyperparameters
The scatter plots computed on NEURO and WESAD datasets are shown in Figs. 8 and 9, respectively, such as window size, time step, image size, and batch size.
Scatter plot of the Accuracy the tested hyperparameters on NEURO dataset
Scatter plot of the Accuracy the tested hyperparameters on WESAD dataset
Appendix B: Confusion matrix and global metrics
In this Section, we show the confusion matrices that evaluates the methods based with GAFd, GAFs and MTF encoding methods on NEURO dataset in Fig. 10a–c. Such images have been classified by a CNN using the LOSOCV technique. Figure 10d show the confusion matrix that evaluates the Random Forest method. Figure 11 show the confusion matrix that evaluates our approach on the WESAD dataset by considering LOSOCV technique. Finally, Fig. 12, 13, and 14 show the values of accuracy, F1 score, precision, recall and specificity calculated on each subject of NEURO dataset considering GAFd, GAFs, MTF rapresentation and CNN method. Figure 15 show the value computed with Random Forest method on the Neuro dataset.
Confusion matrix that evaluates the CNN method with a GAFd, b GAFs, c MTF images d Random Forest method applied to NEURO dataset
Confusion Matrices that evaluates our approach on the WESAD dataset
Values of Accuracy, F1 score, Prediction, Recall and Specificity calculated on the single subject using GAFd encoding method and LOSOCV technique
Values of Accuracy, F1 score, Prediction, Recall and Specificity calculated on the single subject using GAFs encoding method and LOSOCV technique
Values of Accuracy, F1 score, Prediction, Recall and Specificity calculated on the single subject using MTF encoding method and LOSOCV technique
Values of Accuracy, F1 score, Prediction and Recall calculated on the single subject using Random Forest encoding method and LOSOCV technique on NEURO dataset
Appendix C: SWELL-KW
In this Section, we show the data entries built based on the SWELL-WB database for the "personalized" approach. Each data entry consists of images obtained by encoding signal fragments of a particular subject. Therefore, we have a data entry for each subject of the SWELL-KW database. Moreover, we did not consider subjects whose signals did not capture both stress and non-stress states. Such fragments are obtained with a sliding approach, long 112 s, and step equals 1 s. Table 9 show the number of images for each data entries, whereas Fig. 16 show the distribution of labels (stress/no stress) for each data entry. Figure 17 show the values of accuracy, F1 score, precision, recall and specificity calculated on the data entry related to each subject of SWELL-KW dataset.
Distribution of labels for each data entry associated with each subject of the SWELL-KW database. Label 0 indicates no stress, whereas Label 1 identifies stress situations
Values of Accuracy, F1 score, Prediction, Recall and Specificity calculated on the data entries of each subject of SWELL-KW database
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Quadrini, M., Capuccio, A., Falcone, D. et al. Stress detection with encoding physiological signals and convolutional neural network. Mach Learn (2024). https://doi.org/10.1007/s10994-023-06509-4
Received:
Revised:
Accepted:
Published:
DOI: https://doi.org/10.1007/s10994-023-06509-4