Abstract
This paper presents a contactless system based on twin channels of thermal and visual image sequences to register the affective states of an individual during Human–Computer Interaction (HCI). The negative affective states such as stress, anxiety, and depression in students have raised significant concerns. The first phase obtains the dominant emotional state by an ensemble of cues from visual and thermal facial images using a newly proposed cascaded Convolutional Neural Network (CNN) model named as EmoScale. The second phase clusters a sequence of the obtained emotional states using a trained Hidden Markov model (HMM) as one of the three affective states anxiety, depression, and stress. We perform fivefold cross-validation of EmoScale on our self-prepared dataset. The performance of the second phase is compared with a standard Depression Anxiety Stress Scale (DASS) and the results are found to be promising.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Zeng, Z., Pantic, M., Roisman, G.I., Huang, T.S.: A survey of affect recognition methods: audio, visual, and spontaneous expressions. IEEE Tran. Pattern Analy. Mach. Intell. 31(1), 39–58 (2008)
Zheng, W.-L., Liu, W., Lu, Y., Lu, B.-L., Cichocki, A.: Emotionmeter: a multimodal framework for recognizing human emotions. IEEE Trans. Cyber. 49(3), 1110–1122 (2018)
Lovibond, P.F., Lovibond, S.H.: The structure of negative emotional states: comparison of the depression anxiety stress scales (dass) with the beck depression and anxiety inventories. Behav. Res. Therapy 33(3), 335–343 (1995)
Rymaszewska, J., Kiejna, A., Hadryś, T.: Depression and anxiety in coronary artery bypass grafting patients. Euro. Psych. 18(4), 155–160 (2003)
Basu, A., Dasgupta, A., Thyagharajan, A., Routray, A., Guha, R., Mitra, P.: A portable personality recognizer based on affective state classification using spectral fusion of features. IEEE Trans. Affect. Comput. 9(3), 330–342 (2018)
Happy, S., Routray, A.: Automatic facial expression recognition using features of salient facial patches. IEEE Trans. Affec. Comput. 6(1), 1–12 (2014)
Nayak, S., Happy, S., Routray, A., Sarma, M.: A versatile online system for person-specific facial expression recognition. In: TENCON 2019-2019 IEEE Region 10 Conference (TENCON). IEEE, pp. 2513–2518 (2019)
Khan, M.M., Ward, R.D., Ingleby, M.: Classifying pretended and evoked facial expressions of positive and negative affective states using infrared measurement of skin temperature. ACM Trans. Appl. Percept. (TAP) 6(1), 6 (2009)
Marquez, E.S., Hare, J.S., Niranjan, M.: Deep cascade learning. IEEE Trans. Neural Netw. Learn. Syst. 29(11), 5475–5485 (2018)
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1–9 (2015)
Lovibond, S.H., Lovibond, P.F.: Manual for the depression anxiety stress scales. Psychology Foundation of Australia (1996)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Nayak, S., Sharma, V., Panda, S.K., Uttarkabat, S. (2021). Affective State Analysis Through Visual and Thermal Image Sequences. In: Hassanien, A.E., Bhattacharyya, S., Chakrabati, S., Bhattacharya, A., Dutta, S. (eds) Emerging Technologies in Data Mining and Information Security. Advances in Intelligent Systems and Computing, vol 1286. Springer, Singapore. https://doi.org/10.1007/978-981-15-9927-9_7
Download citation
DOI: https://doi.org/10.1007/978-981-15-9927-9_7
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-15-9926-2
Online ISBN: 978-981-15-9927-9
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)