A Non-invasive Facial Visual-Infrared Stereo Vision Based Measurement as an Alternative for Physiological Measurement

  • Mohd Norzali Haji MohdEmail author
  • Masayuki Kashima
  • Kiminori Sato
  • Mutsumi Watanabe
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9009)


Our main aim is to propose a vision-based measurement as an alternative to physiological measurement for recognizing mental stress. The development of this emotion recognition system involved three stages: experimental setup for vision and physiological sensing, facial feature extraction in visual-thermal domain, mental stress stimulus experiment and data analysis and classification based on Support Vector Machine. In this research, 3 vision-based measurement and 2 physiological measurement were implemented in the system. Vision based measurement in facial vision domain consists of eyes blinking and in facial thermal domain consists 3 ROI’s temperature value and blood vessel volume at Supraorbital area. Two physiological measurement were done to measure the ground truth value which is heart rate and salivary amylase level. We also propose a new calibration chessboard attach with fever plaster to locate calibration point in stereo view. A new method of integration of two different sensors for detecting facial feature in both thermal and visual is also presented by applying nostril mask, which allows one to find facial feature namely nose area in thermal and visual domain. Extraction of thermal-visual feature images was done by using SIFT feature detector and extractor to verify the method of using nostril mask. Based on the experiment conducted, 88.6 % of correct matching was detected. In the eyes blinking experiment, almost 98 % match was detected successfully for without glasses and 89 % with glasses. Graph cut algorithm was applied to remove unwanted ROI. The recognition rate of 3 ROI’s was about 90 %–96 %. We also presented new method of automatic detection of blood vessel volume at Supraorbital monitored by LWIR camera. The recognition rate of correctly detected pixel was about 93 %. An experiment to measure mental stress by using the proposed system based on Support Vector Machine classification had been proposed and conducted and showed promising results.


Heart Rate Variability Mental Stress Bilateral Filter Epipolar Line Emotion Recognition System 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



This work was partially funded by MEXT/JSPS Kakenhi grant number: 50325768 and University Tun Hussein Onn Malaysia (UTHM). We would like to give special thanks to the laboratory members for their invaluable inputs and assistance.


  1. 1.
    Viola, P.A., Jones, M.J.: Rapid object detection using a boosted cascade of simple features. In: IEEE CVPR, pp. 511–518 (2001)Google Scholar
  2. 2.
    Sharma, N., Gedeon, T.: Objective measures, sensors and computational techniques for stress recognition and classification: a survey. Comput. Meth. Prog. Biomed. 3, 1287–1301 (2012). ElsevierCrossRefGoogle Scholar
  3. 3.
    Takai, N., Yamaguchi, M., et al.: Effect of physiology stress on the salivary cortisol and amylase levels in healthy young adults. Arch. Oral Biol. 49, 963–968 (2004). ElsevierCrossRefGoogle Scholar
  4. 4.
    Akhloufi, M., Bendada, A., Batsale, J.C.: State of the art in infrared face recognition. QIRT J., X-N \(^{\circ }\) X/2008, 1–24 (2008)Google Scholar
  5. 5.
    Levine, J., Pavlidis, I., Cooper, M.: The face of fear. Lancet 357, 1757 (2001)CrossRefGoogle Scholar
  6. 6.
    Hartley, R., Zisserman, A.: Multiple View Geometry in Computer Vision. Cambridge University Press, Cambridge (2000)zbMATHGoogle Scholar
  7. 7.
    Tan, E., Levine, J.: Human behavior: seeing through the face of deception. Nature 415, 6867 (2002)CrossRefGoogle Scholar
  8. 8.
    Merla, D.A., Tsiamyrtzis, P., Pavlidis, I.: Imaging facial signs of neurophysiological responses. IEEE Trans. Biomed. Eng. 56, 2 (2009)CrossRefGoogle Scholar
  9. 9.
    Lowe, D.G.: Distictive image features fromscale-invariant keypoints. Int. J. Comput. Vis. 60, 91–110 (2004)CrossRefGoogle Scholar
  10. 10.
    Shatri, D., Papadakis, M., Tsiamyrtzis, P.: Perinasal imaging of physiological stress and its affective potential. IEEE Trans. Affect. Comput. 3, 366–378 (2012)CrossRefGoogle Scholar
  11. 11.
    Chang, C.C., Lin, C.J.: LIBSVM: a library for support vector machines (2001).
  12. 12.
    Katsis, C.D., Katertsidis, N., et al.: Toward emotion recognition in car-racing drivers: a biosignal processing approach. IEEE Trans. Syst. Man Cybern. Part A: Syst. Hum 38, 502–512 (2008)CrossRefGoogle Scholar
  13. 13.
    Majumder, A., Bahera, L., Subramanian, V.K.: Automatic and robust detection of facial features in frontal face images. IEEE Trans. Syst. Man UKsim Int. Conf. Comput. Model Simul 3, 331–336 (2011)Google Scholar
  14. 14.
    Mohd, M.N.H., Kashima, M., Sato, K., Watanabe, M.: Thermal-visual facial feature extraction based on nostril mask. IAPR-Mach. Vis. Appl. 4–20, 113–116 (2013)Google Scholar
  15. 15.
    Mohd, M.N.H., Kashima, M., Sato, K., Watanabe, M.: Effective geometric calibration and facial feature extraction using multi sensors. Int. J. Eng. Sci. Innov. Technol. (IJESIT) 1–2, 170–178 (2012)Google Scholar
  16. 16.
    Wang, J.-G.: Facial feature extraction in an infrared image by proxy with visible face image. IEEE Trans. Affect. Comput. 3–3, 2057–2066 (2007)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Mohd Norzali Haji Mohd
    • 1
    • 2
    Email author
  • Masayuki Kashima
    • 1
  • Kiminori Sato
    • 1
  • Mutsumi Watanabe
    • 1
  1. 1.Department of Information Science and Biomedical Engineering, Graduate School of Sciences and EngineeringKagoshima UniversityKagoshimaJapan
  2. 2.Department of Computer Engineering, Faculty of Electrical and Electronic EngineeringUniversity Tun Hussein Onn Malaysia (UTHM)Parit Raja, Batu PahatMalaysia

Personalised recommendations