Skip to main content

Advertisement

Log in

What Happens in Face During a Facial Expression? Using Data Mining Techniques to Analyze Facial Expression Motion Vectors

  • Published:
Information Systems Frontiers Aims and scope Submit manuscript

Abstract

Automatic facial expression recognition is a big challenge in human–computer interaction. Analyzing the changes in the face during a facial expression can be used for this purpose. In this paper, these changes are extracted as a number of motion vectors. These motion vectors are extracted using an optical flow algorithm. Then, they are used to analyze facial expressions by some of the data mining algorithms. This analysis has not only determined what changes occur in the face during facial expression but has also been used to recognize facial expressions. Cohen-Kanade facial expression dataset was used in this research. Based on our findings, the vertical lengths of motion vectors created in the lower part of the face have the greatest impact on the classification of facial expressions. Among the investigated classification algorithms, deep learning, support vector machine, and C5.0 had better performance, yielding an accuracy of 95.3%, 92.8%, and 90.2% respectively.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Data Availability

The data that support the findings of this study are available from the corresponding author upon request.

References

  • Abdellatif, D., El Moutaouakil, K., & Satori, K. (2018). Clustering and Jarque-Bera normality test to face recognition. Procedia Computer Science, 127, 246–255.

    Article  Google Scholar 

  • Abdollahi, M., Jafarizadeh, A., Asbagh, A. G., Sobhi, N., Pourmoghtader, K., Pedrammehr, S., Asadi, H., Alizadehsani, R., Tan, R.-S., & Acharya, U. R. (2023). Artificial intelligence in assessing cardiovascular diseases and risk factors via retinal fundus images: a review of the last decade. arXiv preprint arxiv.org/abs/2311.07609. Accessed 1/1/2024

  • Abdullah, S. M. S., & Abdulazeez, A. M. (2021). Facial expression recognition based on deep learning convolution neural network: A review. Journal of Soft Computing and Data Mining, 2(1), 53–65.

    Google Scholar 

  • Abdulrahman, M., & Eleyan, A. (2015). Facial expression recognition using support vector machines. 2015 23rd Signal Processing and Communications Applications Conference (SIU), (pp. 276-279). IEEE.

  • Abdulrazaq, M. B., Mahmood, M. R., Zeebaree, S. R., Abdulwahab, M. H., Zebari, R. R., & Sallow, A. B. (2021). An analytical appraisal for supervised classifiers’ performance on facial expression recognition based on relief-F feature selection. Journal of Physics: Conference Series, 1804(1), 012055. 

  • Alenazy, W. M., & Alqahtani, A. S. (2021). Gravitational search algorithm based optimized deep learning model with diverse set of features for facial expression recognition. Journal of Ambient Intelligence and Humanized Computing, 12(2), 1631–1646.

    Article  Google Scholar 

  • Alexandre, G. R., Soares, J. M., & Thé, G. A. P. (2020). Systematic review of 3D facial expression recognition methods. Pattern Recognition, 100, 107108.

    Article  Google Scholar 

  • Alizadehsani, R., Habibi, J., Sani, Z. A., Mashayekhi, H., Boghrati, R., Ghandeharioun, A., & Bahadorian, B. (2012). Diagnosis of coronary artery disease using data mining based on lab data and echo features. Journal of Medical and Bioengineering, 1(1), 26.

    Article  Google Scholar 

  • Alizadehsani, R., Sharifrazi, D., Izadi, N. H., Joloudari, J. H., Shoeibi, A., Gorriz, J. M., Hussain, S., Arco, J. E., Sani, Z. A., & Khozeimeh, F. (2021). Uncertainty-aware semi-supervised method using large unlabeled and limited labeled COVID-19 data. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), 17(3s), 1–24.

    Article  Google Scholar 

  • Asadifard, M., & Shanbezadeh, J. (2010). Automatic adaptive center of pupil detection using face detection and cdf analysis. Proceedings of the International Multiconference of Engineers and Computer Scientists, 1, 3–9.

  • Banerjee, R., De, S., & Dey, S. (2021). A novel facial expression recognition system using BMCSA based adaptive neuro-fuzzy inference system. International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems, 29(06), 791–813.

    Article  Google Scholar 

  • Bao, P. T., Nguyen, H., & Nhan, D. (2009). A new approach to mouth detection using neural network. In 2009 IITA International Conference on Control, Automation and Systems Engineering (case 2009), pp. 616-619, IEEE.

  • Barman, A., & Dutta, P. (2019). Facial expression recognition using distance and texture signature relevant features. Applied Soft Computing, 77, 88–105.

    Article  Google Scholar 

  • Barua, P. D., Baygin, N., Dogan, S., Baygin, M., Arunkumar, N., Fujita, H., Tuncer, T., Tan, R.-S., Palmer, E., Azizan, M. M. B., Kadri, N. A., & Acharya, U. R. (2022). Automated detection of pain levels using deep feature extraction from shutter blinds-based dynamic-sized horizontal patches with facial images. Scientific Reports, 12(1), 17297. https://doi.org/10.1038/s41598-022-21380-4

    Article  Google Scholar 

  • Bassili, J. N. (1978). Facial motion in the perception of faces and of emotional expression. Journal of Experimental Psychology: Human Perception and Performance, 4(3), 373.

    Google Scholar 

  • Baygin, M., Tuncer, I., Dogan, S., Barua, P. D., Tuncer, T., Cheong, K. H., & Acharya, U. R. (2023). Automated facial expression recognition using exemplar hybrid deep feature generation technique. Soft Computing, 27(13), 8721–8737. https://doi.org/10.1007/s00500-023-08230-9

    Article  Google Scholar 

  • Bishop, C. M. (2006). Pattern recognition. Machine Learning, 4(4), 1–30. New York: Springer.

  • Brown, M. T., & Wicker, L. R. (2000). 8 - Discriminant Analysis. In H. E. A. Tinsley & S. D. Brown (Eds.), Handbook of Applied Multivariate Statistics and Mathematical Modeling (pp. 209–235). Academic Press. https://doi.org/10.1016/B978-012691360-6/50009-4

  • Byun, S.-W., & Lee, S.-P. (2020). Human emotion recognition based on the weighted integration method using image sequences and acoustic features. Multimedia Tools and Applications, 80, 35871–35885.

  • Calvo, M. G., & Nummenmaa, L. (2016). Perceptual and affective mechanisms in facial expression recognition: An integrative review. Cognition and Emotion, 30(6), 1081–1106.

    Article  Google Scholar 

  • Durmuşoğlu, A., & Kahraman, Y. (2021). Face expression recognition using a combination of local binary patterns and local phase quantization. 2021 International Conference on Communication, Control and Information Sciences (ICCISc), 1, 1–5. IEEE.

  • Fernandez, P. D. M., Pena, F. A. G., Ren, T. I., & Cunha, A. (2019). Feratt: Facial expression recognition with attention net arXiv preprint arXiv:1902.03284, 3. Accessed 1/1/2024

  • Gautama, T., & Hulle, M. A. V. (2002). A phase-based approach to the estimation of the optical flow field using spatial filtering. IEEE Transactions on Neural Networks, 13(5), 1127–1136. https://doi.org/10.1109/TNN.2002.1031944

    Article  Google Scholar 

  • Gogić, I., Manhart, M., Pandžić, I. S., & Ahlberg, J. (2020). Fast facial expression recognition using local binary features and shallow neural networks. The Visual Computer, 36(1), 97–112.

    Article  Google Scholar 

  • Gruber, P., Stadlthanner, K., Böhm, M., Theis, F. J., Lang, E. W., Tomé, A. M., Teixeira, A. R., Puntonet, C. G., & Saéz, J. G. (2006). Denoising using local projective subspace methods. Neurocomputing, 69(13–15), 1485–1501.

    Article  Google Scholar 

  • Happy, S., & Routray, A. (2015). Robust facial expression classification using shape and appearance features. 2015 Eighth International Conference on Advances in Pattern Recognition (ICAPR), pp. 1–5, IEEE

  • Holder, R. P., & Tapamo, J. R. (2017). Improved gradient local ternary patterns for facial expression recognition. EURASIP Journal on Image and Video Processing, 2017(1), 1–15.

    Article  Google Scholar 

  • Kamarol, S. K. A., Jaward, M. H., Kälviäinen, H., Parkkinen, J., & Parthiban, R. (2017). Joint facial expression recognition and intensity estimation based on weighted votes of image sequences. Pattern Recognition Letters, 92, 25–32.

    Article  Google Scholar 

  • Kanade, T., Cohn, J. F., & Yingli, T. (2000). Comprehensive database for facial expression analysis. Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580), pp. 46–53, IEEE.

  • Kirana, K. C., Wibawanto, S., & Herwanto, H. W. (2018). Facial emotion recognition based on viola-jones algorithm in the learning environment. 2018 International Seminar on Application for Technology of Information and Communication, pp. 406–410, IEEE.

  • LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. Nature, 521(7553), 436–444.

    Article  Google Scholar 

  • Li, S., & Deng, W. (2020). Deep facial expression recognition: A survey. IEEE Transactions on Affective Computing, 13(3), 1195–1215.

  • Li, W., Hua, Y., & Liangzheng, X. (2007). Mouth detection based on interest point. 2007 Chinese Control Conference, pp. 610–613, IEEE.

  • Li, Y., Mavadati, S. M., Mahoor, M. H., Zhao, Y., & Ji, Q. (2015). Measuring the intensity of spontaneous facial action units with dynamic Bayesian network. Pattern Recognition, 48(11), 3417–3427.

    Article  Google Scholar 

  • Liliana, D. Y., Basaruddin, T., Widyanto, M. R., & Oriza, I. I. D. (2019). Fuzzy emotion: a natural approach to automatic facial expression recognition from psychological perspective using fuzzy system. Cognitive Processing, 20(4), 391–403. https://doi.org/10.1007/s10339-019-00923-0

    Article  Google Scholar 

  • Loh, W. Y. (2011). Classification and regression trees. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 1(1), 14–23.

    Google Scholar 

  • Lu, Y., Wang, S., Zhao, W., & Zhao, Y. (2019). WGAN-based robust occluded facial expression recognition. IEEE Access, 7, 93594–93610.

    Article  Google Scholar 

  • Lucey, P., Cohn, J. F., Kanade, T., Saragih, J., Ambadar, Z., & Matthews, I. (2010). The Extended Cohn-Kanade Dataset (CK+): A complete dataset for action unit and emotion-specified expression. 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition – Workshops, pp. 94–101, IEEE.

  • Mehrabian, A. (2017). Communication without words. Communication theory (pp. 193–200). Routledge.

    Chapter  Google Scholar 

  • Mehta, D., Siddiqui, M. F. H., & Javaid, A. Y. (2019). Recognition of emotion intensities using machine learning algorithms: A comparative study. Sensors, 19(8), 1897.

    Article  Google Scholar 

  • Minaee, S., Minaei, M., & Abdolrashidi, A. (2021). Deep-emotion: Facial expression recognition using attentional convolutional network. Sensors, 21(9), 3046.

    Article  Google Scholar 

  • Mliki, H., Fourati, N., Hammami, M., & Ben-Abdallah, H. (2013). Data mining-based facial expressions recognition system. In SCAI, pp. 185–194 (book).

  • Mliki, H., Fourati, N., Smaoui, S., & Hammami, M. (2013). Automatic facial expression recognition system. 2013 ACS International Conference on Computer Systems and Applications (AICCSA), pp. 1–6, IEEE.

  • Naghsh-Nilchi, A. R., & Roshanzamir, M. (2006). An efficient algorithm for motion detection based facial expression recognition using optical flow. Proceedings of the World Academy of Science, Engineering and Technology, 20, 23-28.

  • Nahavandi, S., Alizadehsani, R., Nahavandi, D., Lim, C. P., Kelly, K., & Bello, F. (2024). Machine learning meets advanced robotic manipulation. Information Fusion, 105, 102221.

  • Nahavandi, S., Alizadehsani, R., Nahavandi, D., Mohamed, S., Mohajer, N., Rokonuzzaman, M., & Hossain, I. (2022). A comprehensive review on autonomous navigation. arXiv preprint arxiv.org/abs/2212.12808. Accessed 1/1/2024

  • Nasab, R. Z., Ghamsari, M. R. E., Argha, A., Macphillamy, C., Beheshti, A., Alizadehsani, R., Lovell, N. H., Lotfollahi, M., & Alinejad-Rokny, H. (2022). Deep learning in spatially resolved transcriptomics: A comprehensive technical view. arXiv preprint arxiv.org/abs/2210.04453. Accessed 1/1/2024

  • Nasarian, E., Alizadehsani, R., Acharyac, U. R., & Tsui, d. K.-L. (2023). Designing interpretable ML system to enhance trustworthy ai in healthcare: a systematic review of the last decade to a proposed robust framework. arXiv preprint arxiv.org/abs/2311.11055. Accessed 1/1/2024

  • Noble, W. S. (2006). What is a support vector machine? Nature Biotechnology, 24(12), 1565–1567.

    Article  Google Scholar 

  • Pang, S.-L., & Gong, J.-Z. (2009). C5.0 classification algorithm and application on individual credit evaluation of banks. Systems Engineering - Theory & Practice, 29(12), 94–104. https://doi.org/10.1016/S1874-8651(10)60092-0

    Article  Google Scholar 

  • Qin, S., Zhu, Z., Zou, Y., & Wang, X. (2020). Facial expression recognition based on Gabor wavelet transform and 2-channel CNN. International Journal of Wavelets, Multiresolution and Information Processing, 18(02), 2050003.

    Article  Google Scholar 

  • Quinlan, J. R. (1986). Induction of decision trees. Machine Learning, 1(1), 81–106. https://doi.org/10.1007/BF00116251

    Article  Google Scholar 

  • Rahul, M., Kohli, N., & Agarwal, R. (2021a). Facial expression recognition using local multidirectional score pattern descriptor and modified hidden Markov model. International Journal of Advanced Intelligence Paradigms, 18(4), 538–551.

    Article  Google Scholar 

  • Rahul, M., Shukla, R., Goyal, P. K., Siddiqui, Z. A., & Yadav, V. (2021). Gabor Filter and ICA-Based Facial Expression Recognition Using Two-Layered Hidden Markov Model. In Advances in Computational Intelligence and Communication Technology (pp. 511–518). Springer.

  • Roshanzamir, M., Naghsh Nilchi, A. R., & Roshanzamir, M. (2015). A new fuzzy rule-based approach for automatic facial expression recognition 1st National Conference on Soft Computing, pp.1–10.

  • Roshanzamir, M., Roshanzamir, M., Mirzaei, A., Darbandy, M. T., Shoeibi, A., Alizadehsani, R., Khozeimeh, F., & Khosravi, A. (2022). Automatic facial expression recognition in an image sequence using conditional random field. 2022 IEEE 22nd International Symposium on Computational Intelligence and Informatics and 8th IEEE International Conference on Recent Achievements in Mechatronics, Automation, Computer Science and Robotics (CINTI-MACRo), pp. 000271–000278, IEEE.

  • Sadeghi, Z., Alizadehsani, R., Cifci, M. A., Kausar, S., Rehman, R., Mahanta, P., Bora, P. K., Almasri, A., Alkhawaldeh, R. S., & Hussain, S. (2023). A brief review of explainable artificial intelligence in healthcare. arXiv preprint arXiv:2304.01543.http://arxiv.org/abs/2304.01543

  • Saurav, S., Singh, S., Saini, R., & Yadav, M. (2020). Facial expression recognition using improved adaptive local ternary pattern. In Proceedings of 3rd International Conference on Computer Vision and Image Processing, 2, 39–52. Springer Singapore.

  • Savran, A., Sankur, B., & Bilge, M. T. (2012). Regression-based intensity estimation of facial action units. Image and Vision Computing, 30(10), 774–784.

    Article  Google Scholar 

  • Segovia, F., Górriz, J., Ramírez, J., Salas-Gonzalez, D., & Álvarez, I. (2013). Early diagnosis of Alzheimer’s disease based on partial least squares and support vector machine. Expert Systems with Applications, 40(2), 677–683.

    Article  Google Scholar 

  • Sharifrazi, D., Alizadehsani, R., Hoseini Izadi, N., Roshanzamir, M., Shoeibi, A., Khozeimeh, F., Alizadeh Sani, F., Alizadeh Sani, Z., Hussain, S., Harlapur, C., Gorriz, J.M., & Khosravi, A. (2024). Hypertrophic Cardiomyopathy Diagnosis Using Deep Learning Techniques, 14(6), 1–21.

  • Shoeibi, A., Ghassemi, N., Khodatars, M., Moridian, P., Alizadehsani, R., Zare, A., Khosravi, A., Subasi, A., Acharya, U. R., & Gorriz, J. M. (2022). Detection of epileptic seizures on EEG signals using ANFIS classifier, autoencoders and fuzzy entropies. Biomedical Signal Processing and Control, 73, 103417.

    Article  Google Scholar 

  • Sonka, M., Hlavac, V., & Boyle, R. (2014). Image processing, analysis, and machine vision. Cengage Learning.

    Google Scholar 

  • Sultan Zia, M., Hussain, M., & Arfan Jaffar, M. (2018). A novel spontaneous facial expression recognition using dynamically weighted majority voting based ensemble classifier. Multimedia Tools and Applications, 77(19), 25537–25567.

    Article  Google Scholar 

  • Sun, W., Zhao, H., & Jin, Z. (2017). An efficient unconstrained facial expression recognition algorithm based on stack binarized auto-encoders and binarized neural networks. Neurocomputing, 267, 385–395.

    Article  Google Scholar 

  • Sun, W., Zhao, H., & Jin, Z. (2018). A visual attention based ROI detection method for facial expression recognition. Neurocomputing, 296, 12–22.

    Article  Google Scholar 

  • Sut, N., & Simsek, O. (2011). Comparison of regression tree data mining methods for prediction of mortality in head injury. Expert Systems with Applications, 38(12), 15534–15539. https://doi.org/10.1016/j.eswa.2011.06.006

    Article  Google Scholar 

  • Tan, P.-N., Steinbach, M., & Kumar, V. (2016). Introduction to data mining. Pearson Education India.

    Google Scholar 

  • The SAGE Encyclopedia of Theory in Psychology. (2016). In. SAGE Publications, Inc. https://doi.org/10.4135/9781483346274

  • Tripathi, P., Verma, K., Verma, L., & Parveen, N. (2013). Facial Expression Recognition Using Data Mining Algorithm. Journal of Economics, Business and Management, 1(4), 343–346.

    Google Scholar 

  • Tuncer, T., Dogan, S., & Subasi, A. (2021). A new fractal pattern feature generation function based emotion recognition method using EEG. Chaos, Solitons & Fractals, 144, 110671. https://doi.org/10.1016/j.chaos.2021.110671

    Article  Google Scholar 

  • Tuncer, T., Dogan, S., & Subasi, A. (2023). Automated facial expression recognition using novel textural transformation. Journal of Ambient Intelligence and Humanized Computing, 14(7), 9435–9449. https://doi.org/10.1007/s12652-023-04612-x

    Article  Google Scholar 

  • Turan, C., & Lam, K.-M. (2018). Histogram-based local descriptors for facial expression recognition (FER): A comprehensive study. Journal of Visual Communication and Image Representation, 55, 331–341.

    Article  Google Scholar 

  • Valstar, M. F., Mehu, M., Jiang, B., Pantic, M., & Scherer, K. (2012). Meta-analysis of the first facial expression recognition challenge. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 42(4), 966–979. https://doi.org/10.1109/TSMCB.2012.2200675

    Article  Google Scholar 

  • Van der Maaten, L., & Hinton, G. (2008). Visualizing data using t-SNE. Journal of Machine Learning Research, 9(11), 2579–2605.

  • Varma, S., Shinde, M., & Chavan, S. S. (2020). Analysis of pca and lda features for facial expression recognition using svm and hmm classifiers. Techno-Societal 2018 (pp. 109–119). Springer.

    Chapter  Google Scholar 

  • Vasanth, P., & Nataraj, K. (2015). Facial expression recognition using SVM classifier. Indonesian Journal of Electrical Engineering and Informatics (IJEEI), 3(1), 16–20.

    Google Scholar 

  • Wang, C., Wang, S., & Liang, G. (2019). Identity-and pose-robust facial expression recognition through adversarial feature learning. Proceedings of the 27th ACM International Conference on Multimedia, pp. 238–246.

  • Wang, L., Wang, K., & Li, R. (2015). Unsupervised feature selection based on spectral regression from manifold learning for facial expression recognition. IET Computer Vision, 9(5), 655–662.

    Article  Google Scholar 

  • Wang, Q., Yang, S., & Li, X. w. (2009). A fast mouth detection algorithm based on face organs. 2009 2nd International Conference on Power Electronics and Intelligent Transportation System (PEITS).

  • Xiong, J., Yang, J., Yan, L., Awais, M., Khan, A. A., Alizadehsani, R., & Acharya, U. R. (2024). Efficient reinforcement learning-based method for plagiarism detection boosted by a population-based algorithm for pretraining weights. Expert Systems with Applications, 238, 122088.

    Article  Google Scholar 

  • Xu, X., Quan, C., & Ren, F. (2015). Facial expression recognition based on Gabor Wavelet transform and Histogram of Oriented Gradients. 2015 IEEE International Conference on Mechatronics and Automation (ICMA), pp. 2117–2122, IEEE.

  • Zhao, X., & Zhang, S. (2016). A review on facial expression recognition: Feature extraction and classification. IETE Technical Review, 33(5), 505–517.

    Article  Google Scholar 

Download references

Funding

There is no funding.

Author information

Authors and Affiliations

Authors

Contributions

Conceptualization, Mo.R., R.A., M.J. and Ma.R.; methodology, A.S., J.M.G. and A.K.; software, Mo.R., R.A., M.J. and Ma.R.; validation, A.S., J.M.G. and A.K.; formal analysis, Mo.R., R.A. and Ma.R.; investigation, S.N. and U.R.A.; resources, A.S., J.M.G. and A.K.; data curation, Mo.R., R.A., M.J. and Ma.R.; writing—original draft preparation, Mo.R., R.A., M.J. and Ma.R.; visualization, J.M.G. and A.K.; supervision, S.N. and U.R.A.

Corresponding author

Correspondence to Mahboobeh Jafari.

Ethics declarations

Ethics Approval and Consent to Participate

Not applicable.

Consent for Publication

Not applicable.

Competing Interests

The authors have no competing interests to declare.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Roshanzamir, M., Jafari, M., Alizadehsani, R. et al. What Happens in Face During a Facial Expression? Using Data Mining Techniques to Analyze Facial Expression Motion Vectors. Inf Syst Front (2024). https://doi.org/10.1007/s10796-023-10466-7

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10796-023-10466-7

Keywords

Navigation