Abstract
The growth of machine learning and artificial intelligence has made it possible for automatic lie detection systems to emerge. These can be based on a variety of cues, such as facial features. However, there is a lack of knowledge about both the development and the accuracy of such systems. To address this lack, we conducted a review of studies that have investigated automatic lie detection systems by using facial features. Our analysis of twenty-eight eligible studies focused on four main categories: dataset features, facial features used, classifier features and publication features. Overall, the findings showed that automatic lie detection systems rely on diverse technologies, facial features, and measurements. They are mainly based on factual lies, regardless of the stakes involved. On average, these automatic systems were based on a dataset of 52 individuals and achieved an average accuracy ranging from 61.87% to 72.93% in distinguishing between truth-tellers and liars, depending on the types of classifiers used. However, although the leakage hypothesis was the most used explanatory framework, many studies did not provide sufficient theoretical justification for the choice of facial features and their measurements. Bridging the gap between psychology and the computational-engineering field should help to combine theoretical frameworks with technical advancements in this area.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
The iBorderCtrl and AVATAR systems consist of several subsystems, one of which is dedicated to lie detection. The lie detection subsystems have been the subject of experimental studies. For iBorderCtrl, three studies are included in the current review dataset. For AVATAR, two studies were screened but not selected for the final dataset of the current review. See the Study selection and characteristics section for more details.
This moderate effect-size estimate was based on a very small number of studies (DePaulo et al., 2003).
The effect-size, calculated here by Cohen's d, gives an indication of the size of the difference observed between two groups, here people lying and telling the truth. When |d| ≈ 0.20 the difference is considered as small (not visible to the naked eye), when |d| ≈ 0.50 the difference is considered as medium, then when |d| ≈ or > 0.80 the difference is considered as large (Cohen, 1988).
Authors mostly reported the accuracy in their studies. However, it is not possible to conduct a meta-analysis using only the accuracy value of the classifiers. Indeed, to perform a meta-analysis in the context of classification analyses, the confusion matrix obtained from the results of the classifier is needed. The confusion matrix allows the calculation of the log odds ratio, and the associated standard error (Cooper et al., 2009). However, the confusion matrices are not available in the studies selected, making a meta-analytic approach impossible.
References
Avola, D., Cinque, L., Foresti, G. L., Pannone, D. (2019). Automatic deception detection in RGB videos using facial action units. In Proceedings of the 13th international conference on distributed smart cameras, Article 5. doi: https://doi.org/10.1145/3349801.3349806
Avola, D., Cascio, M., Cinque, L., Fagioli, A., & Foresti, G. L. (2021). LieToMe: An ensemble approach for deception detection from facial cues. International Journal of Neural Systems, 31(2), 2050068. https://doi.org/10.1142/S0129065720500689
Bablani, A., Edla, D. R., Kupilli, V., & Dharavath, R. (2021). Lie detection using fuzzy ensemble approach with novel defuzzification method for classification of EEG signals. IEEE Transactions on Instrumentation and Measurement, 70, 1–13. https://doi.org/10.1109/TIM.2021.3082985
Baltrušaitis, T., Robinson, P., & Morency, L.-P. (2016). OpenFace: An open source facial behavior analysis toolkit. IEEE Winter Conference on Applications of Computer Vision (WACV), 2016, 1–10. https://doi.org/10.1109/WACV.2016.7477553
Barathi, C. S. (2016). Lie detection based on facial micro expression body language and speech analysis. International Journal of Engineering Research & Technology.
Bartlett, M. S., Littlewort, G. C., Frank, M. G., & Lee, K. (2014). Automatic decoding of facial movements reveals deceptive pain expressions. Current Biology: CB, 24(7), 738–743. https://doi.org/10.1016/j.cub.2014.02.009
Bedoya-Echeverry, S., Belalcázar-Ramírez, H., Loaiza-Correa, H., Nope-Rodríguez, S. E., Pinedo-Jaramillo, C. R., & Restrepo-Girón, A. D. (2017). Detection of lies by facial thermal imagery analysis. Revista De La Facultad De Ingenieria, 26(44), 47–59. https://doi.org/10.19053/01211129.v26.n44.2017.5771
Bertolino, N., Ferraro, S., Nigri, A., Bruzzone, M. G., Ghielmetti, F., Coma Research Centre (CRC)-Besta Institute. (2014). A neural network approach to fMRI binocular visual rivalry task analysis. PloS One, 9(8), e105206. https://doi.org/10.1371/journal.pone.0105206
Bhamare, A. R., Katharguppe, S., & Silviya Nancy, J. (2020). Deep neural networks for lie detection with attention on bio-signals. In 2020 7th international conference on soft computing & machine intelligence (ISCMI), (pp. 143–147). https://doi.org/10.1109/ISCMI51676.2020.9311575
Bishay, M., Preston, K., Strafuss, M., Page, G., Turcot, J., & Mavadati, M. (2023). AFFDEX 2.0: A real-time facial expression analysis toolkit. In 2023 IEEE 17th international conference on automatic face and gesture recognition (FG), (pp. 1–8). https://doi.org/10.1109/FG57933.2023.10042673
Blandón-Gitlin, I., Fenn, E., Masip, J., & Yoo, A. H. (2014). Cognitive-load approaches to detect deception: Searching for cognitive mechanisms. Trends in Cognitive Sciences, 18(9), 441–444. https://doi.org/10.1016/j.tics.2014.05.004
Blume, J. H., & Helm, R. K. (2014). The unexonerated: Factually innocent defendants who plead guilty. Cornell Law Review, 100, 157.
Bond, C. F., & Depaulo, B. M. (2008). Individual differences in judging deception: Accuracy and bias. Psychological Bulletin, 134(4), 477–492. https://doi.org/10.1037/0033-2909.134.4.477
Burger, W., & Burge, M. J. (2016). Scale-Invariant Feature Transform (SIFT). In W. Burger & M. J. Burge (Eds.), Digital Image Processing: An Algorithmic Introduction Using Java (pp. 609–664). London: Springer.
Burgoon, J. K. (2018). Microexpressions are not the best way to catch a liar. Frontiers in Psychology, 9, 20. https://doi.org/10.3389/fpsyg.2018.01672
Carissimi, N., Beyan, C., & Murino, V. (2018). A multi-view learning approach to deception detection. In 2018 13th IEEE international conference on automatic face gesture recognition (FG 2018), (pp. 599–606). https://doi.org/10.1109/FG.2018.00095
Chebbi, S., & Ben Jebara, S. (2020). An audio-visual based feature level fusion approach applied to deception detection. In Proceedings of the 15th international joint conference on computer vision, imaging and computer graphics theory and applications. 15th international conference on computer vision theory and applications, Valletta, Malta. https://doi.org/10.5220/0008896201970205
Chen, H. (2002). From digital library to digital government: A case study in crime data mapping and mining. Digital libraries: people, knowledge, and technology, (pp. 36–52). https://doi.org/10.1007/3-540-36227-4_4
Chengeta, K. (2019). Automated facial micro-expression recognition using local binary patterns on three orthogonal planes with boosted classifiers: A survey. In Advances in computer communication and computational sciences, (pp. 671–686). https://doi.org/10.1007/978-981-13-6861-5_57
Cohen, J. (1988). Statistical power analysis for the behavioral sciences. Academic Press. https://doi.org/10.1016/B978-0-12-179060-8.50012-8
Cooper, H., Hedges, L. V., & Valentine, J. C. (2009). The handbook of research synthesis and meta-analysis. Russell Sage Foundation.
Craig, K. D., Hyde, S. A., & Patrick, C. J. (1991). Genuine, suppressed and faked facial behavior during exacerbation of chronic low back pain. Pain, 46(2), 161–171. https://doi.org/10.1016/0304-3959(91)90071-5
Crockett, K., O’Shea, J., & Khan, W. (2020). Automated deception detection of males and females from non-verbal facial micro-gestures. International Joint Conference on Neural Networks (IJCNN), 2020, 1–7. https://doi.org/10.1109/IJCNN48605.2020.9207684
Dalal, N., & Triggs, B. (2005). Histograms of oriented gradients for human detection. In 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR’05) (Vol. 1, pp. 886–893). https://doi.org/10.1109/CVPR.2005.177
Dcosta, M., Shastri, D., Vilalta, R., Burgoon, J. K., & Pavlidis, I. (2015). Perinasal indicators of deceptive behavior. In 2015 11th IEEE international conference and workshops on automatic face and gesture recognition (FG), (Vol 1, pp. 1–8). https://doi.org/10.1109/FG.2015.7163080
Delmas, H. (2018). Expressions faciales et mensonges factuels : évaluation des croyances et identification des expressions produites lors d’un mensonge à forte charge cognitive In Urdapilleta, I. Tijus, C., & Demarchi S. (eds.). Paris 8.
Delmas, H. (2020). Vous avez menti : c’est l’IA qui l’a dit ! Cerveau Psycho, 126(10), 66–71.
Delmas, H., Ciocan, C., Novopashyna, M., & Paeye, C. (2023). Resistance of a short-term memory concealed information test with famous faces to countermeasures. Memory & Cognition. https://doi.org/10.3758/s13421-023-01489-1
Denault, V., Plusquellec, P., Jupe, L. M., St-Yves, M., Dunbar, N. E., Hartwig, M., Sporer, S. L., Rioux-Turcotte, J., Jarry, J., Walsh, D., Otgaar, H., Viziteu, A., Talwar, V., Keatley, D. A., Blandón-Gitlin, I., Townson, C., Deslauriers-Varin, N., Lilienfeld, S. O., Patterson, M. L., & van Koppen, P. J. (2019). The analysis of nonverbal communication: The dangers of pseudoscience in security and justice contexts. Clinical and Health, 30(1), 1–12. https://doi.org/10.5093/apj2019a9
Denault, V., & Zloteanu, M. (2022). Darwin’s illegitimate children: How body language experts undermine Darwin’s legacy. Evolutionary Human Sciences, 4, e53. https://doi.org/10.1017/ehs.2022.50
DePaulo, B. M., & Kirkendol, S. E. (1989). The motivational impairment effect in the communication of deception. In J. C. Yuille (Ed.), Credibility Assessment (pp. 51–70). Netherlands: Springer.
DePaulo, B. M., Lindsay, J. J., Malone, B. E., Muhlenbruck, L., Charlton, K., & Cooper, H. (2003). Cues to deception. Psychological Bulletin, 129(1), 74–118. https://doi.org/10.1037/0033-2909.129.1.74
Dervan, L. E., & Edkins, V. A. (2013). The innocent defendant’s dilemma: An innovative empirical study of plea bargaining’s innocence problem. Journal of Criminal Law and Criminology, 103, 1.
Dhanush, T., Sharmila, T. S., & Jennifer, J. S. (2018). Determining response credibility by Blink Count. International Conference on Recent Trends in Advance Computing (ICRTAC), 2018, 143–148. https://doi.org/10.1109/ICRTAC.2018.8679253
Ding, M., Zhao, A., Lu, Z., Xiang, T., & Wen, J.-R. (2019, June). Face-focused cross-stream network for deception detection in videos. In 2019 IEEE/CVF conference on computer vision and pattern recognition (CVPR), Long Beach, CA, USA. https://doi.org/10.1109/cvpr.2019.00799
Ding, X. P., Du, X., Lei, D., Hu, C. S., Fu, G., & Chen, G. (2012). The neural correlates of identity faking and concealment: An FMRI study. PLoS ONE, 7(11), e48639. https://doi.org/10.1371/journal.pone.0048639
Dodia, S., Edla, D. R., Bablani, A., & Cheruku, R. (2019). Lie detection using extreme learning machine: A concealed information test based on short-time Fourier transform and binary bat optimization using a novel fitness function. Computational Intelligence. An International Journal, 50, 247. https://doi.org/10.1111/coin.12256
Duran, N. D., Dale, R., Kello, C. T., Street, C. N. H., & Richardson, D. C. (2013). Exploring the movement dynamics of deception. Frontiers in Psychology, 4, 140. https://doi.org/10.3389/fpsyg.2013.00140
Ekman, P., Friesen, W., & Hager, J. (1978/2002). Facial action coding system. Face and Emotion.
Ekman, P., & Friesen, W. V. (1969). Nonverbal leakage and clues to deception. Psychiatry, 32(1), 88–106.
Ekman, P., & Friesen, W. V. (1976). Measuring facial movement. Journal of Nonverbal Behavior, 1(1), 56–75. https://doi.org/10.1007/BF01115465
Ekman, P., Friesen, W. V., & O’Sullivan, M. (1988). Smiles when lying. Journal of Personality and Social Psychology, 54(3), 414–420.
Ekman, P., Friesen, W. V., & Simons, R. C. (1985). Is the startle reaction an emotion? Journal of Personality and Social Psychology, 49(5), 1416–1426.
Ekman, P., & O’Sullivan, M. (2006). From flawed self-assessment to blatant whoppers: The utility of voluntary and involuntary behavior in detecting deception. Behavioral Sciences & the Law, 24(5), 673–686.
Elkins, A. C., Dunbar, N. E., Adame, B., & Nunamaker, J. F. (2013). Are users threatened by credibility assessment systems? Journal of Management Information Systems, 29(4), 249–262. https://doi.org/10.2753/MIS0742-1222290409
Enos, F., Shriberg, E., Graciarena, M., Hirschberg, J. B., & Stolcke, A. (2007). Detecting deception using critical segments. Columbia University. https://doi.org/10.7916/D8ZG71K7
Fawcett, T. (2006). An introduction to ROC analysis. Pattern Recognition Letters, 27(8), 861–874. https://doi.org/10.1016/j.patrec.2005.10.010
Frank, M. G., & Svetieva, E. (2015). Microexpressions and deception. In M. K. Mandal & A. Awasthi (Eds.), Understanding facial expressions in communication: cross-cultural and multidisciplinary perspectives (pp. 227–242). New Delhi: Springer India.
Friedman, N., Geiger, D., & Goldszmidt, M. (1997). Bayesian network classifiers. Machine Learning, 29(2), 131–163. https://doi.org/10.1023/A:1007465528199
Galin, K. E., & Thorn, B. E. (1993). Unmasking pain: Detection of deception in facial expressions. Journal of Social and Clinical Psychology, 12(2), 182–197. https://doi.org/10.1521/jscp.1993.12.2.182
Girgis, S., Amer, E., & Gadallah, M. (2018). Deep learning algorithms for detecting fake news in online text. In: 2018 13th international conference on computer engineering and systems (ICCES), (pp. 93–97). https://doi.org/10.1109/ICCES.2018.8639198
Gogate, M., Adeel, A., & Hussain, A. (2017). Deep learning driven multimodal fusion for automated deception detection. IEEE Symposium Series on Computational Intelligence (SSCI), 2017, 1–6. https://doi.org/10.1109/SSCI.2017.8285382
Gravanis, G., Vakali, A., Diamantaras, K., & Karadais, P. (2019). Behind the cues: A benchmarking study for fake news detection. Expert Systems with Applications, 128, 201–213. https://doi.org/10.1016/j.eswa.2019.03.036
Gross, S. R. (2008). Convicting the innocent. Annual Review of Law and Social Science, 4(1), 173–192. https://doi.org/10.1146/annurev.lawsocsci.4.110707.172300
Gupta, V., Agarwal, M., Arora, M., Chakraborty, T., Singh, R., & Vatsa, M. (2019). Bag-of-lies: A multimodal dataset for deception detection. IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2019, 83–90. https://doi.org/10.1109/CVPRW.2019.00016
Hadjistavropoulos, H. D., & Craig, K. D. (1994). Acute and chronic low back pain: Cognitive, affective, and behavioral dimensions. Journal of Consulting and Clinical Psychology, 62(2), 341–349.
Hall, L. (2021). Programming the machine: gender, race, sexuality, AI, and the construction of credibility and deceit at the border. Internet Policy Review. https://doi.org/10.14763/2021.4.1601
Hartwig, M., & Bond, C. F., Jr. (2014). Lie detection from multiple cues: A meta‐analysis. Applied Cognitive Psychology.
Harvey, A., Vrij, A., Nahari, G., & Ludwig, K. (2017). Applying the verifiability approach to insurance claims settings: Exploring the effect of the information protocol. Legal and Criminological Psychology, 22(1), 47–59. https://doi.org/10.1111/lcrp.12092
Hasan, K., Rahman, W., Gerstner, L., Sen, T., Lee, S., Haut, K. G., & Hoque, M. (2019). Facial expression based imagination index and a transfer learning approach to detect deception. In 2019 8th international conference on affective computing and intelligent interaction (ACII), (pp. 634–640). https://doi.org/10.1109/ACII.2019.8925473
Hatem, G., Zeidan, J., Goossens, M., & Moreira, C. (2022). Normality testing methods and the importance of skewness and kurtosis in statistical analysis. BAU Journal - Science and Technology, 3(2), 7. https://doi.org/10.54729/KTPE9512
Hauch, V., Blandón-Gitlin, I., Masip, J., & Sporer, S. L. (2015). Are computers effective lie detectors a meta-analysis of linguistic cues to deception. Personality and Social Psychology Review: An Official Journal of the Society for Personality and Social Psychology, Inc, 19(4), 307–342. https://doi.org/10.1177/1088868314556539
Hill, M. L., & Craig, K. D. (2002). Detecting deception in pain expressions: The structure of genuine and deceptive facial displays. Pain, 98(1–2), 135–144.
Hossin, M., & Sulaiman, M. N. (2015). A review on evaluation metrics for data classification evaluations. International Journal of Data Mining & Knowledge Management Process, 5(2), 1.
Howell, D. C., Rogier, M., & Yzerbyt, V. (1998). Méthodes statistiques en sciences humaines. Belgium: De Boeck.
Hu, G., Liu, L., Yuan, Y., Yu, Z., Hua, Y., Zhang, Z., Shen, F., Shao, L., Hospedales, T., Robertson, N., & Yang, Y. (2018). Deep multi-task learning to recognise subtle facial expressions of mental states. Computer Vision – ECCV 2018, 106–123. https://doi.org/10.1007/978-3-030-01258-8_7
Hu, G., Xiao, Y., Cao, Z., Meng, L., Fang, Z., Zhou, J. T., & Yuan, J. (2020). Towards real-time eyeblink detection in the wild: Dataset, theory and practices. IEEE Transactions on Information Forensics and Security, 15, 2194–2208. https://doi.org/10.1109/TIFS.2019.2959978
IBorderCtrl? No! (n.d.). Retrieved May 7, 2023, from https://iborderctrl.no/
Jain, U., Tan, B., & Li, Q. (2012). Concealed knowledge identification using facial thermal imaging. In 2012 IEEE international conference on acoustics, speech and signal processing (ICASSP), (pp. 1677–1680). https://doi.org/10.1109/ICASSP.2012.6288219
Jaiswal, M., Tabibu, S., & Bajpai, R. (2016). The truth and nothing but the truth: Multimodal analysis for deception detection. In 16th international conference on data mining workshops (ICDMW). https://doi.org/10.1109/ICDMW.2016.0137
Jupe, L. M., & Keatley, D. A. (2019). Airport artificial intelligence can detect deception: Or am I lying? Security Journal. https://doi.org/10.1057/s41284-019-00204-7
Kaur, B., Moses, S., Luthra, M., & Ikonomidou, V. N. (2015). Remote stress detection using a visible spectrum camera. Independent Component Analyses Compressive Sampling Large Data Analyses (LDA), Neural Networks. Biosystems, and Nanoengineering, 9496, 949602. https://doi.org/10.1117/12.2177159
Kawulok, M., Nalepa, J., Nurzynska, K., & Smolka, B. (2016). In search of truth: Analysis of smile intensity dynamics to detect deception. Advances in Artificial Intelligence - IBERAMIA, 2016, 325–337. https://doi.org/10.1007/978-3-319-47955-2_27
Khan, W., Crockett, K., O’Shea, J., Hussain, A., & Khan, B. M. (2021). Deception in the eyes of deceiver: A computer vision and machine learning based automated deception detection. Expert Systems with Applications, 169, 114341. https://doi.org/10.1016/j.eswa.2020.114341
Khan, W., Hussain, A., Kuru, K., & Al-Askar, H. (2020). Pupil localisation and eye centre estimation using machine learning and computer vision. Sensors. https://doi.org/10.3390/s20133785
Kleinberg, B., Arntz, A., & Verschuere, B. (2019). Detecting deceptive intentions: Possibilities for large-scale applications. In T. Docan-Morgan (Ed.), The Palgrave Handbook of Deceptive Communication (pp. 403–427). Cham: Springer International Publishing.
Koo, T. K., & Li, M. Y. (2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15(2), 155–163. https://doi.org/10.1016/j.jcm.2016.02.012
Kotsiantis, S. B. (2007). Supervised machine learning: A review of classification techniques. In Proceedings of the 2007 conference on emerging artificial intelligence applications in computer engineering: Real word AI systems with applications in eHealth, HCI, information retrieval and pervasive technologies, (pp. 3–24).
Kotsiantis, S. B., Zaharakis, I. D., & Pintelas, P. E. (2006). Machine learning: A review of classification and combining techniques. Artificial Intelligence Review, 26(3), 159–190. https://doi.org/10.1007/s10462-007-9052-3
Kramer, O. (2016). Scikit-Learn. In O. Kramer (Ed.), Machine Learning for Evolution Strategies (pp. 45–53). Cham: Springer International Publishing.
Krishnamurthy, G., Majumder, N., Poria, S., & Cambria, E. (2018). A deep learning approach for multimodal deception detection. In arXiv [cs.CL]. arXiv. http://arxiv.org/abs/1803.00344
Kwon, O., & Sim, J. M. (2013). Effects of data set features on the performances of classification algorithms. Expert Systems with Applications, 40(5), 1847–1857. https://doi.org/10.1016/j.eswa.2012.09.017
Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159–174. https://doi.org/10.2307/2529310
Lawson, G., Stedmon, A., Zhang, C., Eubanks, D., & Frumkin, L. (n.d.). Deception and Self-awareness. In 9th international conference on engineering psychology and cognitive ergonomics.
Lee, Z.-C., Phan, R.C.-W., Tan, S.-W., & Lee, K.-H. (2017). Multimodal decomposition with magnification on micro-expressions and its impact on facial biometric recognition. IEEE International Symposium on Consumer Electronics (ISCE), 2017, 45–46. https://doi.org/10.1109/ISCE.2017.8355543
Li, Q., Zhan, S., Xu, L., & Wu, C. (2019). Facial micro-expression recognition based on the fusion of deep learning and enhanced optical flow. Multimedia Tools and Applications, 78(20), 29307–29322. https://doi.org/10.1007/s11042-018-6857-9
Liliana, D. Y., & Basaruddin, T. (2019). The fuzzy emotion recognition framework using semantic-linguistic facial features. In 2019 IEEE R10 humanitarian technology conference (R10-HTC)(47129), (pp. 263–268). https://doi.org/10.1109/R10-HTC47129.2019.9042442
Littlewort, G., Whitehill, J., Wu, T., Fasel, I., Frank, M., Movellan, J., & Bartlett, M. (2011). The computer expression recognition toolbox (CERT). In Automatic face gesture recognition and workshops (FG 2011), 2011 IEEE international conference on, (pp. 298–305). https://doi.org/10.1109/FG.2011.5771414
Littlewort, G. C., Bartlett, M. S., & Lee, K. (2009). Automatic coding of facial expressions displayed during posed and genuine pain. Image and Vision Computing, 27(12), 1797–1803. https://doi.org/10.1016/j.imavis.2008.12.010
Luke, T. J. (2019). Lessons from pinocchio: Cues to deception may be highly exaggerated. Perspectives on Psychological Science: A Journal of the Association for Psychological Science, 14(4), 646–671. https://doi.org/10.1177/1745691619838258
Maalouf, M. (2011). Logistic regression in data analysis: An overview. International Journal of Data Analysis Techniques and Strategies, 3(3), 281–299. https://doi.org/10.1504/IJDATS.2011.041335
Marchak, F. M. (2013). Detecting false intent using eye blink measures. Frontiers in Psychology, 4, 736. https://doi.org/10.3389/fpsyg.2013.00736
Margolin, R., Zelnik-Manor, L., & Tal, A. (2014). OTC: A novel local descriptor for scene classification. In Computer vision – ECCV 2014, (pp. 377–391). https://doi.org/10.1007/978-3-319-10584-0_25
Mathur, L., & Matarić, M. J. (2020). Introducing representations of facial affect in automated multimodal deception detection. In Proceedings of the 2020 international conference on multimodal interaction (pp. 305–314). Association for Computing Machinery. https://doi.org/10.1145/3382507.3418864
Mathur, L., & Matarić, M. J. (2021). Unsupervised audio-visual subspace alignment for high-stakes deception detection. In arXiv [cs.CV]. arXiv. http://arxiv.org/abs/2102.03673
Matsumoto, D., & Hwang, H. C. (2017). Clusters of nonverbal behaviors differ according to type of question and veracity in investigative interviews in a mock crime context. Journal of Police and Criminal Psychology, 32(71), 1–14. https://doi.org/10.1007/s11896-017-9250-0
Matsumoto, D., & Hwang, H. (2018). Microexpressions differentiate truths from lies about future malicious intent. Frontiers in Psychology, 9, 2545. https://doi.org/10.3389/fpsyg.2018.02545
Mayya, V., Pai, R. M., & Manohara Pai, M. M. (2016). Combining temporal interpolation and DCNN for faster recognition of micro-expressions in video sequences. In 2016 international conference on advances in computing, communications and informatics (ICACCI), (pp. 699–703). https://doi.org/10.1109/ICACCI.2016.7732128
McClintock, C. C., & Hunt, R. G. (1975). Nonverbal indicators of affect and deception in an interview setting. Journal of Applied Social Psychology, 5(1), 54–67. https://doi.org/10.1111/j.1559-1816.1975.tb00671.x
Mclean, D., Bandar, Z., O’Shea, J., & Crockett, K. A. (2010). Commercialisation of an artificially intelligent deception detection system in the current security climate. In International conference on fuzzy systems, (pp. 1–6). https://doi.org/10.1109/FUZZY.2010.5584016
Mehrnam, A. H., Nasrabadi, A. M., Ghodousi, M., Mohammadian, A., & Torabi, S. (2017). Reprint of “A new approach to analyze data from EEG-based concealed face recognition system.” International Journal of Psychophysiology: Official Journal of the International Organization of Psychophysiology, 122, 17–23. https://doi.org/10.1016/j.ijpsycho.2017.05.006
Meibodi, N., & Bornak, B. (2011). Mouth’s action units recognition base on non-frontal view 3D images. https://doi.org/10.1115/1.859735.paper21
Meng, D., Cao, G., He, Z., & Cao, W. (2016). Facial expression recognition based on LLENet. IEEE International Conference on Bioinformatics and Biomedicine (BIBM), 2016, 1915–1917. https://doi.org/10.1109/BIBM.2016.7822814
Miao, S., Xu, H., Han, Z., & Zhu, Y. (2019). Recognizing facial expressions using a shallow convolutional neural network. IEEE Access, 7, 78000–78011. https://doi.org/10.1109/ACCESS.2019.2921220
Michael, N., Dilsizian, M., Metaxas, D., & Burgoon, J. K. (2010). Motion profiles for deception detection using visual cues. In Computer vision – ECCV 2010, (pp. 462–475). https://doi.org/10.1007/978-3-642-15567-3_34
Monaro, M., Capuozzo, P., Ragucci, F., Maffei, A., Curci, A., Scarpazza, C., Angrilli, A., & Sartori, G. (2020). Using blink rate to detect deception: A study to validate an automatic blink detector and a new dataset of videos from liars and truth-tellers. In Human-computer interaction. Human values and quality of life, (pp. 494–509). https://doi.org/10.1007/978-3-030-49065-2_35
Monaro, M., Maldera, S., Scarpazza, C., Sartori, G., & Navarin, N. (2022). Detecting deception through facial expressions in a dataset of videotaped interviews: A comparison between human judges and machine learning models. Computers in Human Behavior, 127, 107063. https://doi.org/10.1016/j.chb.2021.107063
Müller, A., & Guido, S. (2016). Introduction to Machine Learning with Python: A Guide for Data Scientists (1st ed.). USA: O’Reilly Media.
Nahari, G., Vrij, A., & Fisher, R. P. (2014a). Exploiting liars’ verbal strategies by examining the verifiability of details. Legal and Criminological Psychology, 19(2), 227–239. https://doi.org/10.1111/j.2044-8333.2012.02069.x
Nahari, G., Vrij, A., & Fisher, R. P. (2014b). The verifiability approach: Countermeasures facilitate its ability to discriminate between truths and lies. Applied Cognitive Psychology, 28(1), 122–128.
Nair, A. V., Kumar, K. M., & Mathew, J. (2018). An improved approach for EEG signal classification using autoencoder. In 2018 8th international symposium on embedded computing and system design (ISED), (pp. 6–10). https://doi.org/10.1109/ISED.2018.8704011
Nashaat, M., Ghosh, A., Miller, J., & Quader, S. (2020). Asterisk: Generating large training datasets with automatic active supervision. ACM/IMS Trans. Data Sci., 1(2), 1–25. https://doi.org/10.1145/3385188
National Research Council, Division of Behavioral and Social Sciences and Education, Committee on National Statistics, Board on Behavioral, Cognitive, and Sensory Sciences, & Committee to Review the Scientific Evidence on the Polygraph. (2003). The polygraph and lie detection. National Academies Press. https://doi.org/10.17226/10420
Naven, G., Sen, T., Gerstner, L., Haut, K., Wen, M., & Hoque, E. (2020). Leveraging shared and divergent facial expression behavior between genders in deception detection. In 2020 15th IEEE international conference on automatic face and gesture recognition (FG 2020), (pp. 428–435). https://doi.org/10.1109/FG47880.2020.00124
Nechyba, M. C., Brandy, L., & Schneiderman, H. (2008). PittPatt face detection and tracking for the CLEAR 2007 evaluation. Multimodal Technologies for Perception of Humans. https://doi.org/10.1007/978-3-540-68585-2_10
Ngo, L. M., Wang, W., Mandira, B., Karaoglu, S., Bouma, H., Dibeklioglu, H., & Gevers, T. (2021). Identity unbiased deception detection by 2D-to-3D face reconstruction. IEEE Winter Conference on Applications of Computer Vision (WACV), 2021, 145–154. https://doi.org/10.1109/wacv48630.2021.00019
Ojala, T., Pietikäinen, M., & Harwood, D. (1996). A comparative study of texture measures with classification based on featured distributions. Pattern Recognition, 29(1), 51–59. https://doi.org/10.1016/0031-3203(95)00067-4
Oravec, J. A. (2022). The emergence of “truth machines”?: Artificial intelligence approaches to lie detection. Ethics and Information Technology, 24(1), 6. https://doi.org/10.1007/s10676-022-09621-6
Pérez-Rosas, V., Abouelenien, M., Mihalcea, R., & Burzo, M. (2015). Deception detection using real-life trial data. In Proceedings of the 2015 ACM on International Conference on Multimodal Interaction, (pp. 59–66). https://doi.org/10.1145/2818346.2820758
Petitjean, S. (2019, August 31). Des détecteurs de mensonge expérimentés aux frontières extérieures de l’Europe. Le Monde.
Porter, S., & Brinke, L. (2010). The truth about lies: What works in detecting high-stakes deception? Legal and Criminological Psychology, 15(1), 57–75. https://doi.org/10.1348/135532509x433151
Qu, F., Wang, S. J., Yan, W. J., Li, H., Wu, S., & Fu, X. (2017). CAS(ME)^2: A database for spontaneous macro-expression and micro-expression spotting and recognition. IEEE Transactions on Affective Computing. https://doi.org/10.1109/TAFFC.2017.2654440
Radlak, K., & Smolka, B. (2016). Automated recognition of facial expressions authenticity. In Proceedings of the 18th ACM international conference on multimodal interaction, (pp. 577–581). https://doi.org/10.1145/2993148.2997624
Rakesh Kumar, A. J., Bhanu, B., Casey, C., Cheung, S. G., & Seitz, A. (2021). Depth videos for the classification of micro-expressions. In 2020 25th international conference on pattern recognition (ICPR), (pp. 5278–5285). https://doi.org/10.1109/ICPR48806.2021.9412976
Riggio, R. E., & Friedman, H. S. (1983). Individual differences and cues to deception. Journal of Personality and Social Psychology, 45(4), 899–915. https://doi.org/10.1037/0022-3514.45.4.899
Rill-Garcia, R., Escalante, H. J., Villasenor-Pineda, L., & Reyes-Meza, V. (2019). High-Level features for multimodal deception detection in videos. In 2019 IEEE/CVF conference on computer vision and pattern recognition workshops (CVPRW), Long Beach, CA, USA. https://doi.org/10.1109/cvprw.2019.00198
Rodriguez-Diaz, N., Aspandi, D., Sukno, F. M., & Binefa, X. (2021). Machine learning-based lie detector applied to a novel annotated game dataset. Future Internet, 14(1), 2. https://doi.org/10.3390/fi14010002
Rodriguez-Lozano, F. J., León-García, F., Ruiz de Adana, M., Palomares, J. M., & Olivares, J. (2019). Non-invasive forehead segmentation in thermographic imaging. Sensors. https://doi.org/10.3390/s19194096
Rothwell, J., Bandar, Z., O’Shea, J., & McLean, D. (2006). Silent talker: A new computer-based system for the analysis of facial cues to deception. Applied Cognitive Psychology, 20(6), 757–777. https://doi.org/10.1002/acp.1204
Rublee, E., Rabaud, V., Konolige, K., & Bradski, G. (2011). ORB: An efficient alternative to SIFT or SURF. International Conference on Computer Vision, 2011, 2564–2571. https://doi.org/10.1109/ICCV.2011.6126544
Russell, J. A. (1980). A circumplex model of affect. Journal of Personality and Social Psychology, 39(6), 1161.
Sagi, O., & Rokach, L. (2018). Ensemble learning: A survey. Wiley Interdisciplinary Reviews. Data Mining and Knowledge Discovery, 8(4), e1249. https://doi.org/10.1002/widm.1249
Saha, P., Bhowmik, M. K., Bhattacharjee, D., De, B. K., & Nasipuri, M. (2016). Expressions recognition of North-East Indian (NEI) faces. Multimedia Tools and Applications, 75(24), 16781–16807. https://doi.org/10.1007/s11042-015-2945-2
Saleem, S., Aslam, M., & Shaukat, M. R. (2021). A review and empirical comparison of univariate outlier detection methods. Pakistan Journal of Statistics, 37(4), 447–462.
Sánchez-Monedero, J., & Dencik, L. (2022). The politics of deceptive borders: “biomarkers of deceit” and the case of iBorderCtrl. Information, Communication and Society, 25(3), 413–430. https://doi.org/10.1080/1369118X.2020.1792530
Sen, T., Hasan, M. K., Teicher, Z., & Hoque, M. E. (2018). Automated dyadic data recorder (ADDR) framework and analysis of facial cues in deceptive communication. Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies, 1(4), 1–22. https://doi.org/10.1145/3161178
Sharma, A., & Paliwal, K. K. (2015). Linear discriminant analysis for the small sample size problem: An overview. International Journal of Machine Learning and Cybernetics, 6(3), 443–454. https://doi.org/10.1007/s13042-013-0226-9
Shen, X., Fan, G., Niu, C., & Chen, Z. (2021). Catching a liar through facial expression of fear. Frontiers in Psychology, 12, 675097. https://doi.org/10.3389/fpsyg.2021.675097
Singh, R. D., Mittal, A., & Bhatia, R. K. (2019). 3D convolutional neural network for object recognition: A review. Multimedia Tools and Applications, 78(12), 15951–15995. https://doi.org/10.1007/s11042-018-6912-6
Slowe, T. E., & Govindaraju, V. (2007). Automatic deceit indication through reliable facial expressions. IEEE Workshop on Automatic Identification Advanced Technologies, 2007, 87–92. https://doi.org/10.1109/AUTOID.2007.380598
Sporer, S. L. (2016). Deception and cognitive load: expanding our horizon with a working memory model. Frontiers in Psychology, 7, 420. https://doi.org/10.3389/fpsyg.2016.00420
Sporer, S. L., & Schwandt, B. (2007). Moderators of nonverbal indicators of deception: A meta-analytic synthesis. Psychology, Public Policy, and Law, 13(1), 1–34. https://doi.org/10.1037/1076-8971.13.1.1
Su, L., & Levine, M. (2014). High-Stakes Deception Detection Based on Facial Expressions 22nd International Conference on Pattern Recognition Stockholm, (pp. 2519–2524). Sweden. https://doi.org/10.1109/ICPR.2014.435
Su, L., & Levine, M. (2016). Does “lie to me” lie to you? An evaluation of facial clues to high-stakes deception. Computer Vision and Image Understanding: CVIU, 147, 52–68. https://doi.org/10.1016/j.cviu.2016.01.009
Taylor, R., & Hick, R. F. (2007). Believed cues to deception: Judgments in self-generated trivial and serious situations. Legal and Criminological Psychology, 12(2), 321–331. https://doi.org/10.1348/135532506X116101
Ten Brinke, L., & Porter, S. (2012). Cry me a river: Identifying the behavioral consequences of extremely high-stakes interpersonal deception. Law and Human Behavior, 36(6), 469–477. https://doi.org/10.1037/h0093929
Thannoon, H. H., Ali, W. H., & Hashim, I. A. (2018). Detection of deception using facial expressions based on different classification algorithms. Third Scientific Conference of Electrical Engineering (SCEE), 2018, 51–56. https://doi.org/10.1109/SCEE.2018.8684170
Tsiamyrtzis, P., Dowdall, J., Shastri, D., Pavlidis, I. T., Frank, M. G., & Ekman, P. (2007). Imaging facial physiology for the detection of deceit. International Journal of Computer Vision, 71(2), 197–214. https://doi.org/10.1007/s11263-006-6106-y
Twitchell, D. P., & Fuller, C. M. (2019). Advancing the assessment of automated deception detection systems: Incorporating base rate and cost into system evaluation. Information Systems Journal, 29(3), 738–761. https://doi.org/10.1111/isj.12231
Venkatesh, S., Ramachandra, R., & Bours, P. (2019). Robust algorithm for multimodal deception detection. IEEE Conference on Multimedia Information Processing and Retrieval (MIPR), 2019, 534–537. https://doi.org/10.1109/MIPR.2019.00108
Verbur, M., & Menkovski, V. (2019). Micro-expression detection in long videos using optical flow and recurrent neural networks. In 14th IEEE international conference on automatic face & gesture recognition (FG 2019).
Verma, G. K. (2017). Facial micro-expression recognition using discrete curvelet transform. Conference on Information and Communication Technology (CICT), 2017, 1–6. https://doi.org/10.1109/INFOCOMTECH.2017.8340637
Vrij, A. (2008). Detecting lies and deceit pitfalls and opportunities. New Jersey: John Wiley & Sons.
Vrij, A. (2010). Behavioral correlates of deception in a simulated police interview. The Journal of Psychology. https://doi.org/10.1080/00223980.1995.9914944
Vrij, A., & Fisher, R. P. (2020). Lying and nervous behaviours unravelling the misconception about deception and nervous behaviour. Frontiers in Psychology, 11, 1377.
Vrij, A., Fisher, R. P., & Blank, H. (2017). A cognitive approach to lie detection: A meta-analysis. Legal and Criminological Psychology, 22(1), 1–21. https://doi.org/10.1111/lcrp.12088
Vrij, A., Hartwig, M., & Granhag, P. A. (2019). Reading lies: Nonverbal communication and deception. Annual Review of Psychology, 70, 295–317. https://doi.org/10.1146/annurev-psych-010418-103135
Wang, S.-J., Yan, W.-J., Li, X., Zhao, G., & Fu, X. (2014). Micro-expression recognition using dynamic textures on tensor independent color space. In: 2014 22nd international conference on pattern recognition, (pp. 4678–4683). https://doi.org/10.1109/ICPR.2014.800
Wang, S.-J., Yan, W.-J., Zhao, G., Fu, X., & Zhou, C.-G. (2015). Micro-expression recognition using robust principal component analysis and local spatiotemporal directional features. In Computer vision - ECCV 2014 workshops, (pp. 325–338). https://doi.org/10.1007/978-3-319-16178-5_23
Wang, L., Hou, J., Guo, X., Ma, Z., Liu, X., & Fang, H. (2020). Micro-expression video clip synthesis method based on spatial-temporal statistical model and motion intensity evaluation function. In 2020 IEEE international conference on systems, man, and cybernetics (SMC), (pp. 211–217). https://doi.org/10.1109/SMC42975.2020.9283113
Wang, H., & Schmid, C. (2013). Action recognition with improved trajectories. IEEE International Conference on Computer Vision, 2013, 3551–3558. https://doi.org/10.1109/ICCV.2013.441
Wang, S.-J., Wu, S., Qian, X., Li, J., & Fu, X. (2017). A main directional maximal difference analysis for spotting facial movements from long-term videos. Neurocomputing, 230, 382–389. https://doi.org/10.1016/j.neucom.2016.12.034
Wang, S.-J., Yan, W.-J., Sun, T., Zhao, G., & Fu, X. (2016). Sparse tensor canonical correlation analysis for micro-expression recognition. Neurocomputing, 214, 218–232. https://doi.org/10.1016/j.neucom.2016.05.083
Wen, G., Chang, T., Li, H., & Jiang, L. (2020). Dynamic objectives learning for facial expression recognition. IEEE Transactions on Multimedia, 22(11), 2914–2925. https://doi.org/10.1109/TMM.2020.2966858
Wong, T.-T. (2015). Performance evaluation of classification algorithms by k-fold and leave-one-out cross validation. Pattern Recognition, 48(9), 2839–2846. https://doi.org/10.1016/j.patcog.2015.03.009
Wu, Y., & Ji, Q. (2019). Facial landmark detection: A literature survey. International Journal of Computer Vision, 127(2), 115–142. https://doi.org/10.1007/s11263-018-1097-z
Wu, Z., Singh, B., Davis, L., & Subrahmanian, V. (2018). Deception detection in videos. Proceedings of the AAAI Conference on Artificial Intelligence. https://doi.org/10.1609/aaai.v32i1.11502
Yan, W.-J., Wu, Q., Liu, Y.-J., Wang, S.-J., & Fu, X. (2013). CASME database: A dataset of spontaneous micro-expressions collected from neutralized faces. In 2013 10th IEEE international conference and workshops on automatic face and gesture recognition (FG), (pp. 1–7). https://doi.org/10.1109/FG.2013.6553799
Zage, D., Glass, K., & Colbaugh, R. (2013). Improving supply chain security using big data. In Conference: Intelligence and security informatics (ISI), (pp. 1–6).
Zhang, S., Zhang, C., & Yang, Q. (2003). Data preparation for data mining. Applied Artificial Intelligence: AAI, 17(5–6), 375–381. https://doi.org/10.1080/713827180
Zhang, Z., Singh, V., Slowe, T. E., Tulyakov, S., & Govindaraju, V. (2007). Real-time Automatic Deceit Detection from Involuntary Facial Expressions. https://doi.org/10.1109/CVPR.2007.383383
Zuckerman, M., DePaulo, B. M., & Rosenthal, R. (1981). Verbal and nonverbal communication of deception. In L. Berkowitz (Ed.), Advances in Experimental Social Psychology (Vol. 14, pp. 1–59). Amsterdam: Elsevier.
Zurloni, V., Diana, B., Cavalera, C., Argenton, L., Elia, M., & Mantovani, F. (2015). Deceptive behavior in doping related interviews: The case of Lance Armstrong. Psychology of Sport and Exercise, 16, 191–200. https://doi.org/10.1016/j.psychsport.2014.02.008
Acknowledgements
We would like to thank the anonymous reviewers for their constructive comments.
Author information
Authors and Affiliations
Contributions
Conceptualization: H.D., N.D., V.D. Literature search: V.D., H.D, N.D. Data coding: H.D., N.D. Statistics and R programming: H.D. Writing and editing: H.D, N.D., V.D, J.B.
Corresponding author
Ethics declarations
Conflict of interests
J.B is a founder of Discern Science International, a for-profit company that conducts credibility research and is the developer of the AVATAR. N.D. is a consultant to Discern Science International. The other authors report no conflict of interest.
Large Language Models
Large Language Models have been used by H.D. whose first language is French, to help translation from French to English, and to correct English.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Delmas, H., Denault, V., Burgoon, J.K. et al. A Review of Automatic Lie Detection from Facial Features. J Nonverbal Behav 48, 93–136 (2024). https://doi.org/10.1007/s10919-024-00451-2
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10919-024-00451-2