Skip to main content
SpringerLink
Log in

Visual attention-based deepfake video forgery detection

  • Short Paper
  • Published:
Pattern Analysis and Applications Aims and scope Submit manuscript

Abstract

The prime goal of creating synthetic digital data is to generate something very closer to real ones when the original data are scarce. However, the trustworthiness of such digital content is dipping potentially in society owing to malicious users. Deepfake method that uses computer graphics and computer vision techniques to replace the face of one person with the face of a different person is becoming an area of big concern. Such techniques can easily be used to hide the identity of a person. Therefore, a method is needed to verify the originality of such face images/videos. To this end, we design a deep learning model enhanced with visual attention technique to differentiate manipulated videos/images (generated by deepfake methods) from real ones. At first, we extract the face region from video frames and then pass the same through the pre-trained Xception model to obtain the feature maps. Next, with the help of the visual attention mechanism, we mainly try to focus on the deepfake video manipulation leftover artifacts. We evaluate our model on two publicly available datasets, namely FaceForensics++ and Celeb-DF (V2), and our model outperforms many state-of-the-art methods tested on these two datasets. Source code of the proposed method can be found at: https://github.com/tre3x/Deepfake-Video-Forgery-Detection.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Availability of data and materials

Not Applicable in our case.

Notes

  1. http://kaldir.vc.in.tum.de/faceforensics_benchmark.

References

  1. Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets. Adv Neural Inf Process Syst 27

  2. Li Y, Chang M-C, Lyu S (2018) In ictu oculi: exposing ai created fake videos by detecting eye blinking. In: 2018 IEEE international workshop on information forensics and security (WIFS), pp 1–7. IEEE

  3. Jung T, Kim S, Kim K (2020) Deepvision: deepfakes detection using human eye blinking pattern. IEEE Access 8:83144–83154

    Article  Google Scholar 

  4. Tolosana R, Romero-Tapiador S, Fierrez J, Vera-Rodriguez R (2021) Deepfakes evolution: analysis of facial regions and fake detection performance. In: international conference on pattern recognition, pp 442–456. Springer

  5. Matern F, Riess C, Stamminger M (2019) Exploiting visual artifacts to expose deepfakes and face manipulations. In: 2019 IEEE winter applications of computer vision workshops (WACVW), pp 83–92. IEEE

  6. Walia S, Kumar K, Kumar M, Gao X-Z (2021) Fusion of handcrafted and deep features for forgery detection in digital images. IEEE Access 9:99742–99755

    Article  Google Scholar 

  7. Chollet F (2017) Xception: deep learning with depthwise separable convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1251–1258

  8. Nguyen HH, Yamagishi J, Echizen I (2019) Use of a capsule network to detect fake images and videos. arXiv preprint arXiv:1910.12467

  9. Li Y, Yang X, Sun P, Qi H, Lyu S (2020) Celeb-df: a large-scale challenging dataset for deepfake forensics. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR)

  10. Pan D, Sun L, Wang R, Zhang X, Sinnott RO (2020) Deepfake detection through deep learning. In: 2020 IEEE/ACM international conference on big data computing, applications and technologies (bdccBDCAT), pp 134–143. https://doi.org/10.1109/BDCAT50828.2020.00001

  11. Rossler A, Cozzolino D, Verdoliva L, Riess C, Thies J, Nießner M (2019) FaceForensics++: Learning to detect manipulated facial images. In: International conference on computer vision (ICCV)

  12. Li Y, Yang X, Sun P, Qi H, Lyu S (2020) Celeb-df: A large-scale challenging dataset for deepfake forensics. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 3207–3216

  13. Durall R, Keuper M, Pfreundt F-J, Keuper J (2019) Unmasking deepfakes with simple features. arXiv preprint arXiv:1911.00686

  14. Guarnera L, Giudice O, Battiato S (2020) Deepfake detection by analyzing convolutional traces. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pp 666–667

  15. Moon TK (1996) The expectation-maximization algorithm. IEEE Signal Process Mag 13(6):47–60

    Article  Google Scholar 

  16. Yang X, Li Y, Lyu S (2019) Exposing deep fakes using inconsistent head poses. In: ICASSP 2019-2019 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp 8261–8265. IEEE

  17. Li H, Li B, Tan S, Huang J (2020) Identification of deep network generated images using disparities in color components. Signal Process 174:107616

    Article  Google Scholar 

  18. Koopman M, Rodriguez AM, Geradts Z (2018) Detection of deepfake video manipulation. In: The 20th Irish machine vision and image processing conference (IMVIP), pp 133–136

  19. Welch BL (1947) The generalization of ‘student’s’problem when several different population varlances are involved. Biometrika 34(1–2):28–35

  20. Afchar D, Nozick V, Yamagishi J, Echizen I (2018) Mesonet: a compact facial video forgery detection network. In: 2018 IEEE international workshop on information forensics and security (WIFS), pp 1–7. IEEE

  21. Rossler A, Cozzolino D, Verdoliva L, Riess C, Thies J, Nießner M (2019) Faceforensics++: learning to detect manipulated facial images. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 1–11

  22. Shang Z, Xie H, Zha Z, Yu L, Li Y, Zhang Y (2021) Prrnet: pixel-region relation network for face forgery detection. Pattern Recog 116:107950

    Article  Google Scholar 

  23. Sun K, Zhao Y, Jiang B, Cheng T, Xiao B, Liu D, Mu Y, Wang X, Liu W, Wang J (2019) High-resolution representations for labeling pixels and regions. arXiv preprint arXiv:1904.04514

  24. Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1–9

  25. Bayar B, Stamm MC (2016) A deep learning approach to universal image manipulation detection using a new convolutional layer. In: Proceedings of the 4th ACM workshop on information hiding and multimedia security, pp 5–10

  26. Amerini I, Galteri L, Caldelli R, Del Bimbo A (2019) Deepfake video detection through optical flow based cnn. In: Proceedings of the IEEE/CVF international conference on computer vision workshops, pp 0–0

  27. Sun D, Yang X, Liu M-Y, Kautz J (2018) Pwc-net: Cnns for optical flow using pyramid, warping, and cost volume. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 8934–8943

  28. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556

  29. Li Z, Sun Y, Zhang L, Tang J (2021) Ctnet: context-based tandem network for semantic segmentation. IEEE Trans Pattern Anal Mach Intell

  30. Sun Y, Li Z (2021) Ssa: Semantic structure aware inference for weakly pixel-wise dense predictions without cost. arXiv preprint arXiv:2111.03392

  31. Su Y, Xia H, Liang Q, Nie W (2021) Exposing deepfake videos using attention based convolutional lstm network. Neural Process Lett 53(6):4159–4175

    Article  Google Scholar 

  32. Khormali A, Yuan J-S (2021) Add: attention-based deepfake detection approach. Big Data Cogn Comput 5(4):49

    Article  Google Scholar 

  33. Wang C, Deng W (2021) Representative forgery mining for fake face detection. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 14923–14932

  34. Chen S, Yao T, Chen Y, Ding S, Li J, Ji R (2021) Local relation learning for face forgery detection. Proc AAAI Conf Artif Intell 35:1081–1088

    Google Scholar 

  35. Jiang B, Ren Q, Dai F, Xiong J, Yang J, Gui G (2018) Multi-task cascaded convolutional neural networks for real-time dynamic face recognition method. In: International conference in communications, signal processing, and systems, pp 59–66. Springer

  36. Bahdanau D, Cho K, Bengio Y (2014) Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473

  37. Fridrich J, Kodovsky J (2012) Rich models for steganalysis of digital images. IEEE Trans Inf Forensics Sec 7(3):868–882. https://doi.org/10.1109/TIFS.2012.2190402

    Article  Google Scholar 

  38. Cozzolino D, Poggi G, Verdoliva L (2017) Recasting residual-based local descriptors as convolutional neural networks: an application to image forgery detection. In: Proceedings of the 5th ACM workshop on information hiding and multimedia security, pp 159–164

  39. Rahmouni N, Nozick V, Yamagishi J, Echizen I (2017) Distinguishing computer graphics from natural images using convolution neural networks. In: 2017 IEEE workshop on information forensics and security (WIFS), pp 1–6. https://doi.org/10.1109/WIFS.2017.8267647

  40. Baek J-Y, Yoo Y-S, Bae S-H (2020) Generative adversarial ensemble learning for face forensics. IEEE Access 8:45421–45431. https://doi.org/10.1109/ACCESS.2020.2968612

    Article  Google Scholar 

  41. Guo Z, Yang G, Chen J, Sun X (2021) Fake face detection via adaptive manipulation traces extraction network. Comput Vision Image Underst 204:103170. https://doi.org/10.1016/j.cviu.2021.103170

    Article  Google Scholar 

  42. Ciftci UA, Demir I, Yin L (2020) Fakecatcher: detection of synthetic portrait videos using biological signals. IEEE Trans Pattern Anal Mach Intell. https://doi.org/10.1109/TPAMI.2020.3009287

    Article  Google Scholar 

  43. Khormali A, Yuan J-S (2021) Add: attention-based deepfake detection approach. Big Data Cogn Comput. https://doi.org/10.3390/bdcc5040049

    Article  Google Scholar 

Download references

Acknowledgements

We are thankful to the Center for Microprocessor Applications for Training Education and Research (CMATER) research laboratory of the Computer Science and Engineering Department, Jadavpur University, Kolkata, India for providing infrastructural support.

Funding

Not Applicable in our case.

Author information

Authors and Affiliations

  1. Department of Construction Engineering, Jadavpur University Salt Lake Campus, Kolkata, West Bengal, 700098, India

    Shreyan Ganguly

  2. Department of Computer Science, Asutosh College, Kolkata, West Bengal, 700026, India

    Sk Mohiuddin & Samir Malakar

  3. Departamento de Electrónica, Universidad de Guadalajara, 44430, City, C.P, México

    Erik Cuevas

  4. Department of Computer Science and Engineering, Jadavpur University, Kolkata, West Bengal, 700032, India

    Ram Sarkar

Authors
  1. Shreyan Ganguly
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sk Mohiuddin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Samir Malakar
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Erik Cuevas
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Ram Sarkar
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

SG and SKM contributed to conceptualization, methodology, formal analysis and investigation, writing - original draft preparation, writing - review and editing, and resources; SM and RS were involved in conceptualization, methodology, formal analysis and investigation, writing - original draft preparation, writing - review and editing, resources, and supervision; EC contributed to conceptualization, writing - original draft preparation, writing - review and editing, and resources.

Corresponding author

Correspondence to Erik Cuevas.

Ethics declarations

Conflict of interest/Competing interests

The authors declare that they have no conflict of interest.

Ethics approval

Not Applicable in our case.

Consent to participate

Not Applicable in our case.

Consent for publication

Not Applicable in our case.

Code availability

Not Applicable in our case.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ganguly, S., Mohiuddin, S., Malakar, S. et al. Visual attention-based deepfake video forgery detection. Pattern Anal Applic 25, 981–992 (2022). https://doi.org/10.1007/s10044-022-01083-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10044-022-01083-2

Keywords

Search

Navigation