Skip to main content
SpringerLink
Log in

Uncovering visual attention-based multi-level tampering traces for face forgery detection

  • Original Paper
  • Published:
Signal, Image and Video Processing Aims and scope Submit manuscript

Abstract

With the rise of realistic face forgery techniques, the threat of identity fraud is more significant than ever. Several research works have focused on detecting such forgeries, but they extract forgery clues as a preprocessing step to the feature extraction phase of deep neural networks. A novel DenseTrace-Net architecture is designed in this manuscript to extract more comprehensive and refined face tampering traces locally and globally. Specifically, DenseTrace-Net extracts attentional multi-level tampering traces from facial images. A novel ‘Local Attentional Tamper Trace Extractor’ (LATTE) module extracts face tampering traces locally at the block level. A novel ‘Global Attentional Tamper Trace Extractor’ (GATTE) module aggregates multi-scale tampering traces globally. The LATTE and GATTE modules use visual depth attention to enhance their feature representation capability. Additionally, the proposed DenseTrace-Net is computationally lightweight with just 1.378 million parameters. DenseTrace-Net is evaluated on three benchmark datasets, the FF +  + , CelebDF and DFDC datasets, achieving AUC scores of 0.9784, 0.9843 and 0.9916, respectively. These excellent scores allow the DenseTrace-Net to outperform the existing state-of-the-art face forgery detection methods comfortably.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Data availability

The datasets generated during and/or analyzed during the current study are available through an online web repository via the following web links: CelebDF—https://cse.buffalo.edu/~siweilyu/celeb-deepfakeforensics.html. DFDC—https://ai.facebook.com/datasets/dfdc/. FaceForensics +  +—https://github.com/ondyari/FaceForensics.

References

  1. “DeepFakes,” GitHub, 14 August 2020. [Online]. https://github.com/deepfakes/faceswap. Accessed 08 July 2022

  2. Thies, J., Zollhöfer, M., Stamminger, M., Theobalt, C., Nießner, M.: Face2Face: real-time face capture and reenactment of RGB videos. Commun. ACM 62(1), 96–104 (2019)

    Article  Google Scholar 

  3. “DeepFaceLab,” GitHub, 18 March 2020. [Online]. https://github.com/iperov/DeepFaceLab. Accessed 08 July 2022

  4. Wang, G., Jiang, Q., Jin, X., Li, W., Cui, X.: MC-LCR: Multimodal contrastive classification by locally correlated representations for effective face forgery detection. Knowl. Based Syst. 250, 109114 (2022)

    Article  Google Scholar 

  5. Hu, Z., Duan, Q., Zhang, P., Tao, H.: An attention-erasing stripe pyramid network for face forgery detection. Signal Image Video Process. (2023). https://doi.org/10.1007/s11760-023-02644-6

    Article  Google Scholar 

  6. Masud, U., Sadiq, M., Masood, S., Ahmad, M., El-Latif, A.A.A.: LW-DeepFakeNet: a light-weight time distributed CNN-LSTM network for real-time DeepFake video detection. Signal Image Video Process. (2023). https://doi.org/10.1007/s11760-023-02633-9

    Article  Google Scholar 

  7. Shang, Z., Xie, H., Zha, Z., Yu, L., Li, Y., Zhang, Y.: PRRNet: pixel-region relation network for face forgery detection. Pattern Recognit (2021). https://doi.org/10.1016/j.patcog.2021.107950

    Article  Google Scholar 

  8. Xu, Z., Liu, J., Lu, W., Xu, B., Zhao, X., Li, B., Huang, J.: Detecting facial manipulated videos based on set convolutional neural networks. J. Vis. Commun. Image Represent. (2021). https://doi.org/10.1016/j.jvcir.2021.103119

    Article  Google Scholar 

  9. Chen, Z., Yang, H.: Attentive semantic exploring for manipulated face detection. In: IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Toronto (2021)

  10. Heo, Y.J., Yeo, W.-H., Kim, B.-G.: DeepFake detection algorithm based on improved vision transformer. Appl. Intell. 53, 7512–7527 (2023)

    Article  Google Scholar 

  11. Bonomi, M., Pasquini, C., Boato, G.: Dynamic texture analysis for detecting fake faces in video sequences. J. Vis. Commun. Image Represent. (2021). https://doi.org/10.1016/j.jvcir.2021.103239

    Article  Google Scholar 

  12. Yang, J., Xiao, S., Li, A., Lan, G., Wang, H.: Detecting fake images by identifying potential texture difference. Futur. Gener. Comput. Syst. 125, 127–135 (2021)

    Article  Google Scholar 

  13. Yang, J., Li, A., Xiao, S., Lu, W., Gao, X.: MTD-Net: Learning to Detect Deepfakes Images by Multi-Scale Texture Difference. IEEE Trans. Inf. Forensics Secur. 16, 4234–4245 (2021)

    Article  Google Scholar 

  14. Caldelli, R., Galteri, L., Amerini, I., Bimbo, A.D.: Optical Flow based CNN for detection of unlearnt deepfake. Pattern Recogn. Lett. 146, 31–37 (2021)

    Article  ADS  Google Scholar 

  15. Kohli, A., Gupta, A.: Detecting DeepFake, FaceSwap and Face2Face facial forgeries using frequency CNN. Multimedia Tools and Applications 80, 18461–18478 (2022)

    Article  Google Scholar 

  16. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: An image is worth 16×16 words: transformers for image recognition at scale. In: International Conference on Learning Representations (ICLR), Austria (2021)

  17. Wu, H., Xiao, B., Codella, N., Liu, M., Dai, X., Yuan, L., Zhang, L.: CvT: Introducing convolutions to vision transformers. In: International Conference on Computer Vision (ICCV), Montreal (2021)

  18. Heo, Y.J., Yeo, W.-H., Kim, B.-G.: DeepFake detection algorithm based on improved vision transformer. Appl. Intell. (2022)

  19. Guo, Z., Yang, G., Chen, J., Sun, X.: Fake face detection via adaptive manipulation traces extraction network. Comput. Vis. Image Underst. (2021). https://doi.org/10.1016/j.cviu.2021.103170

    Article  Google Scholar 

  20. Amerini, I., Galteri, L., Caldelli, R., Bimbo, A.D.: Deepfake video detection through optical flow based CNN. In: IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), Seoul (2019)

  21. Caldelli, R., Galteri, L., Amerini, I., Bimbo, A.D.: Optical flow based CNN for detection of unlearnt deepfake manipulations. Pattern Recogn. Lett. 146, 31–37 (2021)

    Article  ADS  Google Scholar 

  22. Hu, J., Liao, X., Wang, W., Qin, Z.: Detecting compressed deepfake videos in social networks using frame-temporality two-stream convolutional network. IEEE Trans. Circuits Syst. Video Technol. (2021). https://doi.org/10.1109/TCSVT.2021.3074259

    Article  Google Scholar 

  23. Dang, H., Liu, F., Stehouwer, J., Liu, X., Jain, A.K.: On the detection of digital face manipulation. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle (2020)

  24. Choi, D.H., Lee, H.J., Lee, S., Kim, J.U., Ro, Y.M.: Fake video detection with certainty-based attention network. In: IEEE International Conference on Image Processing (ICIP), Abu Dhabi (2020)

  25. Lu, C., Liu, B., Zhou, W., Chu, Q., Yu, N.: Deepfake video detection using 3D-attentional inception convolutional neural network. In: IEEE International Conference on Image Processing (ICIP), Anchorage (2021)

  26. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas (2016)

  27. Huang, G., Sun, Y., Liu, Z., Sedra, D., Weinberger, K.Q.: Deep networks with stochastic depth. In: European Conference on Computer Vision, Amsterdam (2016)

  28. Huang, G., Liu, Z., Maaten, L.V.D., Weinberger, K.Q.: Densely connected convolutional networks. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu (2017)

  29. J. Hu, L. Shen and G. Sun, “Squeeze-and-Excitation Networks,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 2018

  30. Park, J., Woo, S., Lee, J.-Y., Kweon, I.S.: BAM: bottleneck attention module (2018). https://arxiv.org/pdf/1807.06514.pdf

  31. Woo, S., Park, J., Lee, J.-Y., Kweon, I.S.: CBAM: convolutional block attention module. In: European Conference on Computer Vision, Munich, Germany (2018)

  32. Zhou, Y., Luo, A., Kang, X., Lyu, S.: Face forgery detection based on segmentation network. In: IEEE International Conference on Image Processing (ICIP), Anchorage, AK, USA (2021)

  33. Zhang, J., Ni, J., Xie, H.: DeepFake videos detection using self-supervised decoupling network. In: IEEE International Conference on Multimedia and Expo (ICME), Shenzhen (2021)

  34. Li, Y., Yang, X., Sun, P., Qi, H., Lyu, S.: Celeb-DF: a large-scale challenging dataset for DeepFake forensics. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA (2020)

  35. Dolhansky, B., Howes, R., Pflaum, B., Baram, N., Ferrer, C.C.: The deepfake detection challenge (DFDC) preview dataset (2019). https://arxiv.org/abs/1910.08854

  36. Rössler, A., Cozzolino, D., Verdoliva, L., Riess, C., Thies, J., Niessner, M.: FaceForensics++: learning to detect manipulated facial images. In: IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Korea (South) (2019)

  37. Yang, G., Wei, A., Fang, X., Zhang, J.: FDS_2D: rethinking magnitude-phase features for DeepFake detection. Multimed. Syst. (2023). https://doi.org/10.1007/s00530-023-01118-6

    Article  Google Scholar 

  38. Guo, Z., Yang, G., Zhang, D., Xia, M.: Rethinking gradient operator for exposing AI-enabled face forgeries. Expert Syst. Appl. (2023). https://doi.org/10.1016/j.eswa.2022.119361

    Article  Google Scholar 

  39. Xu, K., Yang, G., Fang, X., Zhang, J.: Facial depth forgery detection. Multimed. Tools Appl. (2023). https://doi.org/10.1007/s11042-023-14626-4

    Article  PubMed  PubMed Central  Google Scholar 

  40. Lin, H., Huang, W., Luo, W., Lu, W.: DeepFake detection with multi-scale convolution and vision transformer. Digit. Signal Process. (2023). https://doi.org/10.1016/j.dsp.2022.103895

    Article  Google Scholar 

  41. Yang, G., Xu, K., Fang, X., Zhang, J.: Video face forgery detection via facial motion-assisted capturing dense optical flow truncation. Vis. Comput. (2022). https://doi.org/10.1007/s00371-022-02683-z

    Article  PubMed  PubMed Central  Google Scholar 

  42. Nirkin, Y., Wolf, L., Keller, Y., Hassner, T.: DeepFake detection based on discrepancies between faces and their context. IEEE Trans. Pattern Anal. Mach. Intell. 44(10), 6111–6121 (2022)

    Article  PubMed  Google Scholar 

  43. Hou, Q., Zhou, D., Feng, J.: Coordinate attention for efficient mobile network design. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA (2021)

  44. Liu, H., Li, X., Zhou, W., Chen, Y., He, Y., Xue, H., Zhang, W., Yu, N.: Spatial-phase shallow learning: rethinking face forgery detection in frequency domain. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA (2021)

  45. Hu, J., Liao, X., Wang, W., Qin, Z.: Detecting compressed deepfake videos in social networks using frame-temporality two-stream convolutional network. IEEE Trans. Circuits Syst. Video Technol. 32(3), 1089–1102 (2021)

    Article  Google Scholar 

  46. Chen, H.-S., Rouhsedaghat, M., Ghani, H., Hu, S., You, S., Kuo, C.-C.J.: DefakeHop: a light-weight high-performance deepfake detector. In: IEEE International Conference on Multimedia and Expo (ICME), Shenzhen (2021)

  47. Yu, Z., Zhao, C., Wang, Z., Qin, Y., Su, Z., Li, X., Zhou, F., Zhao, G.: Searching central difference convolutional networks for face anti-spoofing. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA (2020)

  48. Baek, J.-Y., Yoo, Y.-S., Bae, S.-H.: Generative adversarial ensemble learning for face forensics. IEEE Access 8, 45421–45431 (2020)

    Article  Google Scholar 

  49. Zi, B., Chang, M., Chen, J., Ma, X., Jiang, Y.-G.: WildDeepfake: a challenging real-world dataset for deepfake detection. In: 28th ACM International Conference on Multimedia (2020)

  50. Afchar, D., Nozick, V., Yamagishi, J., Echizen, I.: MesoNet: a compact facial video forgery detection network. In: IEEE International Workshop on Information Forensics and Security (WIFS), Hong Kong, China (2018)

  51. Mohiuddin, S., Sheikh, K.H., Malakar, S., Velásquez, J.D., Sarkar, R.: A hierarchical feature selection strategy for deepfake video detection. Neural Comput. Appl. (2023). https://doi.org/10.1007/s00521-023-08201-z

    Article  Google Scholar 

  52. Deng, L., Wang, J., Liu, Z.: Cascaded network based on EfficientNet and transformer for deepfake video detection. Neural. Process. Lett. (2023). https://doi.org/10.1007/s11063-023-11249-6

    Article  Google Scholar 

  53. Asha, S., Vinod, P., Menon, V.G.: A defensive framework for deepfake detection under adversarial settings using temporal and spatial features. Int. J. Inf. Secur. (2023). https://doi.org/10.1007/s10207-023-00695-x

    Article  Google Scholar 

  54. Ke, J., Wang, L.: DF-UDetector: An effective method towards robust deepfake detection via feature restoration. Neural Netw. 160, 216–226 (2023)

    Article  PubMed  Google Scholar 

  55. Guo, Z., Yang, G., Wang, D., Zhang, D.: A data augmentation framework by mining structured features for fake face image detection. Comput. Vis. Image Underst. (2023). https://doi.org/10.1016/j.cviu.2022.103587

    Article  Google Scholar 

  56. Zhao, C., Wang, C., Hu, G., Chen, H., Liu, C., Tang, J.: ISTVT: interpretable spatial-temporal video transformer for deepfake detection. IEEE Trans. Inf. Forensics Secur. 18, 1335–1348 (2023)

    Article  Google Scholar 

  57. Yu, Y., Zhao, X., Ni, R., Yang, S., Zhao, Y., Kot, A.C.: Augmented multi-scale spatiotemporal inconsistency magnifier for generalized DeepFake detection. IEEE Trans. Multimed. 99, 1–13 (2023)

    Google Scholar 

  58. Yang, Z., Liang, J., Xu, Y., Zhang, X.-Y., He, R.: Masked relation learning for DeepFake detection. IEEE Trans. Inf. Forensics Secur. 18, 1696–1708 (2023)

    Article  Google Scholar 

  59. Ganguly, S., Ganguly, A., Mohiuddin, S., Malakar, S., Sarkar, R.: ViXNet: vision transformer with xception network for deepfakes based video and image forgery detection. Expert Syst. Appl. (2022). https://doi.org/10.1016/j.eswa.2022.118423

    Article  Google Scholar 

  60. Ganguly, S., Mohiuddin, S., Malakar, S., Cuevas, E., Sarkar, R.: Visual attention-based deepfake video forgery detection. Pattern Anal. Appl. 25, 981–992 (2022)

    Article  Google Scholar 

  61. Nadimpalli, A.V., Rattani, A.: On improving cross-dataset generalization of deepfake detectors. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), New Orleans, LA, USA (2022)

  62. Li, G., Cao, Y., Zhao, X.: Exploiting facial symmetry to expose deepfakes. In: IEEE International Conference on Image Processing (ICIP), Anchorage (2021)

  63. Li, L., Bao, J., Zhang, T., Yang, H., Chen, D., Wen, F., Guo, B.: Face X-ray for more general face forgery detection. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle (2020)

  64. Luo, Z., Kamata, S.-I., Sun, Z.: Transformer and node-compressed Dnn based dual-path system for manipulated face detection. In: IEEE International Conference on Image Processing (ICIP), Anchorage (2021)

  65. Qi, H., Guo, Q., Xu, F., Xie, X., Ma, L., Feng, W., Liu, Y., Zhao, J.: DeepRhythm: exposing DeepFakes with attentional visual heartbeat rhythms. In: 28th ACM International Conference on Multimedia, Lisboa (2020)

  66. Li, X., Lang, Y., Chen, Y., Mao, X., He, Y., Wang, S., Xue, H., Lu, Q.: Sharp multiple instance learning for DeepFake video detection. In: Proceedings of the 28th ACM International Conference on Multimedia, Seattle WA USA (2020)

  67. Mittal, T., Bhattacharya, U., Chandra, R., Bera, A., Manocha, D.: Emotions don’t lie: an audio-visual deepfake detection method using affective cues. In: 28th ACM International Conference on Multimedia, Lisboa (2020)

  68. Montserrat, D.M., Hao, H., Yarlagadda, S.K., Baireddy, S., Shao, R., Horváth, J., Bartusiak, E., Yang, J., Güera, D., Zhu, F., Delp, E.J.: Deepfakes detection with automatic face weighting. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Seattle (2020)

  69. Chugh, K., Gupta, P., Dhall, A., Subramanian, R.: Not made for each other-audio-visual dissonance-based deepfake detection and localization. In: 28th ACM International Conference on Multimedia, Lisboa (2020)

  70. Li, G., Zhao, X., Cao, Y.: Forensic symmetry for DeepFakes. IEEE Trans. Inf. Forensics Secur. 18, 1095–1110 (2023)

    Article  Google Scholar 

  71. Qian, Y., Yin, G., Sheng, L., Chen, Z., Shao, J.: Thinking in frequency: face forgery detection by mining frequency-aware clues. In: European Conference on Computer Vision (2020)

Download references

Funding

No funding was received to assist with the preparation of this manuscript.

Author information

Authors and Affiliations

  1. Department of Information Technology, Delhi Technological University, Bawana Road, Delhi, 110042, India

    Ankit Yadav, Dhruv Gupta & Dinesh Kumar Vishwakarma

Authors
  1. Ankit Yadav
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Dhruv Gupta
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Dinesh Kumar Vishwakarma
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Ankit Yadav: Software, Validation, Investigation, Data Curation, Writing – Original Draft, Visualization. Dhruv Gupta: Software, Validation, Investigation, Data Curation, Writing – Original Draft, Visualization. Dinesh Kumar Vishwakarma: Conceptualization, Methodology, Formal Analysis, Resources, Writing – Review & Editing, Supervision, Project Administration, Funding Acquisition.

Corresponding author

Correspondence to Dinesh Kumar Vishwakarma.

Ethics declarations

Conflict of interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yadav, A., Gupta, D. & Vishwakarma, D.K. Uncovering visual attention-based multi-level tampering traces for face forgery detection. SIViP 18, 1259–1272 (2024). https://doi.org/10.1007/s11760-023-02774-x

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11760-023-02774-x

Keywords

Search

Navigation