Skip to main content
SpringerLink
Log in

TAN-GFD: generalizing face forgery detection based on texture information and adaptive noise mining

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Face forgery detection has become a research hotspot due to security concerns about spreading ultrarealisitc fake faces over social platforms. However, most existing deep learning-based approaches fail to generalize in cross-dataset scenarios since the learning-based methods tend to overfit manipulation-specific artifacts and advanced manipulations tamper with the target face locally or globally. In this work, we find that multiscale texture differences and regional noise inconsistencies are two intrinsic but complementary forged clues in the face manipulation pipeline. To comprehensively dig into generalized forgery clues, we propose a novel framework named TAN-GFD, based on texture information and adaptive noise mining. Specifically, we design a texture difference representation block that combines pixel intensity and gradient information of feature maps to extract multiscale texture difference features from different shallow feature maps. Moreover, since face tampering in real scenes swaps the whole face or partial facial expressions, we thus design the multilevel adaptive noise mining module, which consists of data preprocessing with learnable SRM filters and a cross-modality feature pyramid block, to capture the abundant features of regional noise inconsistencies. In addition, we introduce the cross-entropy loss with supervised contrastive loss collaboration strategy to guide the framework in learning more generalized representations. Extensive experiments on several benchmark datasets demonstrate the effectiveness and superior generalization performance of our framework.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Data Availability

The data that support the findings of this study are available from the corresponding author upon reasonable request.

References

  1. Faceswap. https://www.github.com/MarekKowalski/FaceSwap Accessed 2022-3

  2. Deepfakes. https://github.com/deepfakes/faceswap Accessed 2022-1

  3. Thies J, Zollhofer M, Stamminger M, Theobalt C, Nießner M (2016) Face2face: real-time face capture and reenactment of rgb videos. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2387–2395

  4. Thies J, Zollhöfer M, Nießner M (2019) Deferred neural rendering Image synthesis using neural textures. ACM Trans Graph (TOG) 38(4):1–12

    Article  Google Scholar 

  5. Momina M, Nawaz M, Malik KM, Ali J, Aun I, Malik H (2022) Deepfakes generation and detection: state-of-the-art, open challenges, countermeasures, and way forward. Appl Intell, pp 1–53

  6. Raj C, Meel P (2021) Convnet frameworks for multi-modal fake news detection. Appl Intell 51(11):8132–8148

    Article  Google Scholar 

  7. Afchar D, Nozick V, Yamagishi J, Echizen I (2018) Mesonet: a compact facial video forgery detection network. In: 2018 IEEE international workshop on information forensics and security (WIFS), pp 1–7. IEEE

  8. Rossler A, Cozzolino D, Verdoliva L, Riess C, Thies J, Nießner M (2019) Faceforensics++ Learning to detect manipulated facial images. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 1–11

  9. Bonettini N, Cannas ED, Mandelli S, Bondi L, Bestagini P, Tubaro S (2021) Video face manipulation detection through ensemble of cnns. In: 2020 25th International conference on pattern recognition (ICPR), pp 5012–5019. IEEE

  10. Yang X, Liu S, Dong Y, Su H, Zhang L, Zhu J (2022) Towards generalizable detection of face forgery via self-guided model-agnostic learning. Pattern Recogn Lett 160:98–104

    Article  Google Scholar 

  11. Nadimpalli AV, Rattani A (2022) On improving cross-dataset generalization of deepfake detectors. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 91–99

  12. Chai L, Bau D, Lim S-N, Isola P (2020) What makes fake images detectable? understanding properties that generalize. In: European conference on computer vision, pp 103–120. Springer

  13. Heo Y-J, Yeo W-H, Kim B-G (2022) Deepfake detection algorithm based on improved vision transformer. Appl Intell, pp 1–16

  14. Agarwal A, Agarwal A, Sinha S, Vatsa M, Singh R (2021) Md-csdnetwork: multi-domain cross stitched network for deepfake detection. In: 2021 16th IEEE international conference on automatic face and gesture recognition (FG 2021), pp 1–8. IEEE

  15. Wang X, Yao T, Ding S, Ma L (2020) Face manipulation detection via auxiliary supervision. In: International conference on neural information processing, pp 313–324. Springer

  16. Elhassan A, Al-Fawa’reh M, Jafar MT, Ababneh M, Jafar ST (2022) Dft-mf: enhanced deepfake detection using mouth movement and transfer learning. SoftwareX 19:101115

    Article  Google Scholar 

  17. Kim M, Tariq S, Woo SS (2021) Fretal: generalizing deepfake detection using knowledge distillation and representation learning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 1001–1012

  18. Li Y, Chang M-C, Lyu S (2018) In ictu oculi: exposing ai created fake videos by detecting eye blinking. In: 2018 IEEE International workshop on information forensics and security (WIFS), pp 1–7. IEEE

  19. Yang X, Li Y, Lyu S (2019) Exposing deep fakes using inconsistent head poses. In: ICASSP 2019-2019 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp 8261–8265. IEEE

  20. Li L, Bao J, Zhang T, Yang H, Chen D, Wen F, Guo B (2020) Face x-ray for more general face forgery detection. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 5001–5010

  21. Durall R, Keuper M, Keuper J (2020) Watch your up-convolution: Cnn based generative deep neural networks are failing to reproduce spectral distributions. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 7890–7899

  22. Yu Z, Zhao C, Wang Z, Qin Y, Zhuo S, Li X, Zhou F, Zhao G (2020) Searching central difference convolutional networks for face anti-spoofing. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 5295–5305

  23. Gatys L, Ecker AS, Bethge M (2015) Texture synthesis using convolutional neural networks. Advances in Neural Information Processing Systems, vol 28

  24. Zhou P, Han X, Morariu VI, Davis LS (2018) Learning rich features for image manipulation detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1053–1061

  25. Khosla P, Teterwak P, Wang C, Sarna A, Tian Y, Isola P, Maschinot A, Ce L, Krishnan D (2020) Supervised contrastive learning. Adv Neural Inf Process Syst 33:18661–18673

    Google Scholar 

  26. Li Y, Yang X, Sun P, Qi H, Lyu S (2020) Celeb-df: a large-scale challenging dataset for deepfake forensics. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 3207–3216

  27. Deepfake detection challange. https://www.kaggle.com/c/deepfake-detection-challenge/ Accessed 2022-2

  28. Dale K, Sunkavalli K, Johnson M K, Vlasic D, Matusik W, Pfister H (2011) Video face replacement. In: Proceedings of the 2011 SIGGRAPH asia conference, pp 1–10

  29. Garrido P, Valgaerts L, Rehmsen O, Thormahlen T, Perez P, Theobalt C (2014) Automatic face reenactment. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4217–4224

  30. Wu H, Ma D, Mao Z, Sun J (2022) Ssrfd: single shot real-time face detector. Appl Intell, pp 1–12

  31. Huang Z, Ren F, Hu M, Chen S (2020) Facial expression imitation method for humanoid robot based on smooth-constraint reversed mechanical model (srmm). IEEE Transactions on Human-Machine Systems 50(6):538–549

    Article  Google Scholar 

  32. Faceswap-gan. https://github.com/shaoanlu/faceswap-GAN Accessed 2022-2

  33. Zhang L, Yang H, Qiu T, Li L (2021) Ap-gan: improving attribute preservation in video face swapping. IEEE Trans Circuits Syst Video Technol 32(4):2226–2237

    Article  Google Scholar 

  34. Zhang J, Zeng X, Wang M, Pan Y, Liu L, Liu Y, Ding Y, Changjie Fan. (2020) Freenet: Multi-identity face reenactment. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 5326–5335

  35. Deepfacelab. https://github.com/iperov/DeepFaceLab/?utm_source=catalyzex.com Accessed 2022-1

  36. Faceapp. https://apps.apple.com/gb/app/faceapp-ai-face-editor/id1180884341 Accessed 2022-1

  37. Nguyen HH, Yamagishi J, Echizen I (2019) Capsule-forensics: using capsule networks to detect forged images and videos. In: ICASSP 2019-2019 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp 2307–2311. IEEE

  38. Chollet F (2017) Xception: deep learning with depthwise separable convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1251–1258

  39. Zhao H, Zhou W, Chen D, Wei T, Zhang W, Yu N (2021) Multi-attentional deepfake detection. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 2185–2194

  40. Dang H, Liu F, Stehouwer J, Liu X, Jain AK (2020) On the detection of digital face manipulation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 5781–5790

  41. Liu H, Li X, Zhou W, Chen Y, He Y, Xue H, Zhang W, Yu N (2021) Spatial-phase shallow learning: rethinking face forgery detection in frequency domain. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 772–781

  42. Yang J, Li A, Xiao S, Lu W, Gao X (2021) Mtd-net: learning to detect deepfakes images by multi-scale texture difference. IEEE Trans Inf Forensics Secur 16:4234–4245

    Article  Google Scholar 

  43. Luo Y, Zhang Y, Yan J, Liu W (2021) Generalizing face forgery detection with high-frequency features. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 16317–16326

  44. Zhang K, Zhang Z, Li Z, Yu Q (2016) Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Process Lett 23(10):1499–1503

    Article  Google Scholar 

  45. King DE (2009) Dlib-ml: a machine learning toolkit. The Journal of Machine Learning Research 10:1755–1758

    Google Scholar 

  46. Haralick RM, Shanmugam K, Its’ Hak D (1973) Textural features for image classification. IEEE Transactions on Systems, man, and Cybernetics, (6), pp 610–621

  47. Mahdian B, Saic S (2009) Using noise inconsistencies for blind image forensics. Image Vis Comput 27(10):1497–1503

    Article  Google Scholar 

  48. Cozzolino D, Verdoliva L (2019) Noiseprint: a cnn-based camera model fingerprint. IEEE Trans Inf Forensics Secur 15:144–159

    Article  Google Scholar 

  49. Wang G, Jiang Q, Jin X, Li W, Cui X (2022) Mc-lcr: Multimodal contrastive classification by locally correlated representations for effective face forgery detection. Knowledge-Based Systems, pp 109114

  50. Tan M, Le Q (2019) Efficientnet: rethinking model scaling for convolutional neural networks. In: International conference on machine learning, pp 6105–6114. PMLR

  51. Liu J, Zhu K, Lu W, Luo X, Zhao X (2021) A lightweight 3d convolutional neural network for deepfake detection. Int J Intell Syst 36(9):4990–5004

    Article  Google Scholar 

  52. Lin Tsung-Yi, Dollár P, Girshick R, He K, Hariharan B, Belongie S (2017) Feature pyramid networks for object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2117–2125

  53. Kingma DP, Ba J (2014) Adam: a method for stochastic optimization. arXiv:1412.6980

  54. Fridrich J, Kodovsky J (2012) Rich models for steganalysis of digital images. IEEE Transactions on information Forensics and Security 7(3):868–882

    Article  Google Scholar 

  55. Cozzolino D, Poggi G, Verdoliva L (2017) Recasting residual-based local descriptors as convolutional neural networks: an application to image forgery detection. In: Proceedings of the 5th ACM workshop on information hiding and multimedia security, pp 159–164

  56. Bayar B, Stamm MC (2016) A deep learning approach to universal image manipulation detection using a new convolutional layer. In: Proceedings of the 4th ACM workshop on information hiding and multimedia security, pp 5–10

  57. Nguyen HH, Fang F, Yamagishi J, Echizen I (2019) Multi-task learning for detecting and segmenting manipulated facial images and videos. In: 2019 IEEE 10th international conference on biometrics theory, applications and systems (BTAS), pp 1–8. IEEE

  58. Zhou P, Han X, Morariu VI, Davis LS (2017) Two-stream neural networks for tampered face detection. In: 2017 IEEE conference on computer vision and pattern recognition workshops (CVPRW), pp 1831–1839. IEEE

  59. Nirkin Y, Wolf L, Keller Y, Hassner T (2021) Deepfake detection based on discrepancies between faces and their context. IEEE Transactions on Pattern Analysis and Machine Intelligence

  60. Schwarcz S, Chellappa R (2021) Finding facial forgery artifacts with parts-based detectors. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 933–942

  61. Wang J, Wu Z, Ouyang W, Han X, Chen J, Jiang Y-G, Li Ser-Nam (2022) M2tr: multi-modal multi-scale transformers for deepfake detection. In: Proceedings of the 2022 international conference on multimedia retrieval, pp 615–623

  62. Qian Y, Yin G, Lu S, Chen Z, Shao J (2020) Thinking in frequency Face forgery detection by mining frequency-aware clues. In: European conference on computer vision, pp 86–103. Springer

  63. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  64. Li J, Xie H, Yu L, Gao X, Zhang Y (2021) Discriminative feature mining based on frequency information and metric learning for face forgery detection. IEEE Transactions on Knowledge and Data Engineering

  65. Van der Maaten L, Hinton G (2008) Visualizing data using t-sne. Journal of machine learning research, vol 9(11)

  66. Fung S, Lu X, Zhang C, Li C-T (2021) Deepfakeucl: Deepfake detection via unsupervised contrastive learning. In: 2021 International joint conference on neural networks (IJCNN), pp 1–8. IEEE

Download references

Acknowledgements

This study is supported by the National Natural Science Foundation of China (No. 62002313, 61862067), and Key Areas Research Program of Yunnan Province in China (No. 202001BB050076).

Author information

Authors and Affiliations

  1. School of Information Science Engineering, Yunnan University, Kunming, Yunnan, 650091, China

    Yi Zhao

  2. Engineering Research Center of Cyberspace and the School of Software, Yunnan University, Kunming, Yunnan, 650091, China

    Xin Jin, Song Gao, Liwen Wu, Shaowen Yao & Qian Jiang

Authors
  1. Yi Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xin Jin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Song Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Liwen Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Shaowen Yao
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Qian Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Shaowen Yao or Qian Jiang.

Ethics declarations

Competing interests

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhao, Y., Jin, X., Gao, S. et al. TAN-GFD: generalizing face forgery detection based on texture information and adaptive noise mining. Appl Intell 53, 19007–19027 (2023). https://doi.org/10.1007/s10489-023-04462-2

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-023-04462-2

Keywords

Search

Navigation