Skip to main content
SpringerLink
Log in

Two-Branch Recurrent Network for Isolating Deepfakes in Videos

  • Conference paper
  • First Online:
Book cover Computer Vision – ECCV 2020 (ECCV 2020)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 12352))

Included in the following conference series:

Abstract

The current spike of hyper-realistic faces artificially generated using deepfakes calls for media forensics solutions that are tailored to video streams and work reliably with a low false alarm rate at the video level. We present a method for deepfake detection based on a two-branch network structure that isolates digitally manipulated faces by learning to amplify artifacts while suppressing the high-level face content. Unlike current methods that extract spatial frequencies as a preprocessing step, we propose a two-branch structure: one branch propagates the original information, while the other branch suppresses the face content yet amplifies multi-band frequencies using a Laplacian of Gaussian (LoG) as a bottleneck layer. To better isolate manipulated faces, we derive a novel cost function that, unlike regular classification, compresses the variability of natural faces and pushes away the unrealistic facial samples in the feature space. Our two novel components show promising results on the FaceForensics+ +, Celeb-DF, and Facebook’s DFDC preview benchmarks, when compared to prior work. We then offer a full, detailed ablation study of our network architecture and cost function. Finally, although the bar is still high to get very remarkable figures at a very low false alarm rate, our study shows that we can achieve good video-level performance when cross-testing in terms of video-level AUC.

I. Masi and A. Killekar contributed equally to this paper.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    Throughout this paper \(\mathbf {I}\) indicates a sequence (or window) of aligned faces from video frames of cardinality F.

  2. 2.

    The dimensionality is doubled since the results of the bi-directional streams are concatenated.

References

  1. CNN - business - when seeing is no longer believing inside the pentagon’s race against deepfake videos. https://www.cnn.com/interactive/2019/01/business/pentagons-race-against-deepfakes/

  2. DeepFaceLab. https://github.com/iperov/DeepFaceLab

  3. DeepTrace - the antivirus of deepfakes - the state of deepfakes. https://deeptracelabs.com

  4. ZAO app. https://apps.apple.com/cn/app/zao/

  5. MSR Image Recognition Challenge (IRC) at ACM Multimedia 2016, July 2016

    Google Scholar 

  6. Afchar, D., Nozick, V., Yamagishi, J., Echizen, I.: Mesonet: a compact facial video forgery detection network. In: WIFS, pp. 1–7. IEEE (2018)

    Google Scholar 

  7. Agarwal, S., Farid, H., Gu, Y., He, M., Nagano, K., Li, H.: Protecting world leaders against deep fakes. In: CVPR Workshops, June 2019

    Google Scholar 

  8. Bayar, B., Stamm, M.C.: A deep learning approach to universal image manipulation detection using a new convolutional layer. In: ACM Workshop on Information Hiding and Multimedia Security, pp. 5–10 (2016)

    Google Scholar 

  9. Bulat, A., Tzimiropoulos, G.: How far are we from solving the 2D & 3D face alignment problem? (and a dataset of 230,000 3D facial landmarks). In: ICCV (2017)

    Google Scholar 

  10. Burt, P., Adelson, E.: The laplacian pyramid as a compact image code. IEEE Trans. Commun. 31(4), 532–540 (1983)

    Article  Google Scholar 

  11. Chollet, F.: Xception: deep learning with depthwise separable convolutions. In: CVPR, pp. 1251–1258 (2017). http://openaccess.thecvf.com/content_cvpr_2017/html/Chollet_Xception_Deep_Learning_CVPR_2017_paper.html

  12. Cozzolino, D., Poggi, G., Verdoliva, L.: Recasting residual-based local descriptors as convolutional neural networks: an application to image forgery detection. In: ACM Workshop on Information Hiding and Multimedia Security, pp. 159–164 (2017)

    Google Scholar 

  13. Cozzolino, D., Thies, J., Rössler, A., Riess, C., Nießner, M., Verdoliva, L.: Forensictransfer: weakly-supervised domain adaptation for forgery detection. arXiv preprint arXiv:1812.02510 (2018)

  14. Dolhansky, B., Howes, R., Pflaum, B., Baram, N., Ferrer, C.C.: The Deepfake Detection Challenge (DFDC) Preview Dataset. arXiv:1910.08854, October 2019. http://arxiv.org/abs/1910.08854, arXiv: 1910.08854

  15. Domingos, P.M.: A few useful things to know about machine learning. Commun. ACM 55(10), 78–87 (2012)

    Article  Google Scholar 

  16. Dufour, N., et al.: Deepfakes detection dataset by Google and Jigsaw (2019)

    Google Scholar 

  17. Farid, H.: Photo Forensics. MIT Press, Cambridge (2016)

    Book  Google Scholar 

  18. Fridrich, J., Kodovsky, J.: Rich models for steganalysis of digital images. TIFS 7(3), 868–882 (2012)

    Google Scholar 

  19. Gellately, R.: Lenin, Stalin, and Hitler: The age of social catastrophe. Alfred a Knopf Incorporated (2007)

    Google Scholar 

  20. Goodfellow, I., et al.: Generative adversarial nets. In: NIPS (2014)

    Google Scholar 

  21. Güera, D., Delp, E.J.: Deepfake video detection using recurrent neural networks. In: AVSS, pp. 1–6. IEEE (2018)

    Google Scholar 

  22. Guo, Y., Zhang, L., Hu, Y., He, X., Gao, J.: MS-Celeb-1M: a dataset and benchmark for large-scale face recognition. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9907, pp. 87–102. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46487-9_6

    Chapter  Google Scholar 

  23. Han, X., Morariu, V., Larry Davis, P.I., et al.: Two-stream neural networks for tampered face detection. In: CVPR Workshops, pp. 19–27 (2017)

    Google Scholar 

  24. Heller, S., Rossetto, L., Schuldt, H.: The PS-Battles Dataset - an Image Collection for Image Manipulation Detection. CoRR abs/1804.04866 (2018). http://arxiv.org/abs/1804.04866

  25. Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: CVPR, pp. 4700–4708 (2017)

    Google Scholar 

  26. Huang, G.B., Ramesh, M., Berg, T., Learned-Miller, E.: Labeled faces in the wild: a database for studying face recognition in unconstrained environments. Technical report, 07-49, UMass, Amherst, October 2007

    Google Scholar 

  27. Ioannou, Y., Robertson, D., Cipolla, R., Criminisi, A.: Deep roots: improving CNN efficiency with hierarchical filter groups. In: CVPR, pp. 1231–1240 (2017)

    Google Scholar 

  28. Karras, T., Laine, S., Aila, T.: A style-based generator architecture for generative adversarial networks. In: CVPR, pp. 4401–4410 (2019)

    Google Scholar 

  29. Kemelmacher-Shlizerman, I., Seitz, S.M., Miller, D., Brossard, E.: The MegaFace benchmark: 1 million faces for recognition at scale. In: CVPR (2016)

    Google Scholar 

  30. King, D.E.: Dlib-ml: a machine learning toolkit. JMLR 10, 1755–1758 (2009)

    Google Scholar 

  31. Kingma, D.P., Welling, M.: Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 (2013)

  32. Klare, B.F., et al.: Pushing the frontiers of unconstrained face detection and recognition: IARPA Janus Benchmark A. In: CVPR, pp. 1931–1939 (2015)

    Google Scholar 

  33. Korshunov, P., Marcel, S.: Deepfakes: a new threat to face recognition? assessment and detection. arXiv preprint arXiv:1812.08685 (2018)

  34. Korshunov, P., Marcel, S.: Vulnerability assessment and detection of deepfake videos. In: ICB, Crete, Greece, June 2019

    Google Scholar 

  35. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: NIPS, pp. 1097–1105 (2012)

    Google Scholar 

  36. Li, Y., Chang, M.C., Lyu, S.: In ictu oculi: exposing AI created fake videos by detecting eye blinking. In: WIFS, pp. 1–7 (2018)

    Google Scholar 

  37. Li, Y., Lyu, S.: Exposing deepfake videos by detecting face warping artifacts. In: CVPR Workshops, June 2019

    Google Scholar 

  38. Li, Y., Yang, X., Sun, P., Qi, H., Lyu, S.: Celeb-DF: a large-scale challenging dataset for deepfake forensics. In: CVPR, June 2020

    Google Scholar 

  39. Maaten, L.V.D., Hinton, G.: Visualizing data using t-SNE. J. Mach. Learn. Res. 9(Nov), 2579–2605 (2008)

    Google Scholar 

  40. Marra, F., Gragnaniello, D., Verdoliva, L., Poggi, G.: Do GANs leave artificial fingerprints? In: Conference on Multimedia Information Processing and Retrieval (MIPR), pp. 506–511 (2019)

    Google Scholar 

  41. Matern, F., Riess, C., Stamminger, M.: Exploiting visual artifacts to expose deepfakes and face manipulations. In: WACV Workshops, pp. 83–92. IEEE (2019)

    Google Scholar 

  42. McClish, D.K.: Analyzing a portion of the ROC curve. Med. Decis. Making 9(3), 190–195 (1989)

    Article  Google Scholar 

  43. Nagrani, A., Chung, J.S., Xie, W., Zisserman, A.: Voxceleb: large-scale speaker verification in the wild. Comput. Speech Lang. 60, 101027 (2020)

    Article  Google Scholar 

  44. Nguyen, H.H., Fang, F., Yamagishi, J., Echizen, I.: Multi-task learning for detecting and segmenting manipulated facial images and videos. In: BTAS (2019)

    Google Scholar 

  45. Nguyen, H.H., Yamagishi, J., Echizen, I.: Capsule-forensics: using capsule networks to detect forged images and videos. In: ICASSSP, pp. 2307–2311. IEEE (2019)

    Google Scholar 

  46. Nguyen, T.T., Nguyen, C.M., Nguyen, D.T., Nguyen, D.T., Nahavandi, S.: Deep learning for deepfakes creation and detection. arXiv preprint arXiv:1909.11573 (2019)

  47. Nirkin, Y., Masi, I., Tran, A., Hassner, T., Medioni, G.: On face segmentation, face swapping, and face perception. In: AFGR (2018)

    Google Scholar 

  48. Pedro, D.: A unified bias-variance decomposition and its applications. In: 17th International Conference on Machine Learning, pp. 231–238 (2000)

    Google Scholar 

  49. Rahmouni, N., Nozick, V., Yamagishi, J., Echizen, I.: Distinguishing computer graphics from natural images using convolution neural networks. In: WIFS, pp. 1–6 (2017)

    Google Scholar 

  50. Rössler, A., Cozzolino, D., Verdoliva, L., Riess, C., Thies, J., Nießner, M.: Faceforensics++: learning to detect manipulated facial images. In: ICCV (2019)

    Google Scholar 

  51. Ruff, L., et al.: Deep one-class classification. In: ICML, pp. 4393–4402 (2018)

    Google Scholar 

  52. Sabir, E., Cheng, J., Jaiswal, A., AbdAlmageed, W., Masi, I., Natarajan, P.: Recurrent convolutional strategies for face manipulation detection in videos. In: CVPR Workshops, pp. 80–87 (2019)

    Google Scholar 

  53. Sanderson, C., Lovell, B.C.: Multi-region probabilistic histograms for robust and scalable identity inference. In: Tistarelli, M., Nixon, M.S. (eds.) ICB 2009. LNCS, vol. 5558, pp. 199–208. Springer, Heidelberg (2009). https://doi.org/10.1007/978-3-642-01793-3_21

    Chapter  Google Scholar 

  54. Schroff, F., Kalenichenko, D., Philbin, J.: Facenet: a unified embedding for face recognition and clustering. In: CVPR (2015)

    Google Scholar 

  55. Stehouwer, J., Dang, H., Liu, F., Liu, X., Jain, A.: On the detection of digital face manipulation. arXiv preprint arXiv:1910.01717 (2019)

  56. Thies, J., Zollhöfer, M., Nießner, M.: Deferred neural rendering: image synthesis using neural textures. ACM Trans. Graph. (TOG) 38(4), 1–12 (2019)

    Article  Google Scholar 

  57. Thies, J., Zollhofer, M., Stamminger, M., Theobalt, C., Nießner, M.: Face2face: real-time face capture and reenactment of RGB videos. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2387–2395 (2016)

    Google Scholar 

  58. Tolosana, R., Vera-Rodriguez, R., Fierrez, J., Morales, A., Ortega-Garcia, J.: Deepfakes and beyond: a survey of face manipulation and fake detection. arXiv preprint arXiv:2001.00179 (2020)

  59. Valentini, G., Dietterich, T.G.: Bias-variance analysis of support vector machines for the development of SVM-based ensemble methods. JMLR 5(Jul), 725–775 (2004)

    Google Scholar 

  60. Verdoliva, D.C.G.P.L.: Extracting camera-based fingerprints for video forensics (2019)

    Google Scholar 

  61. Verdoliva, L.: Media forensics and deepfakes: an overview. IEEE J. Sel. Top. Signal Process. (2020)

    Google Scholar 

  62. Wang, F., Cheng, J., Liu, W., Liu, H.: Additive margin softmax for face verification. IEEE Signal Process. Lett. 25(7), 926–930 (2018)

    Article  Google Scholar 

  63. Weisstein, E.W.: Hypersphere (2002)

    Google Scholar 

  64. Wen, Y., Zhang, K., Li, Z., Qiao, Yu.: A discriminative feature learning approach for deep face recognition. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9911, pp. 499–515. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46478-7_31

    Chapter  Google Scholar 

  65. Wu, C.Y., Manmatha, R., Smola, A.J., Krahenbuhl, P.: Sampling matters in deep embedding learning. In: ICCV, October 2017

    Google Scholar 

  66. Xie, C., Wu, Y., Maaten, L.V.D., Yuille, A.L., He, K.: Feature denoising for improving adversarial robustness. In: CVPR, pp. 501–509 (2019)

    Google Scholar 

  67. Yang, X., Li, Y., Lyu, S.: Exposing deep fakes using inconsistent head poses. In: ICASSSP, pp. 8261–8265. IEEE (2019)

    Google Scholar 

  68. Yu, N., Davis, L.S., Fritz, M.: Attributing fake images to GANs: learning and analyzing GAN fingerprints. In: ICCV, pp. 7556–7566 (2019)

    Google Scholar 

  69. Zhang, X., Karaman, S., Chang, S.F.: Detecting and simulating artifacts in GAN fake images. In: WIFS (2019)

    Google Scholar 

Download references

Acknowledgment

This work is based on research sponsored by the Defense Advanced Research Projects Agency under agreement number FA8750-16-2-0204. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright notation thereon. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the Defense Advanced Research Projects Agency or the U.S. Government. The authors would like to thank E. Sabir and A. Jaiswal for the useful discussions and the anonymous reviewers.

Author information

Authors and Affiliations

  1. USC Information Sciences Institute, Marina del Rey, CA, USA

    Iacopo Masi, Aditya Killekar, Royston Marian Mascarenhas, Shenoy Pratik Gurudatt & Wael AbdAlmageed

Authors
  1. Iacopo Masi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Aditya Killekar
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Royston Marian Mascarenhas
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Shenoy Pratik Gurudatt
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Wael AbdAlmageed
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Iacopo Masi .

Editor information

Editors and Affiliations

  1. University of Oxford, Oxford, UK

    Andrea Vedaldi

  2. Graz University of Technology, Graz, Austria

    Horst Bischof

  3. University of Freiburg, Freiburg im Breisgau, Germany

    Thomas Brox

  4. University of North Carolina at Chapel Hill, Chapel Hill, NC, USA

    Jan-Michael Frahm

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Masi, I., Killekar, A., Mascarenhas, R.M., Gurudatt, S.P., AbdAlmageed, W. (2020). Two-Branch Recurrent Network for Isolating Deepfakes in Videos. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, JM. (eds) Computer Vision – ECCV 2020. ECCV 2020. Lecture Notes in Computer Science(), vol 12352. Springer, Cham. https://doi.org/10.1007/978-3-030-58571-6_39

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-58571-6_39

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-58570-9

  • Online ISBN: 978-3-030-58571-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation