Two-Branch Recurrent Network for Isolating Deepfakes in Videos

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12352)


The current spike of hyper-realistic faces artificially generated using deepfakes calls for media forensics solutions that are tailored to video streams and work reliably with a low false alarm rate at the video level. We present a method for deepfake detection based on a two-branch network structure that isolates digitally manipulated faces by learning to amplify artifacts while suppressing the high-level face content. Unlike current methods that extract spatial frequencies as a preprocessing step, we propose a two-branch structure: one branch propagates the original information, while the other branch suppresses the face content yet amplifies multi-band frequencies using a Laplacian of Gaussian (LoG) as a bottleneck layer. To better isolate manipulated faces, we derive a novel cost function that, unlike regular classification, compresses the variability of natural faces and pushes away the unrealistic facial samples in the feature space. Our two novel components show promising results on the FaceForensics+ +, Celeb-DF, and Facebook’s DFDC preview benchmarks, when compared to prior work. We then offer a full, detailed ablation study of our network architecture and cost function. Finally, although the bar is still high to get very remarkable figures at a very low false alarm rate, our study shows that we can achieve good video-level performance when cross-testing in terms of video-level AUC.


Deepfake detection Two-branch recurrent net Loss function 



This work is based on research sponsored by the Defense Advanced Research Projects Agency under agreement number FA8750-16-2-0204. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright notation thereon. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the Defense Advanced Research Projects Agency or the U.S. Government. The authors would like to thank E. Sabir and A. Jaiswal for the useful discussions and the anonymous reviewers.


  1. 1.
    CNN - business - when seeing is no longer believing inside the pentagon’s race against deepfake videos.
  2. 2.
  3. 3.
    DeepTrace - the antivirus of deepfakes - the state of deepfakes.
  4. 4.
  5. 5.
    MSR Image Recognition Challenge (IRC) at ACM Multimedia 2016, July 2016Google Scholar
  6. 6.
    Afchar, D., Nozick, V., Yamagishi, J., Echizen, I.: Mesonet: a compact facial video forgery detection network. In: WIFS, pp. 1–7. IEEE (2018)Google Scholar
  7. 7.
    Agarwal, S., Farid, H., Gu, Y., He, M., Nagano, K., Li, H.: Protecting world leaders against deep fakes. In: CVPR Workshops, June 2019Google Scholar
  8. 8.
    Bayar, B., Stamm, M.C.: A deep learning approach to universal image manipulation detection using a new convolutional layer. In: ACM Workshop on Information Hiding and Multimedia Security, pp. 5–10 (2016)Google Scholar
  9. 9.
    Bulat, A., Tzimiropoulos, G.: How far are we from solving the 2D & 3D face alignment problem? (and a dataset of 230,000 3D facial landmarks). In: ICCV (2017)Google Scholar
  10. 10.
    Burt, P., Adelson, E.: The laplacian pyramid as a compact image code. IEEE Trans. Commun. 31(4), 532–540 (1983)CrossRefGoogle Scholar
  11. 11.
    Chollet, F.: Xception: deep learning with depthwise separable convolutions. In: CVPR, pp. 1251–1258 (2017).
  12. 12.
    Cozzolino, D., Poggi, G., Verdoliva, L.: Recasting residual-based local descriptors as convolutional neural networks: an application to image forgery detection. In: ACM Workshop on Information Hiding and Multimedia Security, pp. 159–164 (2017)Google Scholar
  13. 13.
    Cozzolino, D., Thies, J., Rössler, A., Riess, C., Nießner, M., Verdoliva, L.: Forensictransfer: weakly-supervised domain adaptation for forgery detection. arXiv preprint arXiv:1812.02510 (2018)
  14. 14.
    Dolhansky, B., Howes, R., Pflaum, B., Baram, N., Ferrer, C.C.: The Deepfake Detection Challenge (DFDC) Preview Dataset. arXiv:1910.08854, October 2019., arXiv: 1910.08854
  15. 15.
    Domingos, P.M.: A few useful things to know about machine learning. Commun. ACM 55(10), 78–87 (2012)CrossRefGoogle Scholar
  16. 16.
    Dufour, N., et al.: Deepfakes detection dataset by Google and Jigsaw (2019)Google Scholar
  17. 17.
    Farid, H.: Photo Forensics. MIT Press, Cambridge (2016)CrossRefGoogle Scholar
  18. 18.
    Fridrich, J., Kodovsky, J.: Rich models for steganalysis of digital images. TIFS 7(3), 868–882 (2012)Google Scholar
  19. 19.
    Gellately, R.: Lenin, Stalin, and Hitler: The age of social catastrophe. Alfred a Knopf Incorporated (2007)Google Scholar
  20. 20.
    Goodfellow, I., et al.: Generative adversarial nets. In: NIPS (2014)Google Scholar
  21. 21.
    Güera, D., Delp, E.J.: Deepfake video detection using recurrent neural networks. In: AVSS, pp. 1–6. IEEE (2018)Google Scholar
  22. 22.
    Guo, Y., Zhang, L., Hu, Y., He, X., Gao, J.: MS-Celeb-1M: a dataset and benchmark for large-scale face recognition. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9907, pp. 87–102. Springer, Cham (2016). Scholar
  23. 23.
    Han, X., Morariu, V., Larry Davis, P.I., et al.: Two-stream neural networks for tampered face detection. In: CVPR Workshops, pp. 19–27 (2017)Google Scholar
  24. 24.
    Heller, S., Rossetto, L., Schuldt, H.: The PS-Battles Dataset - an Image Collection for Image Manipulation Detection. CoRR abs/1804.04866 (2018).
  25. 25.
    Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: CVPR, pp. 4700–4708 (2017)Google Scholar
  26. 26.
    Huang, G.B., Ramesh, M., Berg, T., Learned-Miller, E.: Labeled faces in the wild: a database for studying face recognition in unconstrained environments. Technical report, 07-49, UMass, Amherst, October 2007Google Scholar
  27. 27.
    Ioannou, Y., Robertson, D., Cipolla, R., Criminisi, A.: Deep roots: improving CNN efficiency with hierarchical filter groups. In: CVPR, pp. 1231–1240 (2017)Google Scholar
  28. 28.
    Karras, T., Laine, S., Aila, T.: A style-based generator architecture for generative adversarial networks. In: CVPR, pp. 4401–4410 (2019)Google Scholar
  29. 29.
    Kemelmacher-Shlizerman, I., Seitz, S.M., Miller, D., Brossard, E.: The MegaFace benchmark: 1 million faces for recognition at scale. In: CVPR (2016)Google Scholar
  30. 30.
    King, D.E.: Dlib-ml: a machine learning toolkit. JMLR 10, 1755–1758 (2009)Google Scholar
  31. 31.
    Kingma, D.P., Welling, M.: Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 (2013)
  32. 32.
    Klare, B.F., et al.: Pushing the frontiers of unconstrained face detection and recognition: IARPA Janus Benchmark A. In: CVPR, pp. 1931–1939 (2015)Google Scholar
  33. 33.
    Korshunov, P., Marcel, S.: Deepfakes: a new threat to face recognition? assessment and detection. arXiv preprint arXiv:1812.08685 (2018)
  34. 34.
    Korshunov, P., Marcel, S.: Vulnerability assessment and detection of deepfake videos. In: ICB, Crete, Greece, June 2019Google Scholar
  35. 35.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: NIPS, pp. 1097–1105 (2012)Google Scholar
  36. 36.
    Li, Y., Chang, M.C., Lyu, S.: In ictu oculi: exposing AI created fake videos by detecting eye blinking. In: WIFS, pp. 1–7 (2018)Google Scholar
  37. 37.
    Li, Y., Lyu, S.: Exposing deepfake videos by detecting face warping artifacts. In: CVPR Workshops, June 2019Google Scholar
  38. 38.
    Li, Y., Yang, X., Sun, P., Qi, H., Lyu, S.: Celeb-DF: a large-scale challenging dataset for deepfake forensics. In: CVPR, June 2020Google Scholar
  39. 39.
    Maaten, L.V.D., Hinton, G.: Visualizing data using t-SNE. J. Mach. Learn. Res. 9(Nov), 2579–2605 (2008)Google Scholar
  40. 40.
    Marra, F., Gragnaniello, D., Verdoliva, L., Poggi, G.: Do GANs leave artificial fingerprints? In: Conference on Multimedia Information Processing and Retrieval (MIPR), pp. 506–511 (2019)Google Scholar
  41. 41.
    Matern, F., Riess, C., Stamminger, M.: Exploiting visual artifacts to expose deepfakes and face manipulations. In: WACV Workshops, pp. 83–92. IEEE (2019)Google Scholar
  42. 42.
    McClish, D.K.: Analyzing a portion of the ROC curve. Med. Decis. Making 9(3), 190–195 (1989)CrossRefGoogle Scholar
  43. 43.
    Nagrani, A., Chung, J.S., Xie, W., Zisserman, A.: Voxceleb: large-scale speaker verification in the wild. Comput. Speech Lang. 60, 101027 (2020)CrossRefGoogle Scholar
  44. 44.
    Nguyen, H.H., Fang, F., Yamagishi, J., Echizen, I.: Multi-task learning for detecting and segmenting manipulated facial images and videos. In: BTAS (2019)Google Scholar
  45. 45.
    Nguyen, H.H., Yamagishi, J., Echizen, I.: Capsule-forensics: using capsule networks to detect forged images and videos. In: ICASSSP, pp. 2307–2311. IEEE (2019)Google Scholar
  46. 46.
    Nguyen, T.T., Nguyen, C.M., Nguyen, D.T., Nguyen, D.T., Nahavandi, S.: Deep learning for deepfakes creation and detection. arXiv preprint arXiv:1909.11573 (2019)
  47. 47.
    Nirkin, Y., Masi, I., Tran, A., Hassner, T., Medioni, G.: On face segmentation, face swapping, and face perception. In: AFGR (2018)Google Scholar
  48. 48.
    Pedro, D.: A unified bias-variance decomposition and its applications. In: 17th International Conference on Machine Learning, pp. 231–238 (2000)Google Scholar
  49. 49.
    Rahmouni, N., Nozick, V., Yamagishi, J., Echizen, I.: Distinguishing computer graphics from natural images using convolution neural networks. In: WIFS, pp. 1–6 (2017)Google Scholar
  50. 50.
    Rössler, A., Cozzolino, D., Verdoliva, L., Riess, C., Thies, J., Nießner, M.: Faceforensics++: learning to detect manipulated facial images. In: ICCV (2019)Google Scholar
  51. 51.
    Ruff, L., et al.: Deep one-class classification. In: ICML, pp. 4393–4402 (2018)Google Scholar
  52. 52.
    Sabir, E., Cheng, J., Jaiswal, A., AbdAlmageed, W., Masi, I., Natarajan, P.: Recurrent convolutional strategies for face manipulation detection in videos. In: CVPR Workshops, pp. 80–87 (2019)Google Scholar
  53. 53.
    Sanderson, C., Lovell, B.C.: Multi-region probabilistic histograms for robust and scalable identity inference. In: Tistarelli, M., Nixon, M.S. (eds.) ICB 2009. LNCS, vol. 5558, pp. 199–208. Springer, Heidelberg (2009). Scholar
  54. 54.
    Schroff, F., Kalenichenko, D., Philbin, J.: Facenet: a unified embedding for face recognition and clustering. In: CVPR (2015)Google Scholar
  55. 55.
    Stehouwer, J., Dang, H., Liu, F., Liu, X., Jain, A.: On the detection of digital face manipulation. arXiv preprint arXiv:1910.01717 (2019)
  56. 56.
    Thies, J., Zollhöfer, M., Nießner, M.: Deferred neural rendering: image synthesis using neural textures. ACM Trans. Graph. (TOG) 38(4), 1–12 (2019)CrossRefGoogle Scholar
  57. 57.
    Thies, J., Zollhofer, M., Stamminger, M., Theobalt, C., Nießner, M.: Face2face: real-time face capture and reenactment of RGB videos. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2387–2395 (2016)Google Scholar
  58. 58.
    Tolosana, R., Vera-Rodriguez, R., Fierrez, J., Morales, A., Ortega-Garcia, J.: Deepfakes and beyond: a survey of face manipulation and fake detection. arXiv preprint arXiv:2001.00179 (2020)
  59. 59.
    Valentini, G., Dietterich, T.G.: Bias-variance analysis of support vector machines for the development of SVM-based ensemble methods. JMLR 5(Jul), 725–775 (2004)Google Scholar
  60. 60.
    Verdoliva, D.C.G.P.L.: Extracting camera-based fingerprints for video forensics (2019)Google Scholar
  61. 61.
    Verdoliva, L.: Media forensics and deepfakes: an overview. IEEE J. Sel. Top. Signal Process. (2020)Google Scholar
  62. 62.
    Wang, F., Cheng, J., Liu, W., Liu, H.: Additive margin softmax for face verification. IEEE Signal Process. Lett. 25(7), 926–930 (2018)CrossRefGoogle Scholar
  63. 63.
    Weisstein, E.W.: Hypersphere (2002)Google Scholar
  64. 64.
    Wen, Y., Zhang, K., Li, Z., Qiao, Yu.: A discriminative feature learning approach for deep face recognition. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9911, pp. 499–515. Springer, Cham (2016). Scholar
  65. 65.
    Wu, C.Y., Manmatha, R., Smola, A.J., Krahenbuhl, P.: Sampling matters in deep embedding learning. In: ICCV, October 2017Google Scholar
  66. 66.
    Xie, C., Wu, Y., Maaten, L.V.D., Yuille, A.L., He, K.: Feature denoising for improving adversarial robustness. In: CVPR, pp. 501–509 (2019)Google Scholar
  67. 67.
    Yang, X., Li, Y., Lyu, S.: Exposing deep fakes using inconsistent head poses. In: ICASSSP, pp. 8261–8265. IEEE (2019)Google Scholar
  68. 68.
    Yu, N., Davis, L.S., Fritz, M.: Attributing fake images to GANs: learning and analyzing GAN fingerprints. In: ICCV, pp. 7556–7566 (2019)Google Scholar
  69. 69.
    Zhang, X., Karaman, S., Chang, S.F.: Detecting and simulating artifacts in GAN fake images. In: WIFS (2019)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.USC Information Sciences InstituteMarina del ReyUSA

Personalised recommendations