Skip to main content
SpringerLink
Log in

Spatio-Temporal Deepfake Detection with Deep Neural Networks

  • Conference paper
  • First Online:
Diversity, Divergence, Dialogue (iConference 2021)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 12645))

Included in the following conference series:

Abstract

Deepfakes generated by generative adversarial neural networks may threaten not only individuals but also pose a public threat. In this regard, detecting video content manipulations is an urgent task, and many researchers propose various methods to solve it. Nevertheless, the problem remains. In this paper, the existing approaches are evaluated, and a new method for detecting deepfakes in videos is proposed. Considering that deepfakes are inserted into the video frame by frame, when viewing it, even with the naked eye, fluctuations and temporal distortions are noticeable, which are not taken into account by many deepfake detection algorithms that use information from a single frame to search for forgeries out of context with neighboring frames. It is proposed to analyze information from a sequence of multiple consecutive frames to detect deepfakes in video content by processing the video using the sliding window approach, taking into account not only spatial intraframe dependencies but also interframe temporal dependencies. Experiments have shown the advantage and potential for further development of the proposed approach over simple intraframe recognition.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 99.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 129.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://github.com/shaoanlu/faceswap-GAN.

  2. 2.

    https://apps.apple.com/cn/app/id1465199127.

  3. 3.

    https://apps.apple.com/gb/app/faceapp-ai-face-editor/id1180884341.

  4. 4.

    https://github.com/NVlabs/stylegan.

  5. 5.

    https://www.faceapp.com/.

  6. 6.

    https://conradsanderson.id.au/vidtimit.

  7. 7.

    https://ai.facebook.com/datasets/dfdc.

  8. 8.

    https://generated.photos/.

  9. 9.

    https://thispersondoesnotexist.com.

References

  1. Sebyakin, A.S., Zolotaryuk, A.V.: Tracking emotional state of a person with artificial intelligence methods and its application to customer services. In: Proceedings of the 2019 Twelfth International Conference “Management of Large-Scale System Development” (MLSD), pp. 1–5 (2019). https://doi.org/10.1109/MLSD.2019.8911054

  2. Tolosana, R., Vera-Rodriguez, R., Fierrez, J., Morales, A., Ortega-Garcia, J.: Deepfakes and beyond: a survey of face manipulation and fake detection. Inf. Fusion 64, 131–148 (2020). https://doi.org/10.1016/j.inffus.2020.06.014

    Article  Google Scholar 

  3. Korshunov, P., Marcel, S.: DeepFakes: a new threat to face recognition? Assessment and detection. Idiap-RR-18-2018 (2018). https://arxiv.org/pdf/1812.08685.pdf

  4. Nirkin, Y., Masi, I., Tuan, A.T., Hassner, T., Medioni, G.: On face segmentation, face swapping, and face perception. In: Proceedings of the 2018 13th IEEE International Conference on Automatic Face Gesture Recognition (FG 2018), pp. 98–105 (2018). https://doi.org/10.1109/FG.2018.00024

  5. Ashok Kumar, M., Rajeyyagari, S.: A novel mechanism for dynamic multifarious and disturbed human face recognition using advanced stance coalition (ASC). Comput. Electr. Eng. 86, 1–9 (2020). https://doi.org/10.1016/j.compeleceng.2020.106642

    Article  Google Scholar 

  6. Biggio, B., Korshunov, P., Mensink, T., Patrini, G., Rao, D., Sadhu, A.: Synthetic realities: deep learning for detecting audiovisual fakes. https://sites.google.com/view/audiovisualfakes-icml2019. Accessed 25 Dec 2020

  7. Ding, Zh.,Guo, Y., Zhang, L., Fu, Y.: One-shot face recognition via generative learning. In: Proceedings of the 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition, pp.1–7 (2018). https://doi.org/10.1109/FG.2018.00011

  8. Canton, C., et al.: Applications of computer vision and pattern recognition to media forensics. https://sites.google.com/view/mediaforensics2019. Accessed 25 Dec 2020

  9. Verdoliva, L., Bestagini, P.: Multimedia forensics. In: Proceedings of the 27th ACM International Conference on Multimedia, pp. 2701–2702 (2019). https://doi.org/10.1145/3343031.3350542

  10. Raja, K., et al.: Workshop on deepfakes and presentation attacks in biometrics. https://sites.google.com/view/wacv2020-deeppab. Accessed 25 Dec 2020

  11. Barni, M., Battiato, S., Boato, G., Farid, H., Memon, N.: Multimedia forensics in the wild. https://iplab.dmi.unict.it/mmforwild. Accessed 25 Dec 2020

  12. Hosler, B.C., Stamm, M.C.: Detecting video speed manipulation. In: Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 2860–2869 (2020). https://doi.org/10.1109/CVPRW50498.2020.00343

  13. Lazer, D.M.J., et al.: The science of fake news. Science 359(6380), 1094–1096 (2018). https://doi.org/10.1126/science.aao2998

    Article  Google Scholar 

  14. Karras, T., Aila, T., Laine, S., Lehtinen, J.: Progressive growing of GANs for improved quality, stability, and variation. In: Proceedings of the Sixth International Conference on Learning Representations, pp. 1–26 (2018). https://arxiv.org/pdf/1710.10196.pdf

  15. Karras, T., Laine, S., Aila, T.: A style-based generator architecture for generative adversarial networks. In: Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4401–4410 (2019). https://arxiv.org/pdf/1812.04948v3.pdf

  16. Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., Aila, T.: Analyzing and improving the image quality of StyleGAN. In: Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8110–8119 (2020). https://arxiv.org/pdf/1912.04958.pdf

  17. Gonzalez-Sosa, E., Fierrez, J., Vera-Rodriguez, R., Alonso-Fernandez, F.: Facial soft biometrics for recognition in the wild: recent works, annotation and COTS evaluation. IEEE Trans. Inf. Forensics Secur. 13(8), 2001–2014 (2018). https://doi.org/10.1109/TIFS.2018.2807791

    Article  Google Scholar 

  18. Choi, Y., Choi, M., Kim, M., Ha, J., Kim, S., Choo, J.: StarGAN: unified generative adversarial networks for multi-domain image-to-image translation. In: Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8789–8797 (2018). https://arxiv.org/pdf/1711.09020.pdf

  19. Liu, M., et al.: STGAN: a unified selective transfer network for arbitrary image attribute editing. In: Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3673–3682 (2019). https://arxiv.org/pdf/1904.09709.pdf

  20. Thies, J., Zollhöfer, M., Nießner, M.: Deferred neural rendering: image synthesis using neural textures. ACM Trans. Graph. 38(4), 66 (2019). https://doi.org/10.1145/3306346.3323035

    Article  Google Scholar 

  21. Rössler, A., Cozzolino, D., Verdoliva, L., Riess, C., Thies, J., Nießner, M.: FaceForensics++: learning to detect manipulated facial images. In: Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 11 (2019). https://arxiv.org/pdf/1901.08971.pdf

  22. Li, Y., Yang, X., Sun, P., Qi, H., Lyu, S.: Celeb-DF: a large-scale challenging dataset for DeepFake forensics. In: Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3207–3216 (2020). https://arxiv.org/pdf/1909.12962.pdf

  23. Cao, Q., Shen, L., Xie, W., Parkhi, O., Zisserman, A.: VGGFace2: a dataset for recognizing faces across pose and age. In: Proceedings of the 13th IEEE International Conference on Automatic Face & Gesture Recognition, pp. 67–74 (2018). https://doi.org/10.1109/FG.2018.00020

  24. Zhang, Y., et al.: CelebA-spoof: large-scale face anti-spoofing dataset with rich annotations. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV. LNCS, vol. 12357, pp. 70–85. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58610-2_5

    Chapter  Google Scholar 

  25. Dang, H., Liu, F., Stehouwer, J., Liu, X., Jain, A.: On the detection of digital face manipulation. In: Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5781–5790 (2020). https://arxiv.org/pdf/1910.01717.pdf

  26. Neves, J., Tolosana, R., Vera-Rodriguez, R., Lopes, V., Proença, H., Fierrez, J.: GANprintR: improved fakes and evaluation of the state-of-the-art in face manipulation detection. IEEE J. Sel. Top. Signal Process. 14(5), 1038–1048 (2020). https://doi.org/10.1109/JSTSP.2020.3007250

    Article  Google Scholar 

  27. McCloskey, S., Albright, M.: Detecting GAN-generated imagery using color cues (2018). https://arxiv.org/pdf/1812.08247.pdf

  28. Wang, R., Ma, L., Juefei-Xu, F., Xie, X., Wang, J., Liu, Y.: FakeSpotter: a simple baseline for spotting AI-synthesized fake faces. In: Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, pp. 3444–3451 (2019). https://arxiv.org/pdf/1909.06122v3.pdf

  29. Shen, Y., Gu, J., Tang, X., Zhou, B.: Interpreting the latent space of GANs for semantic face editing. In: Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 12 (2020). https://arxiv.org/pdf/1907.10786.pdf

  30. Guarnera, L., Giudice, O., Battiato, S.: DeepFake detection by analyzing convolutional traces. In: Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern RecognitionWorkshops, p. 10 (2020). https://arxiv.org/pdf/2004.10448v1.pdf

  31. He, Z., Zuo, W., Kan, M., Shan, S., Chen, X.: Attgan: facial attribute editing by only changing what you want. IEEE Trans. Image Process. 28(11), 5464–5478 (2019). https://doi.org/10.1109/TIP.2019.2916751

    Article  MathSciNet  MATH  Google Scholar 

  32. Cho, W., Choi, S., Park, D.K., Shin I., Choo, J.: Image-to-image translation via group-wise deep whitening-and-coloring transformation. In: Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10639–10647 (2019). https://arxiv.org/pdf/1812.09912.pdf

  33. Yu, N., Davis, L., Fritz M.: Attributing fake images to GANs: analyzing fingerprints in generated images. In: Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 1–41 (2019). https://arxiv.org/pdf/1811.08180.pdf

  34. Miyato, T., Kataoka, T., Koyama, M., Yoshida, Y.: Spectral normalization for generative adversarial networks. In: Proceedings of the International Conference on Learning Representations, pp. 1–26 (2018). https://arxiv.org/pdf/1802.05957.pdf

  35. Binkowski, M., Sutherland, D., Arbel, M., Gretton, A.: Demystifying MMD GANs. In: Proceedings of the International Conference on Learning Representations, pp. 1–36 (2018). https://arxiv.org/pdf/1801.01401.pdf

  36. Dolhansky, B., Bitton, J., Pflaum, B., Lu, J., Howes, R., Wang, M., Ferrer, C.C.: The DeepFake detection challenge dataset (2020). https://arxiv.org/pdf/2006.07397.pdf

  37. Li, L., Bao, J., Yang, H., Chen, D., Wen, F.: FaceShifter: towards high fidelity and occlusion aware face swapping (2020). https://arxiv.org/pdf/1912.13457.pdf

  38. Tran, D., Wang, H., Torresani L., Ray J., Le Cun, Y., Paluri, M.: A closer look at spatiotemporal convolutions for action recognition. In: Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6450–6459 (2018). https://arxiv.org/pdf/1711.11248.pdf

  39. Lin, J., Gan, C., Han, S.: TSM: temporal shift module for efficient video understanding. In: Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 7083–7093 (2019). https://arxiv.org/pdf/1811.08383.pdf

  40. Ghadiyaram, D., Mahajan, D.: Large-scale weakly-supervised pretraining for video action recognition. In: Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12038–12047 (2019). https://arxiv.org/pdf/1905.00561.pdf

Download references

Author information

Authors and Affiliations

  1. Financial University Under the Government of the Russian Federation, 38 Shcherbakovskaya, Moscow, 105187, Russia

    Andrey Sebyakin, Vladimir Soloviev & Anatoly Zolotaryuk

Authors
  1. Andrey Sebyakin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Vladimir Soloviev
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Anatoly Zolotaryuk
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Anatoly Zolotaryuk .

Editor information

Editors and Affiliations

  1. Humboldt University of Berlin, Berlin, Germany

    Katharina Toeppe

  2. Renmin University of China, Beijing, China

    Hui Yan

  3. The University of Hong Kong, Hong Kong, Hong Kong

    Samuel Kai Wah Chu

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Sebyakin, A., Soloviev, V., Zolotaryuk, A. (2021). Spatio-Temporal Deepfake Detection with Deep Neural Networks. In: Toeppe, K., Yan, H., Chu, S.K.W. (eds) Diversity, Divergence, Dialogue. iConference 2021. Lecture Notes in Computer Science(), vol 12645. Springer, Cham. https://doi.org/10.1007/978-3-030-71292-1_8

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-71292-1_8

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-71291-4

  • Online ISBN: 978-3-030-71292-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation