Skip to main content
SpringerLink
Log in

LW-DeepFakeNet: a lightweight time distributed CNN-LSTM network for real-time DeepFake video detection

  • Original Paper
  • Published:
Signal, Image and Video Processing Aims and scope Submit manuscript

Abstract

With the large-scale and pervading social media platforms and the recent advances in generative deep learning techniques, it is nowadays quite common to forge highly-realistic and credible misleading videos known as DeepFakes. These videos mean to alter the original intention behind the video to put forth their hidden ploys. In this work, a simple yet effective lightweight time distributed (LW-DeepFakeNet) model that uses both spatial and temporal information to determine whether the video has been altered is proposed. The model utilizes a transfer learning approach with pre-trained convolutional networks for spatial feature extraction, topped up with LSTMs for temporal information extraction, requiring little training data and time. This research also considers a special use case of DeepFake where a particular video sequence has a scene change and proposes a way to counter the class-imbalance present in the dataset. The resulting model is much lighter with up to 152x times reduction in parameter count while achieving a significant accuracy of 99.24% at a remarkable rate of 80 fps.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Data availability

All data will be available from the corresponding author on reasonable request.

References

  1. Juefei-Xu, F., Wang, R., Huang, Y., Guo, Q., Ma, L., Liu, Y.: Countering malicious deepfakes: survey, battleground, and horizon (2021). https://doi.org/10.48550/ARXIV.2103.00218. arxiv:https://arxiv.org/abs/2103.00218

  2. Zhang, Y., Zheng, L., Thing, V.L.L.: Automated face swapping and its detection. In: 2017 IEEE 2nd International Conference on Signal and Image Processing (ICSIP), pp. 15–19 (2017). https://doi.org/10.1109/SIPROCESS.2017.8124497

  3. Zhou, P., Han, X., Morariu, V.I., Davis, L.S.: Two-stream neural networks for tampered face detection (2018). https://doi.org/10.48550/ARXIV.1803.11276 . arXiv:1803.11276

  4. Marra, F., Gragnaniello, D., Cozzolino, D., Verdoliva, L.: Detection of GAN-generated fake images over social networks. In: IEEE Conference on Multimedia Information Processing and Retrieval (MIPR), vol. 2018, pp. 384–389 (2018). https://doi.org/10.1109/MIPR.2018.00084

  5. Li, Y., Chang, M.-C., Lyu, S.: In Ictu Oculi: exposing ai created fake videos by detecting eye blinking. In: IEEE International Workshop on Information Forensics and Security (WIFS), vol. 2018, pp. 1–7 (2018). https://doi.org/10.1109/WIFS.2018.8630787

  6. Mo, H., Chen, B., Luo, W.: Fake faces identification via convolutional neural network. In: IH &MMSec ’18, Association for Computing Machinery, New York, NY, USA, pp. 43–47 (2018). https://doi.org/10.1145/3206004.3206009

  7. Koopman, M., Macarulla Rodriguez, A., Geradts, Z.: Detection of deepfake video manipulation (2018)

  8. Güera, D., Delp, E.J.: Deepfake video detection using recurrent neural networks. In: 2018 15th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), pp. 1–6 (2018). https://doi.org/10.1109/AVSS.2018.8639163

  9. Matern, F., Riess, C., Stamminger, M.: Exploiting visual artifacts to expose deepfakes and face manipulations. In: IEEE Winter Applications of Computer Vision Workshops (WACVW), vol. 2019, pp. 83–92 (2019). https://doi.org/10.1109/WACVW.2019.00020

  10. Xuan, X., Peng, B., Wang, W., Dong, J.: On the generalization of GAN image forensics (2019). https://doi.org/10.48550/ARXIV.1902.11153. arXiv:1902.11153

  11. Tarasiou, M., Zafeiriou, S.: Extracting deep local features to detect manipulated images of human faces (2019). https://doi.org/10.48550/ARXIV.1911.13269. arXiv:1911.13269

  12. Li, Y., Yang, X., Sun, P., Qi, H., Lyu, S.: Celeb-df: A large-scale challenging dataset for deepfake forensics (2019). https://doi.org/10.48550/ARXIV.1909.12962. arXiv:1909.12962

  13. Kumar, P., Vatsa, M., Singh, R.: Detecting face2face facial reenactment in videos (2020). https://doi.org/10.48550/ARXIV.2001.07444. arXiv:2001.07444

  14. Li, L., Bao, J., Zhang, T., Yang, H., Chen, D., Wen, F., Guo, B.: Face x-ray for more general face forgery detection (2019). https://doi.org/10.48550/ARXIV.1912.13458. arXiv:1912.13458

  15. Sun, K., Xiao, B., Liu, D., Wang, J.: Deep high-resolution representation learning for human pose estimation (2019). https://doi.org/10.48550/ARXIV.1902.09212. arXiv:1902.09212

  16. Sun, K., Zhao, Y., Jiang, B., Cheng, T., Xiao, B., Liu, D., Mu, Y., Wang, X., Liu, W., Wang, J.: High-resolution representations for labeling pixels and regions (2019). https://doi.org/10.48550/ARXIV.1904.04514. arXiv:1904.04514

  17. Lima, O. de., Franklin, S., Basu, S., Karwoski, B., George, A.: Deepfake detection using spatiotemporal convolutional networks (2020). https://doi.org/10.48550/ARXIV.2006.14749. arXiv:2006.14749

  18. Hernandez-Ortega, J., Tolosana, R., Fierrez, J., Morales, A.: Deepfakeson-phys: Deepfakes detection based on heart rate estimation (2020). https://doi.org/10.48550/ARXIV.2010.00400. arXiv:2010.00400

  19. Schwarcz, S., Chellappa, R.: Finding facial forgery artifacts with parts-based detectors (2021). https://doi.org/10.48550/ARXIV.2109.10688. arXiv:2109.10688

  20. Saif, S., Tehseen, S., Ali, S.S., Kausar, S., Jameel, A.: Generalized deepfake video detection through time-distribution and metric learning. IT Prof. 24(2), 38–44 (2022). https://doi.org/10.1109/MITP.2022.3168351

    Article  Google Scholar 

  21. Saikia, P., Dholaria, D., Yadav, P., Patel, V., Roy, M.: A hybrid CNN-LSTM model for video deepfake detection by leveraging optical flow features (2022)

  22. Ke, J., Wang, L.: Df-udetector: an effective method towards robust deepfake detection via feature restoration. Neural Netw. 160, 216–226 (2023). https://doi.org/10.1016/j.neunet.2023.01.001

  23. Dolhansky, B., Bitton, J., Pflaum, B., Lu, J., Howes, R., Wang, M., Ferrer, C.C.: The deepfake detection challenge (DFDC) dataset (2020). https://doi.org/10.48550/ARXIV.2006.07397. arXiv:2006.07397

  24. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition (2014). https://doi.org/10.48550/ARXIV.1409.1556. arXiv:1409.1556

  25. Vamsi, V.V.V.N.S., Shet, S.S., Reddy, S.S.M., Rose, S.S., Shetty, S.R., Sathvika, S., Shankar, S.M.S., S.P.: Deepfake detection in digital media forensics. Glob. Transit. Proc. 3(1), 74–79 (2022). International Conference on Intelligent Engineering Approach (ICIEA-2022). https://doi.org/10.1016/j.gltp.2022.04.017

  26. Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: IEEE Conference on Computer Vision and Pattern Recognition, vol. 2009, pp. 248–255 (2009). https://doi.org/10.1109/CVPR.2009.5206848

Download references

Acknowledgements

This work was supported by the EIAS Data Science Lab, College of Computer and Information Sciences, Prince Sultan University, Riyadh, Saudi Arabia.

Funding

No funding was obtained for this study.

Author information

Authors and Affiliations

  1. Department of Electronics and Communications, Jamia Millia Islamia, New Delhi, India

    Umar Masud

  2. Department of Computer Engineering, Jamia Millia Islamia, New Delhi, 110025, India

    Mohd. Sadiq, Sarfaraz Masood & Musheer Ahmad

  3. EIAS Data Science Lab, College of Computer and Information Sciences, Prince Sultan University, 11586, Riyadh, Saudi Arabia

    Ahmed A. Abd El-Latif

  4. Department of Mathematics and Computer Science, Faculty of Science, Menoufia University, Shebin El-Koom, 32511, Egypt

    Ahmed A. Abd El-Latif

Authors
  1. Umar Masud
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mohd. Sadiq
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sarfaraz Masood
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Musheer Ahmad
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Ahmed A. Abd El-Latif
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Conceptualization, UM, MS, SM, MA, and AAAE-L; methodology, UM, MS, SM; software, UM, and MS; validation, MA, and AAAE-L; formal analysis, UM, and MA; writing-original draft preparation, UM, MS, and SM; writing-review and editing, MA, and AAAE-L; All authors have read and agreed to the published version of the manuscript.

Corresponding author

Correspondence to Ahmed A. Abd El-Latif.

Ethics declarations

Conflict of interest

The authors declare that there are no known financial or conflicts of interest associated with this article.

Ethical approval

Ethical approval is not applicable for this article.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Masud, U., Sadiq, M., Masood, S. et al. LW-DeepFakeNet: a lightweight time distributed CNN-LSTM network for real-time DeepFake video detection. SIViP 17, 4029–4037 (2023). https://doi.org/10.1007/s11760-023-02633-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11760-023-02633-9

Keywords

Search

Navigation