Skip to main content
SpringerLink
Log in

A deep multimodal system for provenance filtering with universal forgery detection and localization

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Traditional multimedia forensics techniques inspect images to identify, localize forged regions and estimate forgery methods that have been applied. Provenance filtering is the research area that has been evolved recently to retrieve all the images that are involved in constructing a morphed image in order to analyze an image, completely forensically. This task can be performed in two stages: one is to detect and localize forgery in the query image, and the second integral part is to search potentially similar images from a large pool of images. We propose a multimodal system which covers both steps, forgery detection through deep neural networks(CNN) followed by part based image retrieval. Classification and localization of manipulated region are performed using a deep neural network. InceptionV3 is employed to extract key features of the entire image as well as for the manipulated region. Potential donors and nearly duplicates are retrieved by using the Nearest Neighbour Algorithm. We take the CASIA-v2, CoMoFoD and NIST 2018 datasets to evaluate the proposed system. Experimental results show that deep features outperform low-level features previously used to perform provenance filtering with achieved Recall@50 of 92.8%.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  1. Murabayashi A (2018) The problem of fake photos in fake news. https://petapixel.com/2017/01/19/problem-fake-photos-fake-news/, Accessed: 2019-01-11

  2. Fridrich AJ, Soukal BD, Lukáš AJ (2003) Detection of copy-move forgery in digital images. In: Proceedings of Digital Forensic Research Workshop, Citeseer

  3. Lloret J, Bosch I, Sendra S, Serrano A (2011) A wireless sensor network for vineyard monitoring that uses image processing. Sensors 11(6):6165–6196

    Article  Google Scholar 

  4. Birajdar GK, Mankar VH (2013) Digital image forgery detection using passive techniques: A survey. Elsevier Digital investigation 10(3):226–245

    Article  Google Scholar 

  5. Farid H (2009) Image forgery detection. IEEE Signal Process. Mag. 26(2):16–25

    Article  Google Scholar 

  6. Ahmed N, Natarajan T, Rao KR (1974) Discrete cosine transform. IEEE transactions on Computers 100(1):90–93

    Article  MathSciNet  Google Scholar 

  7. Lowe DG (2004) Distinctive image features from scale-invariant keypoints. International journal of computer vision 60(2):91–110

    Article  Google Scholar 

  8. Bay H, Tuytelaars T, VanGool L (2006) Surf: Speeded up robust features. In: European conference on computer vision, Springer, pp 404–417

  9. Amerini I, Ballan L, Caldelli R, DelBimbo A, Serra G (2011) A sift-based forensic method for copy–move attack detection and transformation recovery. IEEE Trans. Informat. Forensics and Security 6(3):1099–1110

    Article  Google Scholar 

  10. Shivakumar BL, Baboo SS (2011) Detection of region duplication forgery in digital images using surf. International Journal of Computer Science Issues (IJCSI) 8(4):199

    Google Scholar 

  11. Bhatti MH, Khan J, Khan M UG, Iqbal R, Aloqaily M, Jararweh Y, Gupta B (2019) Soft computing-based eeg classification by optimal feature selection and neural networks. IEEE Transactions on Industrial Informatics 15 (10):5747–5754

    Article  Google Scholar 

  12. Khan MZ, Harous S, Hassan SU, Khan M UG, Iqbal R, Mumtaz S (2019) Deep unified model for face recognition based on convolution neural network and edge computing. IEEE Access 7:72622–72633

    Article  Google Scholar 

  13. Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, Huang Z, Karpathy A, Khosla A, Bernstein M, Berg AC, Fei-Fei L (2015) ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV) 115(3):211–252. https://doi.org/10.1007/s11263-015-0816-y

    Article  MathSciNet  Google Scholar 

  14. Simonyan K, Zisserman A (2014Sept.) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556

  15. Rao Y, Ni J (2016) A deep learning approach to detection of splicing and copy-move forgeries in images. In: 2016 IEEE International Workshop on Information Forensics and Security (WIFS), IEEE, pp 1–6

  16. Bayar B, Stamm MC (2016) A deep learning approach to universal image manipulation detection using a new convolutional layer. In: Proceedings of the 4th ACM Workshop on Information Hiding and Multimedia Security, ACM, pp 5–10

  17. Chen J, Kang X, Liu Y, Wang ZJ (2015) Median filtering forensics based on convolutional neural networks. IEEE Signal Processing Letters 22(11):1849–1853

    Article  Google Scholar 

  18. Wu Y, Abd-Almageed W, Natarajan P (2018September) Busternet: Detecting copy-move image forgery with source/target localization. In: Proceedings of the European Conference on Computer Vision (ECCV), pp 168–184

  19. Pinto A, Moreira D, Bharati A, Brogan J, Bowyer K, Flynn P, Scheirer W, Rocha A (2017) Provenance filtering for multimedia phylogeny. In: 2017 IEEE International Conference on Image Processing (ICIP), IEEE, pp 1502–1506

  20. Asghar K, Habib Z, Hussain M (2017) Copy-move and splicing image forgery detection and localization techniques: a review. Taylor & Francis Australian Journal of Forensic Sciences 49(3):281–307

    Article  Google Scholar 

  21. Steinbach M, Karypis G, Kumar V, et al. (2000) A comparison of document clustering techniques. In: KDD workshop on text mining, vol 400, Boston, pp 525–526

  22. Bo X, Junwen W, Guangjie L, Yuewei D (2010) Image copy-move forgery detection based on surf. In: Proc. IEEE Int. Conf. on Multimedia Information Netw. and Security, pp 889–892

  23. Qu Z, Qiu G, Huang J (2009) Detect digital image splicing with visual cues. In: Springer Int. workshop on Information hiding, pp 247–261

  24. Lin SD, Wu T (2011) An integrated technique for splicing and copy-move forgery image detection. In: Proc. IEEE 4th International Congress on Image and Signal Processing, vol 2, pp 1086–1090

  25. Lin Z, He J, Tang X, Tang C-K (2009) Fast, automatic and fine-grained tampered jpeg image detection via dct coefficient analysis. Pattern Recogn 42(11):2492–2501

    Article  Google Scholar 

  26. Cao Y, Gao T, Fan L, Yang Q (2012) A robust detection algorithm for copy-move forgery in digital images. Forensic science international 214 (1-3):33–43

    Article  Google Scholar 

  27. Mahmood T, Mehmood Z, Shah M, Saba T (2018) A robust technique for copy-move forgery detection and localization in digital images via stationary wavelet and discrete cosine transform. J Vis Commun Image Represent 53:202–214

    Article  Google Scholar 

  28. Muhammad G, Al-Hammadi MH, Hussain M, Bebis G (2014) Image forgery detection using steerable pyramid transform and local binary pattern. Mach Vis Appl 25(4):985–995

    Article  Google Scholar 

  29. Yosinski J, Clune J, Bengio Y, Lipson H (2014) How transferable are features in deep neural networks?. In: Prof. Advances in neural information processing systems, pp 3320–3328

  30. Wang P, Wei Z, Xiao L (2016) Pure spatial rich model features for digital image steganalysis. Multimedia Tools and Applications 75(5):2897–2912

    Article  Google Scholar 

  31. Kang X, Stamm MC, Peng A, Liu KJRay (2013) Robust median filtering forensics using an autoregressive model. IEEE Transactions on Information Forensics and Security 8(9):1456–1468

    Article  Google Scholar 

  32. Yuan H-D (2011) Blind forensics of median filtering in digital images. IEEE Transactions on Information Forensics and Security 6(4):1335–1345

    Article  Google Scholar 

  33. Cao G, Zhao Y, Ni R, Yu L, Tian H (2010) Forensic detection of median filtering in digital images. In: 2010 IEEE International Conference on Multimedia and Expo, IEEE, pp 89–94

  34. Bunk J, Bappy JH, Mohammed TM, Nataraj L, Flenner A, Manjunath BS, Chandrasekaran S, Roy-Chowdhury AK, Peterson L (2017) Detection and localization of image forgeries using resampling features and deep learning. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), IEEE, pp 1881–1889

  35. Helgason S, Helgason S (1999) The radon transform. Springer, vol 2

  36. Chum O, Philbin J, Zisserman A, et al. (2008September) Near duplicate image detection: min-hash and tf-idf weighting.. In: BMVC, vol 810, pp 812–815

  37. Li M, Wang B, Ma W-Y, Li Z (2010) Detecting duplicate images using hash code grouping, Google Patents. US Patent 7,647,331

  38. Dong W, Wang Z, Charikar M, Li K (2012) High-confidence near-duplicate image detection. In: Proceedings of the 2nd acm international conference on multimedia retrieval, ACM, p 1

  39. Agarwal R, Verma OP (2019) An efficient copy move forgery detection using deep learning feature extraction and matching algorithm. Multimedia Tools and Applications 79:7355–7376

    Article  Google Scholar 

  40. Jaiswal AK, Srivastava R (2019) Image splicing detection using deep residual network. Available at SSRN 3351072

  41. Ahmed B, Gulliver TA, et al. (2020) Image splicing detection using mask-rcnn. SIViP, pp 1–8

  42. Dias Z, Goldenstein S, Rocha A (2013) Toward image phylogeny forests: Automatically recovering semantically similar image relationships. Forensic science international 231(1-3):178–189

    Article  Google Scholar 

  43. (2016) Nist media forensics challenge. https://www.nist.gov/itl/iad/mig/media-forensics-challenge, Accessed: 2018-07-26

  44. Medibrowser. http://medifor.rankone.io/

  45. Moreira D, Bharati A, Brogan J, Pinto A, Parowski M, Bowyer KW, Flynn PJ, Rocha A, Scheirer WJ (2018) Image provenance analysis at scale. IEEE Trans Image Process 27(12):6109–6123

    Article  MathSciNet  Google Scholar 

  46. Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z (2016) Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2818–2826

  47. Wojna Z, Ferrari V, Guadarrama S, Silberman N, Chen L-C, Fathi A, Uijlings J (2017July) The devil is in the decoder. arXiv preprint arXiv:1707.05847

  48. Dong J, Wang W, Tan T (2013) Casia image tampering detection evaluation database. In: 2013 IEEE China Summit and International Conference on Signal and Information Processing, IEEE, pp 422–426

  49. Tralic D, Zupancic I, Grgic S, Grgic M (2013) Comofodnew database for copy-move forgery detection. In: Proceedings ELMAR-2013, IEEE, pp 49–54

  50. (2018) Nist media forensics challenge. https://www.nist.gov/itl/iad/mig/media-forensics-challenge-2018, Accessed: 2019-06-11

  51. Christlein V, Riess C, Jordan J, Riess C, Angelopoulou E (2012) An evaluation of popular copy-move forgery detection approaches. IEEE Transactions on information forensics and security 7(6):1841–1854

    Article  Google Scholar 

  52. Cozzolino D, Poggi G, Verdoliva L (2015) Efficient dense-field copy–move forgery detection. IEEE Transactions on Information Forensics and Security 10 (11):2284–2297

    Article  Google Scholar 

  53. Wu Y, Abd-Almageed W, Natarajan P (2017) Deep matching and validation network: An end-to-end solution to constrained image splicing localization and detection. In: Proceedings of the 25th ACM international conference on Multimedia, ACM, pp 1480–1502

  54. Garcia M, Canovas A, Edo M, Lloret J (2008) A QoE management system for ubiquitous IPTV devices. In: The Third International Conference on Mobile Ubiquitous Computing, Systems, Services and Technologies (UBICOMM)

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, University of Engineering and Technology, Lahore, Pakistan

    Saira Jabeen & Usman Ghani Khan

  2. Al-Khawarizi Institute of Computer Science, University of Engineering and Technology, Lahore, Pakistan

    Razi Iqbal

  3. Guangdong Provincial Key Laboratory of Petrochemical Equipment Fault Diagnosis, Guangdong University of Petrochemical Technology, Maoming, 525000, China

    Mithun Mukherjee

  4. Universitat Politecnica de Valencia, 46022, Valencia, Spain

    Jaime Lloret

  5. School of Computing and Digital Technologies, Staffordshire University, Stoke, UK

    Jaime Lloret

Authors
  1. Saira Jabeen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Usman Ghani Khan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Razi Iqbal
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Mithun Mukherjee
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jaime Lloret
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Saira Jabeen.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jabeen, S., Khan, U.G., Iqbal, R. et al. A deep multimodal system for provenance filtering with universal forgery detection and localization. Multimed Tools Appl 80, 17025–17044 (2021). https://doi.org/10.1007/s11042-020-09623-w

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-020-09623-w

Keywords

Search

Navigation