Skip to main content
SpringerLink
Log in

Ensembling of text and images using Deep Convolutional Neural Networks for Intelligent Information Retrieval

  • Published:
Wireless Personal Communications Aims and scope Submit manuscript

Abstract

Information retrieval (IR) defines the process of searching and attaining specific information resources which are related to the specific information requirements from the available resource pool. It finds useful in several real time application areas namely digital library, healthcare, education, internet browsing, etc. Recently, deep learning (DL) models become popular in different fields of image processing, object detection, and natural language processing. Therefore, in this paper, DL models are employed to retrieve the text and images proficiently. This paper presents an ensemble of DL based IR models for text and images. The proposed model intends to develop DL models individually for text and images. Initially, convolutional neural network based VGGNet-19 model is used as a feature extractor and Euclidian distance based similarity measurement for the retrieval of images. At the same time, bidirectional-long short-term memory (BiLSTM) technique is applied for retrieval of textual documents. The presented BiLSTM model sequentially considers every word in a sentence, extracts the details and embeds it to the semantic vector. In addition to the feature extraction using deep learning techniques, the similarity measurement emphasis the closeness of the document to the given query. The proposed retrieval system has tested on text and images for both general and specific domain (agriculture) with the datasets of Yahoo, Google and Corel10K. With the datasets the performance has been computed by the standard measures such as precision, recall and F-score where the proposed deep learning model produces better results when compared to existing techniques. The proposed model has been tested for the specific domain and achieves the performance of 93% precision and 85% recall and 90% F-score when compared to the existing model.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Le, Q., & Mikolov, T. (2014). Distributed representations of sentences and documents. In Proceedings of the 31st International Conference on Machine Learning (ICML-14) (pp. 1188–1196).

  2. Kiros, R., Zhu, Y., Salakhutdinov, R. R., Zemel, R., Urtasun, R., Torralba, A., & Fidler, S. (2015). Skip-thought vectors. In Advances in neural information processing systems (pp. 3294–3302).

  3. Palangi, H., Deng, L., Shen, Y., Gao, J., He, X., Chen, J., et al. (2016). Deep sentence embedding using long short-term memory networks: Analysis and application to information retrieval. IEEE/ACM Transactions on Audio, Speech and Language Processing (TASLP), 24(4), 694–707.

    Article  Google Scholar 

  4. Krizhevsky, A., Sutskever, I., Hinton, G.E., Imagenet classification with deep convolutional neural networks, In Advances in neural information processing systems, 2012, pp. 1097–1105.

  5. Zhou, D., Li, X., Zhang, Y.J., (2016). A novel cnn-based match kernel for image retrieval. In IEEE International Conference on Image Processing

  6. Gordo, A., Almazán, J., Revaud, J., Larlus, D., (2016). Deep image retrieval: Learning global representations for image search, In Computer Vision—ECCV 2016—14th European Conference, Amsterdam, The Netherlands, October 11–14, Proceedings, Part VI, 2016, pp. 241–257.

  7. Fu, R., Li, B., Gao, Y., Ping, W., (2017). Content-based image retrieval based on cnn and svm. In IEEE International Conference on Computer & Communications, 2017.

  8. Sun, P.X. Lin, H.T., Tao, L., (2016). Learning discriminative cnn features and similarity metrics for image retrieval. In IEEE International Conference on Signal Processing

  9. Shimoda, K. Yanai, Learning food image similarity for food image retrieval, in: IEEE Third International Conference on Multimedia Big Data, 2017.

  10. Liu, P. Guo, J.M., Wu, C.Y., Cai, D., (2017). Fusion of deep learning and compressed domain features for content based image retrieval, IEEE Transactions on Image Processing A Publication of the IEEE Signal Processing Society, PP (99) 1–1

  11. Li, Z., & Tang, J. (2015). Weakly supervised deep metric learning for community-contributed image retrieval. IEEE Transactions on Multimedia, 17(11), 1989–1999.

    Article  Google Scholar 

  12. Wang, X., Xiong, D., & Xiang, B. (2016). Deep sketch feature for cross-domain image retrieval. Neurocomputing, 207, S0925231216303198.

    Article  Google Scholar 

  13. Chung, Y.-A., Weng, W.-H., (2017). Learning Deep Representations of Medical Images using Siamese CNNs with Application to Content-Based Image Retrieval, (2017). Sourced from Microsoft Academic—https://academic.microsoft.com/ paper/2768570904.

  14. Qayyum, A., Anwar, S.M., Awais, M., Majid, M., (2017). Medical image retrieval using deep convolutional neural network. Neurocomputing. S0925231217308445

  15. Do, T., Hoang, T., Tan, D.L., Pham, T., Le, H., Cheung, N., Reid, I.D., (2019) Binary constrained deep hashing network for image retrieval without manual annotation, In IEEE Winter Conference on Applications of Computer Vision, WACV 2019, Waikoloa Village, HI, USA, January 7–11, 2019, 2019, pp. 695–704.

  16. Palangi, H., Deng, L., Shen, Y., Gao, J., He, X., Chen, J., et al. (2016). Deep sentence embedding using long short-term memory networks: Analysis and application to information retrieval. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 24(4), 694–707.

    Article  Google Scholar 

  17. Bala, A., & Kaur, T. (2016). Local texton XOR patterns: A new feature descriptor for content-based image retrieval. Engineering Science and Technology, an International Journal, 19(1), 101–112.

    Article  Google Scholar 

  18. Siami-Namini, S., Tavakoli, N. and Namin, A.S., 2019, December. The performance of LSTM and BiLSTM in forecasting time series. In 2019 IEEE International Conference on Big Data (Big Data) (pp. 3285–3292). IEEE.

  19. Khalifi, H., Elqadi, A., & Ghanou, Y. (2018). Support vector machines for a new hybrid information retrieval system. Procedia Computer Science, 127, 139–145.

    Article  Google Scholar 

  20. Chu, K., & Liu, G.H. (2020). Image Retrieval Based on a multi-integration features model. Mathematical Problems in Engineering, 2020(2020), 1–10.

  21. Alsmadi, M. K. (2018). Query-sensitive similarity measure for content-based image retrieval using meta-heuristic algorithm. Journal of King Saud University-Computer and Information Sciences, 30(3), 373–381.

    Article  Google Scholar 

  22. Alsmadi, M. K. (2017). An efficient similarity measure for content based image retrieval using memetic algorithm. Egyptian Journal of Basic and Applied Sciences, 4(2), 112–122.

    Article  Google Scholar 

  23. Madhavi, K. V., Tamilkodi, R., & Sudha, K. J. (2016). An innovative method for retrieving relevant images by getting the top-ranked images first using interactive genetic algorithm. Procedia Computer Science, 79, 254–261.

    Article  Google Scholar 

  24. Jhanwar, N., Chaudhuri, S., Seetharaman, G., & Zavidovique, B. (2004). Content based image retrieval using motif cooccurrence matrix. Image and Vision Computing, 22(14), 1211–1220.

    Article  Google Scholar 

  25. ElAlami, M. E. (2011). A novel image retrieval model based on the most relevant features. Knowledge-Based Systems, 24(1), 23–32.

    Article  Google Scholar 

  26. Pavithra, L. K., & Sharmila, T. S. (2018). An efficient framework for image retrieval using color, texture and edge features. Computers & Electrical Engineering, 70, 580–593.

    Article  Google Scholar 

  27. Yuan, B.H., & Liu, G.H. (2020). Image retrieval based on gradient-structures histogram. Neural Computing and Applications, 1–11.

  28. Sadeghi-Tehran, P., Angelov, P., Virlet, N., & Hawkesford, M. J. (2019). Scalable database indexing and fast image retrieval based on deep learning and hierarchically nested structure applied to remote sensing and plant biology. Journal of Imaging, 5(33), 1–5.

    Google Scholar 

  29. Sezavar, A., Farsi, H., & Mohamadzadeh, S. (2019). Content-based image retrieval by combining convolutional neural networks and sparse representation. Multimedia Tools and Applications, 78(6), 1–18.

    Google Scholar 

  30. Kanwal, K., Ahmad, K. T., Khan, R., Abbasi, A. T., & Li, J. (2020). Deep Learning Using Symmetry, FAST Scores, Shape-Based Filtering and Spatial Mapping Integrated with CNN for Large Scale Image Retrieval. Symmetry, 12, 612.

    Article  Google Scholar 

  31. Pang, L., Lan, Y., Guo, J., Xu, J., Xu, J., Cheng, X. (2019). DeepRank: A New Deep Architecture for Relevance Ranking in Information Retrieval. In Proceedings of 26thACM Conference on Information and Knowledge Management, Singapore.

  32. Hu, B., Lu, Z., Li, H., and Chen, Q., (2014). Convolutional neural network architectures for matching natural language sentences. In NIPS. 2042–2050.

  33. Pang, L., Lan, Y., Guo, J., Xu, J., and Cheng, X., (2016). A study of matchpyramid models on ad-hoc retrieval. In Neu-IR16 SIGIR Workshop on Neural Information Retrieval.

  34. Pang, L., Lan, Y., Guo, J., Xu, J., Wan, S., and Cheng, X., 2016. Text matching as image recognition. In AAAI. AAAI Press, 2793–2799.

  35. Wan, S., Lan, Y., Guo, J., Xu, J., Pang, L., Cheng, X. (2016). Match-SRNN: Modeling the Recursive Matching Structure with Spatial RNN. In IJCAI. pp. 2922–2928

  36. Qin, X., Zhang, H., Zheng, H., (2019). Research on Intelligent Retrieval System for agricultural information resources based on ontology. In IOP Conference Series. Journal of Physics

  37. Dang, V., Bendersky, M., Croft, W. B., (2013). Two-Stage Learning to Rank for Information Retrieval, Lecture Notes in Computer Science book series, pp. 423–434.

  38. Prabhu, L. A. J., Sengan, S., Kamalam, G. K., Vellingiri, J., Gopal, J., Velayutham, P., Subramaniyaswamy, V., (2020). Medical Information Retrieval Systems for e-Health Care Records using Fuzzy Based Machine Learning Model, Microprocessors and Microsystems.

  39. Ramkumar, J., Baskar, M., Nipun, P., Aithagani, A., (2020). Effective Framework to Monitor Patient Health Care through Intelligent System, International Journal of Advanced Science and Technology, 29(4), 1828–1835, ISSN: 2005–4238, April 2020.

  40. Ramkumar, J., Baskar, M., Kondru, S., Kuchipudi, J., (2020). Wearable Biometric authentication for health monitoring system using RedTacton, International Journal of Advanced Science and Technology, 29(4), 1819–1827, ISSN: 2005–4238, April 2020.

  41. Arulananth, T. S., Balaji, L., Baskar, M., et al. (2020). PCA based dimensional data reduction and segmentation for DICOM images. Neural Processing Letters. https://doi.org/10.1007/s11063-020-10391-9.

    Article  Google Scholar 

  42. Baskar. M, Gnansekaran. T., (2017). Developing Efficient Intrusion Tracking System using Region Based Traffic Impact Measure Towards the Denial of Service Attack Mitigation, Journal of Computational and Theoretical Nanoscience, 14(7), 3576–3582, ISSN: 1546–1955 (Print): EISSN: 1546–1963 (Online) , July 2017.

  43. Suchithra, M., Baskar, M., Ramkumar, J., Kalyanasundaram, P., & Amutha, B. (2020). Invariant packet feature with network conditions for efficient low rate attack detection in multimedia networks for improved QoS. Journal of Ambient Intell Human Computation. https://doi.org/10.1007/s12652-020-02056-1.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, B.S.Abdur Rahman Crescent Institute of Science & Technology, Chennai, India

    P. Mahalakshmi & N. Sabiyath Fatima

  2. Department of Computer Science and Engineering, School of Computing, College of Engineering and Technology, SRM Institute of Science and Technology, Kattankulathur, India

    P. Mahalakshmi

Authors
  1. P. Mahalakshmi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. N. Sabiyath Fatima
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to P. Mahalakshmi.

Ethics declarations

Conflicts of interest

We authors not having any conflict of interest among ourselves to submit and publish our articles in Wireless Personal Communications journal.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mahalakshmi, P., Fatima, N.S. Ensembling of text and images using Deep Convolutional Neural Networks for Intelligent Information Retrieval. Wireless Pers Commun 127, 235–253 (2022). https://doi.org/10.1007/s11277-021-08211-x

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11277-021-08211-x

Keywords

Search

Navigation