Abstract
The widespread adoption of Convolutional Neural Networks (CNNs) in image recognition has undeniably marked a significant breakthrough. However, these networks need a lot of data to learn well, which can be challenging. This can make models prone to overfitting, where they perform well on training data but not on new data. Various strategies have emerged to address this issue, including reasonably selecting an appropriate network architecture. This study delves into mitigating data scarcity by undertaking a comparative analysis of two distinct methods: utilizing compact CNN architectures and applying transfer learning with pre-trained models. Our investigation extends across three disparate datasets, each hailing from a different domain. Remarkably, our findings unveil nuances in performance. The study reveals that using a complex pre-trained model like ResNet50 yields better results for the flower and Maize disease identification datasets, emphasizing the advantages of leveraging prior knowledge for specific data types. Conversely, starting from a simpler CNN architecture trained from scratch is the superior strategy with the Pneumonia dataset, highlighting the need to adapt the approach based on the specific dataset and domain.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data availibility
The authors confirm that the data associated with the experiments can be available upon request.
References
Bai S, He Z, Dong Y, Bai H (2020) Multi-hierarchical independent correlation filters for visual tracking. In: 2020 IEEE international conference on multimedia and expo (ICME), pp 1–6. IEEE
Baldi P, Sadowski PJ (2013) Understanding dropout. Adv Neural Inf Process Syst 26
Barz B, Denzler J (2020) Deep learning on small datasets without pre-training using cosine loss. In: Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp 1371–1380
Bornschein J, Visin F, Osindero S (2020) Small data, big decisions: model selection in the small-data regime. In: International conference on machine learning, pp 1035–1044. PMLR
Brigato L, Iocchi L (2021) A close look at deep learning with small data. In: 2020 25th international conference on pattern recognition (ICPR), pp 2490–2497. IEEE
Bruintjes R-J, Lengyel A, Rios MB, Kayhan OS, Zambrano D, Tomen N, Gemert J (2023) Vipriors 3: visual inductive priors for data-efficient deep learning challenges. arXiv preprint arXiv:2305.19688
Dar JA, Srivastava KK, Lone SA (2022) Design and development of hybrid optimization enabled deep learning model for Covid-19 detection with comparative analysis with dcnn, biat-gru, xgboost. Comput Biol Med 150:106123
Dar JA, Srivastava KK, Mishra A (2023) Lung anomaly detection from respiratory sound database (sound signals). Comput Biol Med 164:107311
Elshamy R, Abu-Elnasr O, Elhoseny M, Elmougy S (2023) Improving the efficiency of RMSProp optimizer by utilizing nestrove in deep learning. Sci Rep 13(1):8814
Geirhos R, Jacobsen J-H, Michaelis C, Zemel R, Brendel W, Bethge M, Wichmann FA (2020) Shortcut learning in deep neural networks. Nat Mach Intell 2(11):665–673
He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778
Hu J, Shen L, Sun G (2018) Squeeze-and-excitation networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 7132–7141
Japkowicz N (2013) Assessment metrics for imbalanced learning. Imbalanc Learn Found, Algorithm, Appl 187–206
Jin Q, Yuan M, Wang H, Wang M, Song Z (2022) Deep active learning models for imbalanced image classification. Knowl-Based Syst 257:109817
Kermany DS, Goldbaum M, Cai W, Valentim CC, Liang H, Baxter SL, McKeown A, Yang G, Wu X, Yan F et al (2018) Identifying medical diagnoses and treatable diseases by image-based deep learning. Cell 172(5):1122–1131
Krizhevsky A, Sutskever I, Hinton GE (2017) Imagenet classification with deep convolutional neural networks. Commun ACM 60(6):84–90
Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. Adv Neural Inf Process Syst 25
Kursun R, Cinar I, Taspinar YS, Koklu M (2022) Flower recognition system with optimized features for deep features. In: 2022 11th Mediterranean conference on embedded computing (MECO), pp 1–4. IEEE
Li E, Wang L, Xie Q, Gao R, Su Z, Li Y (2023) A novel deep learning method for maize disease identification based on small sample-size and complex background datasets. Eco Inform 75:102011
Long J, Shelhamer E, Darrell T (2015) Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3431–3440
Ren S, He K, Girshick R, Sun J (2015) Faster r-cnn: towards real-time object detection with region proposal networks. Adv Neural Inf Process Syst 28
Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556
Szegedy C, Ioffe S, Vanhoucke V, Alemi A (2017) Inception-v4, inception-resnet and the impact of residual connections on learning. In: Proceedings of the AAAI conference on artificial intelligence, vol 31
Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1–9
Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z (2016) Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2818–2826
Targ S, Almeida D, Lyman K (2016) Resnet in resnet: generalizing residual architectures. arXiv preprint arXiv:1603.08029
Zhang Z (2018) Improved adam optimizer for deep neural networks. In: 2018 IEEE/ACM 26th international symposium on quality of service (IWQoS), pp 1–2. IEEE
Funding
The author(s) declare that this research was supported under the Promotion of University Research and Scientific Excellence(PURSE)(SR/PURSE/2022/121) grant from the Department of Science and Technology, Govt of India, New Delhi to the Islamic University of Science and Technology(IUST), Awantipora. The study was also supported under Employment and Skill enhancement Enablement of High Computing and e-learning through IUST Cloud accorded by the Higher Education Department Government of Jammu and Kashmir vide Order No. 77-JK(HE) of 2021 for HEDSS2021100686.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that there are no Conflict of interest regarding the publication of this article.
Ethical approval
This article contains no studies with human participants or animals performed by any of the authors.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Sabha, S.U., Assad, A., Din, N.M.U. et al. From scratch or pretrained? An in-depth analysis of deep learning approaches with limited data. Int J Syst Assur Eng Manag (2024). https://doi.org/10.1007/s13198-024-02345-4
Received:
Revised:
Accepted:
Published:
DOI: https://doi.org/10.1007/s13198-024-02345-4