Skip to main content
SpringerLink
Log in

Enhancing Convolutional Neural Network Performance Using Domain Transforms in Constrained Networks

  • Published:
Circuits, Systems, and Signal Processing Aims and scope Submit manuscript

Abstract

Convolutional neural network (CNN) has emerged as one of the primary tools for image classification. In particular, deep CNNs are now considered the main tool for this purpose. However, these networks are often large and require computing and storage power that may not be available in very small sensor devices such as IoT (Internet of Things) devices; their training is also time and computing power consuming. As a result, in some applications, reducing the size of input data (images) and the processing network becomes necessary. Such reduction usually comes at the cost of reduced classification performance. In this paper, we consider networks with under 200k learnable parameters, as opposed to millions in deeper networks. We examine how domain transforms can be used for efficient size reduction and improvement of classification accuracy for small networks. We emphasize that finding optimal hyperparameters or network configurations is not our objective in this paper. It is shown that by using transforms such as discrete wavelet transforms (DWT) or discrete cosine transform (DCT), it is possible to efficiently improve the performance of size-reduced networks and inputs. We demonstrate that in most cases, the improvement can be traced to higher entropy of resized input using transforms. While transforms such as DCT allow variable input and network sizes to be utilized, DWT proves to be very effective when significant size reduction is needed (improving the result by up to 5%). It is also shown that input size reduction of up to 75% is possible, without loss of classification accuracy in some cases. We use two datasets of small images, including Fashion MNIST and CIFAR-10, to evaluate the performance of size reduction methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Data Availability

The datasets generated during the current study are available from the corresponding author on reasonable request. The public datasets analyzed during the current study are publicly available from sources in References [13, 23, 39].

Code Availability

MATLAB neural network toolbox is used in this research, which is commercially available.

References

  1. R. Asadi, S.A. Kareem, Review of Feed Forward Neural Network classification preprocessing techniques. In Proceedings of the 3rd Int. Conf. on Math. Sciences, AIP Conf. Proc., vol. 1602 (2014), p. 567

  2. Al. Aussem, F. Murtagh, combining neural network forecasts on wavelet-transformed time series. Connect. Sci. 9, 113–122 (1997)

    Article  Google Scholar 

  3. A. Ciurana, A. Mosella-Montoro, J. Ruiz-Hidalgo, Hybrid cosine based convolutional neural networks (2019). https://arxiv.org/pdf/1904.01987.pdf

  4. A. Coates, H. Lee, A. Ng, An analysis of single-layer networks in unsupervised feature learning. In NIPS Workshop on Deep Learning (2010)

  5. A. Cohen, I. Daubechies, J.-C. Feauveau, Biorthogonal bases of compactly supported wavelets. Commun. Pure Appl. Math. 45(5), 485–560 (1992). https://doi.org/10.1002/cpa.3160450502

    Article  MathSciNet  MATH  Google Scholar 

  6. I. Daubechies, Ten lectures on wavelets. SIAM (1992). https://doi.org/10.1137/1.9781611970104

    Article  MATH  Google Scholar 

  7. D. Fu, G. Guimaraes, Using compression to speed up image classification in artificial neural networks “the discrete cosine transforms” (2016), pp. 1–10

  8. A. Ghosh, R. Chellappa, Deep feature extraction in the DCT domain. In 23rd International Conference on Pattern Recognition (ICPR), pp. 3525–3530 (2016)

  9. B. Graham, :Fractional max-pooling. In arxiv:cs/arXiv:1412.6071, 2015.

  10. M. Hashemi, Enlarging smaller images before inputting into the convolutional neural network: zero-padding vs interpolation. J. Big Data 6, 98 (2019). https://doi.org/10.1186/s40537-019-0263-7

    Article  Google Scholar 

  11. K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition (2015). arXiv:1512.03385

  12. L. Hertel, E. Barth, T. Käster, T. Martinetz, Deep convolutional neural networks as generic feature extractors. In IJCNN (2015)

  13. https://github.com/zalandoresearch/fashion-mnist

  14. https://www.cs.toronto.edu/~kriz/cifar.html

  15. https://www.mathworks.com/solutions/deep-learning/examples/training-a-model-from-scratch.html

  16. D. Huang, X.-R. Bai, A wavelet neural network optimal control model for traffic-flow prediction in intelligent transport systems. Adv. intelligent computing theories and apps. with aspects of A.I.. ICIC 2007. In LNCS, vol. 4682 (2007), pp. 1233–1244

  17. X. Jiang, H. Adeli, Dynamic wavelet neural network model for traffic flow forecasting. J. Transport. Eng. 131(10), 771–779 (2005)

    Article  Google Scholar 

  18. M. Kayed, A. Anter, H. Mohamed,: Classification of garments from fashion MNIST dataset using CNN LeNet-5 architecture. In Int. Conf. on Innovative Trends in Communication and Computer Engineering (ITCE) (2020), pp. 238–243. https://doi.org/10.1109/ITCE48509.2020.9047776

  19. M.K. Khandani, W.B. Mikhael, Using mixed DCT and Haar transforms for efficient compression of car trajectory data. In 2018 IEEE 61st International Midwest Symposium on Circuits and Systems (MWSCAS) (2018), pp. 692–695. https://doi.org/10.1109/MWSCAS.2018.8623987

  20. M.K. Khandani, W.B. Mikhael, Efficient time series forecasting using time delay neural networks with domain pre-transforms. In 2019 IEEE 62nd International Midwest Symposium on Circuits and Systems (MWSCAS) (2019), pp. 682–685. https://doi.org/10.1109/MWSCAS.2019.8884826

  21. M.K. Khandani, W.B. Mikhael, A study on network size reduction using sparse input representation in time delay neural networks. In 2020 IEEE 63rd International Midwest Symposium on Circuits and Systems (MWSCAS) (2020), pp. 864–867. https://doi.org/10.1109/MWSCAS48704.2020.9184438

  22. M.K. Khandani, W.B. Mikhael, Effect of sparse representation of time series data on learning rate of time delay neural networks. Circuits Syst. Signal Process. (2021). https://doi.org/10.1007/s00034-020-01610-8

    Article  Google Scholar 

  23. A. Krizhevsky. Learning multiple layers of features from tiny images. Technical report (2009)

  24. A. Krizhevsky, I. Sutskever, G.E. Hinton, ImageNet classification with deep convolutional neural networks. Commun. ACM 60(6), 84–90 (2017). https://doi.org/10.1145/3065386

    Article  Google Scholar 

  25. J. Langelaar, MNIST neural network training and testing (https://www.mathworks.com/matlabcentral/fileexchange/73010-mnist-neural-network-training-and-testing), MATLAB Central File Exchange. Retrieved December 11, 2020

  26. B. Lee, Y.S. Tarng, Application of the discrete wavelet transform to the monitoring of tool failure in end milling using the spindle motor current. Int. J. Adv. Manuf. Technol. (1999). https://doi.org/10.1007/s001700050062

    Article  Google Scholar 

  27. P. Liu, H. Zhang, W. Lian, W. Zuo, Multi-level wavelet convolutional neural networks: https://arxiv.org/pdf/1907.03128.pdf

  28. Z. Pan, H. Bolouri, High speed face recognition based on discrete cosine transforms and neural networks. http://citeseer.ist.psu.edu/270448.html. Submitted to IEEE Trans. On PAMI 1999

  29. B. Rajesh, M. Javed, Ratnesh, S. Srivastava, DCT-CompCNN: a novel image classification network using JPEG compressed DCT coefficients

  30. S. Said, O. Jemai, S. Hassairi, R. Ejbali, M. Zaied, C.B. Amar, Deep wavelet network for image classification. In 2016 IEEE International Conference on Systems, Man, and Cybernetics (20160, pp. 922–927

  31. S. Shubathra, P.C.D. Kalaivaani, S. Santhoshkumar, Clothing image recognition based on multiple features using deep neural networks. In International Conference on Electronics and Sustainable Communication Systems (ICESC) (IEEE , 2020)

  32. K. Simonyan, A. Zisserman, Very deep convolutional networks for large-scale image recognition, 2015, arXiv:1409.1556v6

  33. J.T. Springenberg, A. Dosovitskiy, T. Brox, M. Riedmiller, Striving for simplicity: the all convolutional net, workshop contribution at ICLR 2015, arXiv:1412.6806v3

  34. C. Thum, Measurement of the entropy of an image with application to image focusing (2010), pp. 203–211. https://doi.org/10.1080/713821475

  35. A. Torralba, R. Fergus, W. Freeman, 80 million tiny images: A large data set for nonparametric object and scene recognition. IEEE PAMI 30(11), 1958–1970 (2008)

    Article  Google Scholar 

  36. Y. Wang, C. Xu, C. Xu, D. Tao, Packing convolutional neural networks in the frequency domain. IEEE Trans. Pattern Anal. Mach. Intel. 41(10), 2495–2510 (2019)

    Article  Google Scholar 

  37. T. Williams, R. Li, Advanced image classification using wavelets and convolutional neural networks. In 2016 15th IEEE International Conference on Machine Learning and Applications (ICMLA) (2016), pp. 233–239

  38. H. Xiao, K. Rasul, R. Vollgraf, Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms (2017). https://arxiv.org/pdf/1708.07747.pdf

  39. B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, A. Torralba: Learning deep features for discriminative localization. In CVPR (2016)

Download references

Funding

Not Applicable.

Author information

Authors and Affiliations

  1. Department of Electrical and Computer Engineering, University of Central Florida, Orlando, USA

    Masoumeh Kalantari Khandani & Wasfy B. Mikhael

Authors
  1. Masoumeh Kalantari Khandani
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Wasfy B. Mikhael
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Masoumeh Kalantari Khandani.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kalantari Khandani, M., Mikhael, W.B. Enhancing Convolutional Neural Network Performance Using Domain Transforms in Constrained Networks. Circuits Syst Signal Process 41, 5160–5182 (2022). https://doi.org/10.1007/s00034-022-02026-2

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00034-022-02026-2

Keywords

Search

Navigation