Advertisement

Discrete Cosine Transform Spectral Pooling Layers for Convolutional Neural Networks

  • James S. SmithEmail author
  • Bogdan M. WilamowskiEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10841)

Abstract

Pooling operations for convolutional neural networks provide the opportunity to greatly reduce network parameters, leading to faster training time and less data overfitting. Unfortunately, many of the common pooling methods such as max pooling and mean pooling lose information about the data (i.e., they are lossy methods). Recently, spectral pooling has been utilized to pool data in the spectral domain. By doing so, greater information can be retained with the same network parameter reduction as spatial pooling. Spectral pooling is currently implemented in the discrete Fourier domain, but it is found that implementing spectral pooling in the discrete cosine domain concentrates energy in even fewer spectra. Although Discrete Cosine Transforms Spectral Pooling Layers (DCTSPL) require extra computation compared to normal spectral pooling, the overall time complexity does not change and, furthermore, greater information preservation is obtained, producing networks which converge faster and achieve a lower misclassification error.

References

  1. 1.
    Cooley, J.W., Tukey, J.W.: An algorithm for the machine calculation of complex Fourier series. Math. Comput. 19(90), 297–301 (1965)MathSciNetCrossRefGoogle Scholar
  2. 2.
    Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT Press, Cambridge (2016)zbMATHGoogle Scholar
  3. 3.
    Highlander, T., Rodriguez, A.: Very efficient training of convolutional neural networks using fast Fourier transform and overlap-and-add. arXiv:1601.06815 [cs], January 2016
  4. 4.
    Jarrett, K., Kavukcuoglu, K., Ranzato, M., LeCun, Y.: What is the best multi-stage architecture for object recognition? In: 2009 IEEE 12th International Conference on Computer Vision, pp. 2146–2153, September 2009Google Scholar
  5. 5.
    Ko, J.H., Mudassar, B., Na, T., Mukhopadhyay, S.: Design of an energy-efficient accelerator for training of convolutional neural networks using frequency-domain computation. In: Proceedings of the 54th Annual Design Automation Conference 2017, DAC 2017, pp. 59:1–59:6. ACM, New York (2017)Google Scholar
  6. 6.
    Lecun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)CrossRefGoogle Scholar
  7. 7.
    Mathieu, M., Henaff, M., LeCun, Y.: Fast training of convolutional networks through FFTs. arXiv:1312.5851 [cs], December 2013
  8. 8.
    Olejczak, A., Korniak, J., Wilamowski, B.M.: Discrete cosine transformation as alternative to other methods of computational intelligence for function approximation. In: Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) ICAISC 2017. LNCS (LNAI), vol. 10245, pp. 143–153. Springer, Cham (2017).  https://doi.org/10.1007/978-3-319-59063-9_13CrossRefGoogle Scholar
  9. 9.
    Rippel, O., Snoek, J., Adams, R.P.: Spectral representations for convolutional neural networks. arXiv:1506.03767 [cs, stat], June 2015
  10. 10.
    Scherer, D., Müller, A., Behnke, S.: Evaluation of pooling operations in convolutional architectures for object recognition. In: Diamantaras, K., Duch, W., Iliadis, L.S. (eds.) Part III, ICANN 2010. LNCS, vol. 6354, pp. 92–101. Springer, Heidelberg (2010).  https://doi.org/10.1007/978-3-642-15825-4_10CrossRefGoogle Scholar
  11. 11.
    Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15, 1929–1958 (2014)MathSciNetzbMATHGoogle Scholar
  12. 12.
    Wang, Y., Xu, C., You, S., Tao, D., Xu, C.: CNNpack: packing convolutional neural networks in the frequency domain. In: Lee, D.D., Sugiyama, M., Luxburg, U.V., Guyon, I., Garnett, R. (eds.) Advances in Neural Information Processing Systems, vol. 29, pp. 253–261. Curran Associates, Inc., Red Hook (2016)Google Scholar
  13. 13.
    Watson, A.B.: Image compression using the discrete cosine transform. Math. J. 4(1), 81–88 (1994)MathSciNetGoogle Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  1. 1.ECE DepartmentAuburn UniversityAuburnUSA
  2. 2.University of IT and Management, RzeszowRzeszówPoland

Personalised recommendations