Abstract
In this paper, we propose different strategies for simplifying filters, used as feature extractors, to be learnt in convolutional neural networks (ConvNets) in order to modify the hypothesis space, and to speed-up learning and processing times. We study two kinds of filters that are known to be computationally efficient in feed-forward processing: fused convolution/sub-sampling filters, and separable filters. We compare the complexity of the back-propagation algorithm on ConvNets based on these different kinds of filters. We show that using these filters allows to reach the same level of recognition performance as with classical ConvNets for handwritten digit recognition, up to 3.3 times faster.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Lecun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. of the IEEE (November 1998)
Chellapilla, K., Puri, S., Simard, P.: High Performance Convolutional Neural Networks for Document Processing. In: Proc. of the Int. Workshop on Frontiers in Handwriting Recognition, IWFHR 2006 (2006)
Garcia, C., Delakis, M.: Convolutional Face Finder: a neural architecture for fast and robust face detection. IEEE Trans. on Pattern Analysis and Machine Intelligence (November 2004)
Osadchy, M., LeCun, Y., Miller, M.L., Perona, P.: Synergistic face detection and pose estimation with energy-based model. In: Proc. of Advances in Neural Information Processing Systems, NIPS 2005 (2005)
Garcia, C., Duffner, S.: Facial image processing with convolutional neural networks. In: Proc. Int. Workshop on Advances in Pattern Recognition (2007)
Delakis, M., Garcia, C.: Text detection with Convolutional Neural Networks. In: Proc. of the Int. Conf. on Computer Vision Theory and Applications (2008)
Saidane, Z., Garcia, C.: Automatic scene text recognition using a convolutional neural network. In: Proc. of Int. Workshop on Camera-Based Document Analysis and Recognition (2007)
Hadsell, R., Sermanet, P., Scoffier, M., Erkan, A., Kavackuoglu, K., Muller, U., LeCun, Y.: Learning long-range vision for autonomous off-road driving. Journal of Field Robotics (February 2009)
Raiko, T., Valpola, H., LeCun, Y.: Deep learning made easier by linear transformations in perceptrons. In: Conf. on AI and Statistics (2012)
Reed, R.: Pruning algorithms - a survey. IEEE Trans. on Neural Networks (1993)
Jarrett, K., Kavukcuoglu, K., Ranzato, M., LeCun, Y.: What is the best multi-stage architecture for object recognition? In: Proc. Int. Conf. on Computer Vision (2009)
Mrazova, I., Kukacka, M.: Hybrid convolutional neural networks. In: Proc. of IEEE Int. Conf. on Industrial Informatics, INDIN 2008 (2008)
Holt, J., Baker, T.: Back propagation simulations using limited precision calculations. In: Proc. of Int. Joint Conf. on Neural Networks, IJCNN 1991 (1991)
Petrowski, A.: Choosing among several parallel implementations of the backpropagation algorithm. In: Proc. of IEEE Int. Conf. on Neural Networks (1994)
Ciresan, D., Meier, U., Gambardella, L.M., Schmidhuber, J.: Handwritten digit recognition with a committee of deep neural nets on GPUs. In: Computing Research Repository (2011)
Mamalet, F., Roux, S., Garcia, C.: Real-time video convolutional face finder on embedded platforms. EURASIP Journal on Embedded Systems (2007)
Mamalet, F., Roux, S., Garcia, C.: Embedded facial image processing with convolutional neural networks. In: Proc. of Int. Symp. on Circuits and Systems (2010)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2012 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Mamalet, F., Garcia, C. (2012). Simplifying ConvNets for Fast Learning. In: Villa, A.E.P., Duch, W., Érdi, P., Masulli, F., Palm, G. (eds) Artificial Neural Networks and Machine Learning – ICANN 2012. ICANN 2012. Lecture Notes in Computer Science, vol 7553. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-33266-1_8
Download citation
DOI: https://doi.org/10.1007/978-3-642-33266-1_8
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-33265-4
Online ISBN: 978-3-642-33266-1
eBook Packages: Computer ScienceComputer Science (R0)