Input Fast-Forwarding for Better Deep Learning

  • Ahmed Ibrahim
  • A. Lynn Abbott
  • Mohamed E. Hussein
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10317)

Abstract

This paper introduces a new architectural framework, known as input fast-forwarding, that can enhance the performance of deep networks. The main idea is to incorporate a parallel path that sends representations of input values forward to deeper network layers. This scheme is substantially different from “deep supervision”, in which the loss layer is re-introduced to earlier layers. The parallel path provided by fast-forwarding enhances the training process in two ways. First, it enables the individual layers to combine higher-level information (from the standard processing path) with lower-level information (from the fast-forward path). Second, this new architecture reduces the problem of vanishing gradients substantially because the fast-forwarding path provides a shorter route for gradient backpropagation. In order to evaluate the utility of the proposed technique, a Fast-Forward Network (FFNet), with 20 convolutional layers along with parallel fast-forward paths, has been created and tested. The paper presents empirical results that demonstrate improved learning capacity of FFNet due to fast-forwarding, as compared to GoogLeNet (with deep supervision) and CaffeNet, which are \(4{\times }\) and \(18{\times }\) larger in size, respectively. All of the source code and deep learning models described in this paper will be made available to the entire research community (https://github.com/aicentral/FFNet).

References

  1. 1.
    BVLC reference CaffeNet model. https://github.com/BVLC/caffe/tree/master/models/bvlc_reference_caffenet. Accessed June 2016
  2. 2.
    Chetlur, S., Woolley, C., Vandermersch, P., Cohen, J., Tran, J., Catanzaro, B., Shelhamer, E.: cuDNN: efficient primitives for deep learning. arXiv preprint arXiv:1410.0759 (2014)
  3. 3.
    Cristianini, N., Shawe-Taylor, J.: An Introduction to Support Vector Machines and Other Kernel-Based Learning Methods. Cambridge University Press, Cambridge (2000)CrossRefMATHGoogle Scholar
  4. 4.
    Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 248–255 (2009)Google Scholar
  5. 5.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778 (2016)Google Scholar
  6. 6.
    Ibrahim, A., Abbott, A.L., Hussein, M.E.: An image dataset of text patches in everyday scenes. In: Bebis, G., et al. (eds.) ISVC 2016. LNCS, vol. 10073, pp. 291–300. Springer, Cham (2016). doi:10.1007/978-3-319-50832-0_28 CrossRefGoogle Scholar
  7. 7.
    Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., Darrell, T.: Caffe: convolutional architecture for fast feature embedding. In: 22nd ACM International Conference on Multimedia, pp. 675–678 (2014)Google Scholar
  8. 8.
    Krizhevsky, A., Hinton, G.: Learning multiple layers of features from tiny images. Master’s thesis, Department of Computer Science, University of Toronto (2009)Google Scholar
  9. 9.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: Proceedings of Advances in Neural Information Processing Systems (NIPS), pp. 1097–1105 (2012)Google Scholar
  10. 10.
    LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86, 2278–2324 (1998)CrossRefGoogle Scholar
  11. 11.
    Lee, C.Y., Xie, S., Gallagher, P.W., Zhang, Z., Tu, Z.: Deeply-supervised nets. In: Proceedings of the 18th International Conference on Artificial Intelligence and Statistics (AISTATS), vol. 2, pp. 562–570 (2015)Google Scholar
  12. 12.
    Liu, Y., Lew, M.S.: Learning relaxed deep supervision for better edge detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 231–240 (2016)Google Scholar
  13. 13.
    Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1–9 (2015)Google Scholar
  14. 14.
    Torralba, A., Fergus, R., Freeman, W.T.: 80 million tiny images: a large data set for nonparametric object and scene recognition. IEEE Trans. Pattern Anal. Mach. Intell. (PAMI) 30(11), 1958–1970 (2008)CrossRefGoogle Scholar
  15. 15.
    Veit, A., Matera, T., Neumann, L., Matas, J., Belongie, S.: COCO-Text: dataset and benchmark for text detection and recognition in natural images. arXiv preprint arXiv:1601.07140 (2016). http://vision.cornell.edu/se3/wp-content/uploads/2016/01/1601.07140v1.pdf
  16. 16.
    Wang, Q., JaJa, J.: From maxout to channel-out: encoding information on sparse pathways. In: Wermter, S., et al. (eds.) ICANN 2014. LNCS, vol. 8681, pp. 273–280. Springer, Cham (2014). doi:10.1007/978-3-319-11179-7_35 Google Scholar
  17. 17.
    Whaley, R.C., Dongarra, J.J.: Automatically tuned linear algebra software. In: Proceedings of the ACM/IEEE Conference on Supercomputing, pp. 1–27 (1998)Google Scholar
  18. 18.
    Xianyi, Z., Qian, W., Chothia, Z.: OpenBLAS (2014). http://xianyi.github.io/OpenBLAS
  19. 19.
    Xie, S., Tu, Z.: Holistically-nested edge detection. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 1395–1403 (2015)Google Scholar
  20. 20.
    Zeiler, M., Fergus, R.: Stochastic pooling for regularization of deep convolutional neural networks. In: Proceedings of the International Conference on Learning Representation (ICLR) (2013)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Ahmed Ibrahim
    • 1
    • 4
  • A. Lynn Abbott
    • 1
  • Mohamed E. Hussein
    • 2
    • 3
  1. 1.Virginia Polytechnic Institute and State UniversityBlacksburgUSA
  2. 2.Egypt-Japan University of Science and TechnologyNew Borg El ArabEgypt
  3. 3.Alexandria UniversityAlexandriaEgypt
  4. 4.Benha UniversityBanhaEgypt

Personalised recommendations