Abstract
The chapter describes modern neural network designs and discusses their advantages and disadvantages. The state-of-the-art neural networks are usually too much computationally difficult which limits their use in mobile and IoT applications. However, they can be modified with special design techniques which would make them suitable for mobile or IoT applications with limited computational power. These techniques for designing more efficient neural networks are described in great detail. Using them opens a way to create extremely efficient neural networks for mobile or even IoT applications. Such neural networks make the applications very intelligent which paves the way for very smart sensors.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al.: TensorFlow: a system for large-scale machine learning. In: OSDI 16(2016), pp. 265–283 (2016)
Brabandere, B., Neven, D., Gool, L.: Semantic instance segmentation with a discriminative loss function. In: CoRR (2017). arXiv:1708.02551v1
Braun, M., Krebs, S., Flohr, F., Gavrila, D.M.: The EuroCity persons dataset: a novel benchmark for object detection. In: CoRR (2018). arXiv:1805.07193v2
Chen, L.-C., Papandreou, G., Schroff, F., Adam, H.: Rethinking atrous convolution for semantic image segmentation. In: CoRR preprint (2017). arXiv:1706.05587
Chen, L.-C., Hermans, A., Papandreou, G., Schroff, F., Wang, P., Adam, H.: MaskLab: instance segmentation by refining object detection with semantic and direction features. In: CVPR, pp. 4013–4022 (2018). arXiv:1712.04837v1
Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.L.: Deeplab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE TPAMI 40(4), pp. 834–848 (2018)
Chen, T., Li, M., Li, Y., Lin, M., Wang, N., Wang, M., Xiao, T., Xu, B., Zhang, Ch., Zhang, Z.: Mxnet: a flexible and efficient machine learning library for heterogeneous distributed systems. In: CoRR (2015). arXiv:1512.01274
Chen, Y., Li, J., Xiao, H., Jin, X., Yan, S., Feng, J.: Dual path networks. In: NIPS (2017) arXiv:1707.01629v2
Chollet, F.: Xception: deep learning with depthwise separable convolutions. In: CVPR, pp. 1251–1258 (2017). arXiv:1610.02357v3
Clevert, D., Unterthiner, T., Hochreiter, S.: Fast and accurate deep network learning by exponential linear units (ELUs). In: ICLR (2016). arXiv:1511.07289v5
Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., Schiele. B.: The cityscapes dataset for semantic urban scene understanding. In: CVPR (2016)
Dai, J., Li, Y., He, K., Sun, J.: R-FCN: object detection via region-based fully convolutional networks. In: NIPS, pp. 379–387 (2016). arXiv:1605.06409v2
Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: a large-scale hierarchical image database. In: CVPR (2009)
Denker, J.S., Gardner, W.R., Graf, H.P, Henderson, D., Howard, R.E., Hubbard, W., Jackel, L.D., BaIrd, H.S., Guyon I.: Neural Network Recognizer for Hand-Written Zip Code Digits. AT&T Bell Laboratories (1989)
Everingham, M., Van Gool, L., Williams, C.K., Winn, J., Zisserman, A.: The PASCAL Visual Object Classes (VOC) challenge. In: IJCV 88(2), pp. 303–338 (2010)
Everingham, M., Eslami, S.M.A., Gool, L.V., Williams, Ch.K.I., Winn, J., Zisserma, A.: The pascal visual object classes challenge a retrospective. In: IJCV 111(1), pp. 98–136 (2014)
Fukushima, K.: Neocognitron: a self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. In: Biol. Cybern. 36(4), pp. 193–202 (1980). https://doi.org/10.1007/bf00344251
Girshick, R., Donahue, J., Darrell, T., Malik, J.: Rich feature hierarchies for accurate object detection and semantic segmentation. In: CVPR (2014). arXiv:1311.2524v5
Girshick, R.: Fast R-CNN. In: ICCV (2015)
Han, S., Pool, J., Tran, J., Dally, W.: Learning both weights and connections for efficient neural network. In: Proceedings of Advanced in NIPS, Montreal, Canada, pp. 1135–1143 (2015). arXiv:1506.02626v3
Han, S., Mao, H., Dally, W.J.: Deep compression: compressing deep neural networks with pruning, trained quantization and Huffman coding. In: ICLR (2016). arXiv:1510.00149v5
Han, S., Liu, X., Mao, H., Pu, J., Pedram, A., Horowitz, M.A., Dally, W.J.: EIE: efficient inference engine on compressed deep neural network (2016). arXiv:1602.01528v1
He, K., Zhang, X., Ren, S., Sun, J.: Delving deep into rectifiers: surpassing human-level performance on ImageNet classification. In: ICCV (2015). arXiv:1502.01852v1
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR (2016), pp. 770–778. arXiv:1512.03385v1
He, K., Zhang, X., Ren, S., Sun, J.: Identity mappings in deep residual networks. In: ECCV (2016), pp. 630–645. arXiv:1603.05027v3
He, K., Gkioxari, G., Dollar, P., Girshick, R.: Mask R-CNN. In: ICCV (2017). arXiv:1703.06870v3
He, Y., Zhang, X., Sun, J.: Channel pruning for accelerating very deep neural networks. In: Proceedings of IEEE ICCV, vol. 2, pp. 1389–1397 (2017)
Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks. In: CVPR (2018), pp. 7132–7141. arXiv:1709.01507v2
Hinton, G.E., Srivastava, N., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.R.: Improving neural networks by preventing co-adaptation of feature detectors (2012). arXiv:1207.0580
Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., Adam, H.: MobileNets: efficient convolutional neural networks for mobile vision applications. In: CoRR (2017). arXiv:1704.04861v1
Huang, G., Sun, Y., Liuy, Z., Sedra, D., Weinberger, K.Q.: Deep networks with stochastic depth. In: ECCV (4), vol. 9908 of Lecture Notes in Computer Science, pp. 646–661 (2016). arXiv:1603.09382v3
Huang, G., Liu, Z., van der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: CVPR (2017). arXiv:1608.06993v5
Huang, J., Rathod, V., Sun, C., Zhu, M., Korattikara, A., Fathi, A., Fischer, I., Wojna, Z., Song, Y., Guadarrama, S., Murphy, K.: Speed/accuracy trade-offs for modern convolutional object detectors. In: CVPR (2017). arXiv:1611.10012v3
Hubara, I., Courbariaux, M., Soudry, D., El-Yaniv, R., Bengio, Y.: Quantized neural networks: training neural networks with low precision weights and activations. In: CoRR (2016). arXiv:1609.07061
Huh, M., Agrawal, P., Efros, A.: What makes ImageNet good for transfer learning? In: CoRR (2016). arXiv:1608.08614
Fu, C.-Y., Liu, W., Ranga, A., Tyagi, A., Berg, A.C.: DSSD: Deconvolutional single shot detector (2016). arXiv:1701.06659
Iandola, F.N., Moskewicz, M.W., Ashraf, K., Han, S., Dally, W.J., Keutzer, K.: SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <1 MB model size. In: CoRR (2016). arXiv:1602.07360v4
Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift. In: ICML, Lille, France, pp. 448–456 (2015)
Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R.B., Guadarrama, S., Darrell, T.: Caffe: convolutional architecture for fast feature embedding. In: Proceedings of ACM International Conference on Multimedia, pp. 675–678 (2014). arXiv:1408.5093
Krizhevsky, A.: Learning multiple layers of features from tiny images. Technical Report, University of Toronto (2009)
Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. Adv. Neural Inf. Process. Syst. 1097–1105 (2012)
Krizhevsky, A.: cuda-convnet: High-performance c++/cuda implementation of convolutional neural networks (2012)
LeCun, Y., Boser, B., Denker, J.S., Henderson, D., Howard, R.E., Hubbard, W., Jackel, L.D.: Backpropagation applied to handwritten zip code recognition. Neural Comput. 1(4), pp. 541–551 (1989)
LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), pp. 2278–2324 (1998)
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollar, P., Zitnick, C.L.: Microsoft COCO: Common objects in context. In: ECCV (2014). arXiv:1405.0312v3
Lin, T.-Y., Dollar, P., Girshick, R., He, K., Hariharan, B., Belongie, S.: Feature pyramid networks for object detection. In: CVPR (2017). arXiv:1612.03144v2
Lin, T.-Y., Goyal, P., Girshick, R., He, K., Dollar, P.: Focal loss for dense object detection. In: ICCV (2017). arXiv:1708.02002v2
Liu, W., Rabinovich, A., Berg, A.C.: ParseNet: looking wider to see better. In: ICLR (2016). arXiv:1506.04579v2
Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S.: SSD: single shot multibox detector. In: ECCV (2016)
Liu, S., Jia, J., Fidler, S., Urtasun, R.: SGN: sequential grouping networks for instance segmentation. In: ICCV, pp. 3516–3524 (2017)
Liu, S., Qi, L., Qin, H., Shi, J., Jia, J.: Path aggregation network for instance segmentation. In: CVPR, pp. 8759–8768 (2018). arXiv:1803.01534v1
Long, J., Shelhamer, E., Darrell, T.: ‘Fully convolutional networks for semantic segmentation’. In: CVPR, pp. 3431–3440 (2015). arXiv:1411.4038v2
Luo, W., Li, Y., Urtasun, R., Zemel, R.L.: Understanding the effective receptive field in deep convolutional neural networks. In: CoRR (2017). arXiv:1701.04128v2
Maas, A.L., Hannun, A.Y., Ng, A.Y.: Rectifier nonlinearities improve neural network acoustic models. In: ICML (2013)
Mahajan, D., Girshick, R., Ramanathan, V., He, K., Paluri, M., Li, Y., Bharambe, A., Maaten, L.: Exploring the limits of weakly supervised pretraining. In: ECCV (2), vol. 11206 of Lecture Notes in Computer Science, pp. 185–201 (2018). arXiv:1805.00932v1
Markidis, S., Chien, S.W.D., Laure, E., Peng, I.B., Vetter, J.S.: NVIDIA tensor core programmability, performance & precision. In: IEEE IPDPS Workshops, pp. 522–531 (2018). arXiv:1803.04014v1
Mavromoustakis, C.X., Batalla, J.M., Mastorakis, G., Markakis, E., Pallis, E.: Socially oriented edge computing for energy awareness in IoT architectures. IEEE Commun. Mag. 56(7), pp. 139–145 (2018)
Miyashita, D., Lee, E.H., Murmann, B.: Convolutional neural networks using logarithmic data representation. In: CoRR (2016). arXiv:1603.01025
Nair, V., Hinton, G.E.: Rectified linear units improve restricted Boltzmann machines. In: Proceedings of the 27th ICML, pp. 807–814 (2010)
Neuhold, G., Ollmann, T., Bulo, S.R., Kontschieder, P.: The mapillary vistas dataset for semantic understanding of street scenes. In: ICCV (2017)
Pleiss, G., Chen, D., Huang, G., Li, T., Maaten, L., Weinberger, K.Q.: Memory-efficient implementation of DenseNets. In: CoRR (2017). arXiv:1707.06990v1
Peng, C., Zhang, X., Yu, G., Luo, G., Sun, J.: ‘Large kernel matters—improve semantic segmentation by global convolutional network. In: CVPR, pp. 1743–1751 (2017). arXiv:1703.02719v1
Rastegari, M., Ordonez, V., Redmon, J., Farhadi, A.: XNOR-Net: imagenet classification using binary convolutional neural networks. In: Leibe, B., Matas, J., Sebe, N., Welling, M., (eds.) ECCV, vol. 4, pp. 525–542 (2016)
Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: You only look once: unified, real-time object detection. In: CVPR, pp. 779–788 (2016). arXiv:1506.02640v5
Redmon, J., Farhadi, A.: YOLO9000: better, faster, stronger. In: CVPR (2017). arXiv:1612.08242v1
Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. In: NIPS, pp. 91–99 (2015)
Ronneberger, O., Fischer, P., Brox, T.: ’U-Net: Convolutional networks for biomedical image segmentation. In: MICCAI (2015). arXiv:1505.04597v1
Ruan, J., Hu, X., Huo, X., Shi, Y., Chan, F.T.S., Wang, X., Manogaran, G., Mastorakis, G., Mavromoustakis, C.X., Zhao, X.: An IoT-based E-business model of intelligent vegetable greenhouses and its key operations management issues. In: Neural Comput. Appl. 2019, pp. 1–16 (2019)
Rumelhart, D.E., Hinton, G.E., Williams, R.J.: Learning representations by back-propagating errors. In: Nature 323(6088), pp. 533–536 (1986)
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A.C., Fei-Fei, L.: ImageNet large scale visual recognition challenge. In: IJCV 115(3), pp. 211–252 (2015)
Sanchez, J., Perronnin, F., Mensink, T., Verbeek, J.: Image classification with the fisher vector: theory and practice. In: RR-8209, INRIA (2013)
Sandler, M., Howard, A.G., Zhu, M., Zhmoginov, A., Chen, L.C.: MobileNetV2: Inverted Residuals and Linear Bottlenecks. In: ‘CVPR’ IEEE Computer Society, pp. 4510–4520 (2018). arXiv:1801.04381v3
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: ICLR (2015). arXiv:1409.1556v6
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. In: JMLR 15, pp. 1929–1958 (2014)
Srivastava, R.K., Greff, K., Schmidhuber, J.: Training very deep networks. In: NIPS (2015). arXiv:1507.06228v2
Sun, C., Shrivastava, A., Singh, S., Gupta, A.: Revisiting unreasonable effectiveness of data in deep learning era. In: Proceedings of ICCV (2017). arXiv:1707.02968v2
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: CVPR, pp. 1–9 (2015). arXiv:1409.4842v1
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision. In: CVPR, pp. 2818–2826 (2016). arXiv:1512.00567v3
Szegedy, C., Ioffe, S., Vanhoucke, V.: Inception-v4, inception-resnet and the impact of residual connections on learning. In: ICLR Workshop, p. 12 (2016). arXiv:1602.07261v2
Taigman, Y., Yang, M., Ranzato, M., Wolf, L.: Deepface: closing the gap to human-level performance in face verification. In: CVPR (2014), pp. 1701–1708 (2014)
Uijlings, J.R., van de Sande, K.E., Gevers, T., Smeulders, A.W.: Selective search for object recognition. In: IJCV (2013)
Welinder, P., Branson, S., Mita, T., Wah, C., Schroff, F., Belongie, S., Perona, P.: Caltech-UCSD Birds 200. Technical Report, Caltech (2010)
Xie, S., Girshick, R., Dollar, P., Tu, Z., He, K.: Aggregated residual transformations for deep neural networks. In: CVPR, pp. 5987–5995 (2017). arXiv:1611.05431v2
Zeiler, M.D., Fergus, R.: Visualizing and understanding convolutional neural networks. In: ECCV (2014). arXiv:1311.2901v3
Zhang, D., Yang, J., Ye, D., Hua, G.: Lq-nets: learned quantization for highly accurate and compact deep neural networks. In: Proceedings of ECCV (2018). arXiv:1807.10029v1
Zhang, X., Li, Z., Loy, C., Lin, D.: Polynet: a pursuit of structural diversity in very deep networks. In: CVPR (2017). arXiv:1611.05725v2
Zhang, X., Zhou, X., Lin, M., Sun, J.: ShuffleNet: an extremely efficient convolutional neural network for mobile devices. In: CVPR (2018). arXiv:1707.01083v2
Zhao, H., Shi, J., Qi, X., Wang, X., Jia, J.: Pyramid scene parsing network. In: CVPR (2017). arXiv:1612.01105v2
Zhou, A., Yao, A., Guo, Y., Xu, L., Chen, Y.: Incremental network quantization: towards lossless cnns with low-precision weights. In: Proceedings of ICLR (2017)
Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., Torralba, A.: Places: a 10 million image database for scene recognition. In: IEEE TPAMI (2017)
Zhuang, B., Shen, Ch., Tan, M., Liu, L., Reid, I.: Structured binary neural networks for accurate image classification and semantic segmentation. In: CoRR (2018). arXiv:1811.10413v2
Acknowledgements
This work has been supported by Slovak national project VEGA 2/0155/19.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this chapter
Cite this chapter
Malík, P., Krištofík, Š. (2020). AI Architectures for Very Smart Sensors. In: Mastorakis, G., Mavromoustakis, C., Batalla, J., Pallis, E. (eds) Convergence of Artificial Intelligence and the Internet of Things. Internet of Things. Springer, Cham. https://doi.org/10.1007/978-3-030-44907-0_16
Download citation
DOI: https://doi.org/10.1007/978-3-030-44907-0_16
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-44906-3
Online ISBN: 978-3-030-44907-0
eBook Packages: EngineeringEngineering (R0)