Skip to main content
SpringerLink
Log in

TinyEmergencyNet: a hardware-friendly ultra-lightweight deep learning model for aerial scene image classification

  • Research
  • Published:
Journal of Real-Time Image Processing Aims and scope Submit manuscript

Abstract

In the context of emergency response applications, real-time situational awareness is vital. Unmanned aerial vehicles (UAVs) with imagers have emerged as crucial tools for providing timely information in such scenarios. Convolutional neural networks (CNN) are effective in image processing. However, the deployment of CNN models in UAVs faces significant challenges. The CNN models involve large number of parameters and energy-costly floating-point computations beyond the memory and power available on-board the UAVs. To address these challenges, we propose a co-design optimization approach for deploying the EmergencyNet CNN model on resource-constrained UAVs. Our strategy includes channel-wise pruning to reduce the size and optimize the network architecture. Additionally, we apply additive powers-of-two (APoT) quantization to further compress the model and enhance computational efficiency. Using channel-wise network pruning we derive TinyEmergencyNet that is only 155KB in memory size and 50% smaller than EmergencyNet. This proposed approach is evaluated on Aerial Image Disaster Event Recognition (AIDER) dataset. We have achieved an F1-score of 93.6% with 4-bit APoT quantization that closely approaches the full precision (32-bit) accuracy of 94%. Furthermore, hardware-friendly bit-shifting operations as a result of APoT quantization present an added advantage in hardware accelerator implementations. This work pioneers the joint application of channel-wise pruning and non-uniform APoT quantization on EmergencyNet, presenting a suitable solution tailored for UAV-based emergency response applications.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Data availability

The data that support the findings of this study are available from the corresponding author upon request.

References

  1. Alawad, W., Halima, NB., Aziz, L.: An unmanned aerial vehicle (UAV) system for disaster and crisis management in smart cities. Electronics (2023)

  2. Bi, Q., Qin, K., Zhang, H., Xie, J., Li, Z., Xu, K.: APDC-Net: attention pooling-based convolutional network for aerial scene classification. IEEE Geosci. Remote Sens. Lett. 17, 1603–1607 (2020)

    Article  Google Scholar 

  3. Cai, H., Zhu, L., Han, S.: ProxylessNAS: direct neural architecture search on target task and hardware. (2018). arXiv preprint arXiv:1812.00332

  4. Cai, H., Gan, C., Wang, T., Zhang, Z., Han, S.: Once-for-All: train one network and specialize it for efficient deployment. (2019). arXiv preprint arXiv:1908.09791

  5. Chitty-Venkata, K.T., Somani, A.K.: Neural architecture search survey: a hardware perspective. ACM Comput. Surv. 55(4), 1–36 (2022)

    Article  Google Scholar 

  6. Crowley, EJ., Turner, J., Storkey, AJ., O’Boyle, MFP.: A closer look at structured pruning for neural network compression. (2018) arXiv: Machine Learning

  7. Dong, Z., Gao, Y., Huang, Q., Wawrzynek, J., So, HK., Keutzer, K.: HAO: Hardware-aware neural architecture optimization for efficient inference. In: 2021 IEEE 29th Annual International Symposium on Field-Programmable Custom Computing Machines (FCCM), IEEE, pp 50–59 (2021)

  8. Elhoushi, M., Shafiq, F., Tian, YH., Li, JY., Chen, Z.: DeepShift: towards multiplication-less neural networks. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) pp 2359–2368 (2019)

  9. Fang, G., Ma, X., Song, M., Mi, MB., Wang, X.: DepGraph: towards any structural pruning. arXiv:2301.12900 (2023)

  10. Ghanbari, H., Mahdianpari, M., Homayouni, S., Mohammadimanesh, F.: A meta-analysis of convolutional neural networks for remote sensing applications. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 14, 3602–3613 (2021)

    Article  Google Scholar 

  11. He, Y., Lin, J., Liu, Z., Wang, H., Li, LJ., Han, S.: AMC: AutoML for model compression and acceleration on mobile devices. In: Proceedings of the European conference on computer vision (ECCV), pp 784–800 (2018)

  12. Herman, JT., Zewail, R., Ogawa, T., Elsagheer, SA.: A lightweight transfer learning-based model for building classification in aerial imagery. 2023 15th International Conference on Computer Research and Development (ICCRD) pp 181–186 (2023)

  13. Howard, AG., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., Adam, H.: MobileNets: Efficient convolutional neural networks for mobile vision applications. ArXiv abs/1704.04861 (2017)

  14. Hu, J., Shu, Q., Pan, J., Tu, J., Zhu, Y., Wang, M.: MINet: multilevel inheritance network-based aerial scene classification. IEEE Geosci. Remote Sens. Lett. 19, 1–5 (2022)

    Google Scholar 

  15. Huanhuan, L., Guofeng, P., Hui, Z.: Depthwise separable residual network for remote sensing image scene classification. 2022 International Symposium on Advances in Informatics, Electronics and Education (ISAIEE) pp 533–539 (2022)

  16. Iandola, FN., Moskewicz, MW., Ashraf, K., Han, S., Dally, WJ., Keutzer, K.: SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and \(<\)1MB model size. arXiv:1602.07360 (2016)

  17. Jiang, W., Lou, Q., Yan, Z., Yang, L., Hu, J., Hu, X.S., Shi, Y.: Device-circuit-architecture co-exploration for computing-in-memory neural accelerators. IEEE Trans. Comput. 70(4), 595–605 (2020)

    Article  MathSciNet  Google Scholar 

  18. Kyrkou, C., Theocharides, T.: Deep-learning-based aerial image classification for emergency response applications using unmanned aerial vehicles. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) pp 517–525 (2019)

  19. Kyrkou, C., Theocharides, T.: EmergencyNet: efficient aerial image classification for drone-based emergency monitoring using atrous convolutional feature fusion. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 13, 1687–1699 (2020)

    Article  Google Scholar 

  20. Li, Y., Dong, X., Wang, W.: Additive powers-of-two quantization: An efficient non-uniform discretization for neural networks. arXiv preprint arXiv:1909.13144 (2019)

  21. Lin, Y., Yang, M., Han, S.: NAAS: neural accelerator architecture search. In: 2021 58th ACM/IEEE Design Automation Conference (DAC), IEEE, pp 1051–1056 (2021)

  22. Liu, H., Simonyan, K., Yang, Y.: DARTS: differentiable architecture search. arXiv preprint arXiv:1806.09055 (2018)

  23. Lu, Q., Jiang, W., Xu, X., Shi, Y., Hu, J.: On neural architecture search for resource-constrained hardware platforms. arXiv preprint arXiv:1911.00105 (2019)

  24. Lu, Y., Gong, M., Hu, Z., Zhao, W., Guan, Z., Zhang, M.: Energy-based cnn pruning for remote sensing scene classification. IEEE Trans. Geosci. Remote Sens. 61, 1–14 (2023)

    Google Scholar 

  25. Ma, X., Lin, S., Ye, S., He, Z., Zhang, L., Yuan, G., Tan, S.H., Li, Z., Fan, D., Qian, X., Lin, X., Ma, K., Wang, Y.: Non-structured DNN weight pruning-is it beneficial in any platform? IEEE Trans. Neural Netw. Learn. Syst. 33, 4930–4944 (2019)

    Article  Google Scholar 

  26. Mao, H., Han, S., Pool, J., Li, W., Liu, X., Wang, Y., Dally, WJ.: Exploring the regularity of sparse structure in convolutional neural networks. CoRR abs/1705.08922 (2017)

  27. Mills, KG., Han, FX., Zhang, J., Changiz Rezaei, SS., Chudak, F., Lu, W., Lian, S., Jui, S., Niu, D.: Profiling neural blocks and design spaces for mobile neural architecture search. In: Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pp 4026–4035 (2021)

  28. Miyashita, D., Lee, EH., Murmann, B.: Convolutional neural networks using logarithmic data representation. arXiv:1603.01025 (2016)

  29. Munawar, HS., Ullah, F., Qayyum, S., Heravi, A.: Application of deep learning on UAV-based aerial images for flood detection. Smart Cities. (2021)

  30. Nagel, M., van Baalen, M., Blankevoort, T., Welling, M.: Data-free quantization through weight equalization and bias correction. 2019 IEEE/CVF International Conference on Computer Vision (ICCV) pp 1325–1334 (2019)

  31. Nagel, M., Amjad, RA., van Baalen, M., Louizos, C., Blankevoort, T.: Up or down? adaptive rounding for post-training quantization. arXiv:2004.10568 (2020)

  32. Nagel, M., Fournarakis, M., Amjad, RA., Bondarenko, Y., van Baalen, M., Blankevoort, T.: A white paper on neural network quantization. arXiv:2106.08295 (2021)

  33. Osco, LP., Junior, JM., Ramos, APM., Jorge, L., Fatholahi, SN., de Andrade Silva, J., Matsubara, ET., Pistori, H., Gonçalves, WN., Li, J.: A review on deep learning in UAV remote sensing. arXiv:2101.10861 (2021)

  34. Petrides, P., Kolios, P., Kyrkou, C., Theocharides, T., Panayiotou, C. Disaster prevention and emergency response using unmanned aerial systems. Smart Cities in the Mediterranean: Coping with Sustainability Objectives in Small and Medium-sized Cities and Island Communities pp 379–403 (2017)

  35. Petso, T., Jamisola, RS.: A review on deep learning on UAV monitoring systems for agricultural applications. Artificial Intelligence for Robotics and Autonomous Systems Applications pp 335–368 (2023)

  36. Przewlocka-Rus, D., Sarwar, SS., Sumbul, HE., Li, Y., de Salvo, B. Power-of-two quantization for low bitwidth and hardware compliant neural networks. arXiv:2203.05025 (2022)

  37. Sekanina, L.: Neural architecture search and hardware accelerator co-search: a survey. IEEE Access 9, 151337–151362 (2021)

    Article  Google Scholar 

  38. Sen, O., Keles, H.Y.: On the evaluation of CNN models in remote-sensing scene classification domain. PFG - J. Photogramm. Remote Sens. Geoinf. Sci. 88, 477–492 (2020)

    Google Scholar 

  39. Sheng, T., Feng, C., Zhuo, S., Zhang, X., Shen, L., Aleksic, M. A quantization-friendly separable convolution for MobileNets. 2018 1st Workshop on Energy Efficient Machine Learning and Cognitive Computing for Embedded Applications (EMC2) pp 14–18 (2018)

  40. Sui, X., Lv, Q., Bai, Y., Zhu, B., Zhi, L., Yang, Y., Tan, Z.: A hardware-friendly low-bit power-of-two quantization method for CNNs and its FPGA implementation. Sensors (Basel, Switzerland) 22, 6618 (2022)

    Article  Google Scholar 

  41. Sze, V., Chen, Y.H., Yang, T.J., Emer, J.S.: Efficient processing of deep neural networks: A tutorial and survey. Proc. IEEE 105(12), 2295–2329 (2017)

    Article  Google Scholar 

  42. Tan, M., Le, QV.: EfficientNetV2: smaller models and faster training. hyperimagehttp://arxiv.org/abs/2104.00298arXiv:2104.00298 (2021)

  43. Wang, K., Liu, Z., Lin, Y., Lin, J., Han, S.: HAQ: Hardware-aware automated quantization with mixed precision. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 8612–8620 (2019)

  44. Wang, X., Yuan, L., Xu, H., Wen, X.: CSDS: End-to-end aerial scenes classification with depthwise separable convolution and an attention mechanism. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 14, 10484–10499 (2021)

    Article  Google Scholar 

  45. White, C., Safari, M., Sukthanker, R., Ru, B., Elsken, T., Zela, A., Dey, D., Hutter, F.: Neural architecture search: Insights from 1000 papers. arXiv preprint arXiv:2301.08727 (2023)

  46. Yan, Z., Juan, DC., Hu, XS., Shi, Y.: Uncertainty modeling of emerging device based computing-in-memory neural accelerators with application to neural architecture search. In: Proceedings of the 26th Asia and South Pacific Design Automation Conference, pp 859–864 (2021)

  47. Yuan, X., Azimi, S., Henry, C., Gstaiger, V., Codastefano, M., Manalili, M., Cairo, S., Modugno, S., Wieland, M., Schneibel, A., et al.: Automated building segmentation and damage assessment from satellite images for disaster relief. Int. Arch. Photogramm. Remote. Sens. Spat. Inf. Sci. 43, 741–748 (2021)

    Article  Google Scholar 

  48. Zhang, C., Yuan, G., Niu, W., Tian, J., Jin, S., Zhuang, D., Jiang, Z., Wang, Y., Ren, B., Song, S.L., Tao, D.: ClickTrain: efficient and accurate end-to-end deep learning training via fine-grained architecture-preserving pruning. Proceedings of the ACM International Conference on Supercomputing (2020)

  49. Zhou, A., Yao, A., Guo, Y., Xu, L., Chen, Y.: Incremental network quantization: Towards lossless cnns with low-precision weights. arXiv:1702.03044 (2017)

  50. Zoph, B., Le, QV.: Neural architecture search with reinforcement learning. arXiv preprint arXiv:1611.01578 (2016)

Download references

Acknowledgements

We would like to thank Egypt-Japan University of Science and Technology (E-JUST) for continued support. Furthermore, we extend our heartfelt appreciation for the support from the TICAD7 (Tokyo International Conference on African Development) Scholarship program.

Author information

Authors and Affiliations

  1. Department of Electronics and Communications Engineering, Egypt-Japan University of Science and Technology (E-JUST), New Borg El-Arab City, Alexandria, 21934, Egypt

    Obed M. Mogaka & Mohammed S. Sayed

  2. Department of Computer Science and Engineering, Egypt-Japan University of Science and Technology (E-JUST), New Borg El-Arab City, Alexandria, 21934, Egypt

    Rami Zewail

  3. Department of Advanced Information Technology, Kyushu University, Nishi-ku, Fukuoka, 819-0395, Japan

    Koji Inoue

  4. Department of Electronics and Communications Engineering, Zagazig University, Zagazig, Egypt

    Mohammed S. Sayed

Authors
  1. Obed M. Mogaka
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Rami Zewail
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Koji Inoue
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Mohammed S. Sayed
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Obed M. Mogaka.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mogaka, O.M., Zewail, R., Inoue, K. et al. TinyEmergencyNet: a hardware-friendly ultra-lightweight deep learning model for aerial scene image classification. J Real-Time Image Proc 21, 51 (2024). https://doi.org/10.1007/s11554-024-01430-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11554-024-01430-y

Keywords

Search

Navigation