Abstract
State-of-the-art object detection in search and rescue robotics relies on CNNs, which reach excellent scores but lack explainability due to their “black box” characteristic. In such a domain, mission failure or misdetections can have drastic consequences. Therefore research should strive to increase the explainability regarding the CNNs’ classification strategies. In this paper, existing methods for object detection are applied and investigated in the context of search and rescue robotics in order to compose a fully explainable pipeline. Unlike existing object detection methods, the presented method is based on an exhaustive model investigation concerning post-hoc explainability. The method is applied to detecting handwheels of gate-valves, with a post-hoc analysis of the classification strategies learned by the object detector. In order to train and test the object detection model, a novel dataset is composed, including 2447 images, nine handwheel types, and 6696 annotations. Five CNN object detectors (R-CNN, Faster R-CNN, RetinaNet, YOLOv5s, and SSD) are compared based on mAP0.5 and mAP0.5:0.95. Two CNN object detectors’ classification strategies were investigated with the SpRAy method. Comparing mAP0.5, mAP0.5:0.95, and inference times reveals that YOLOv5s is the superior model across all categories. SpRAy analysis of R-CNN and YOLOv5s did not reveal any abnormal classification strategies, which indicates a well-balanced dataset. YOLOv5s appears to have learned different classification strategies for different handwheel types. Our handwheels dataset is available at: https://www.kaggle.com/hoenigpet/handwheels-for-gatevalves.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Adadi, A., Berrada, M.: Peeking inside the black-box: a survey on explainable artificial intelligence (XAI). IEEE Access 6, 52138–52160 (2018). https://doi.org/10.1109/ACCESS.2018.2870052
Anders, C.J., Neumann, D., Marin, T., Samek, W., Müller, K.R., Lapuschkin, S.: XAI for analyzing and unlearning spurious correlations in imagenet. In: ICML’20 Workshop on Extending Explainable AI Beyond Deep Models and Classifiers (XXAI) (2020)
Anders, C.J., Neumann, D., Samek, W., Müller, K., Lapuschkin, S.: Software for dataset-wide XAI: from local explanations to global insights with zennit, corelay, and virelay. CoRR abs/2106.13200 (2021). https://arxiv.org/abs/2106.13200
Anders, C.J., Weber, L., Neumann, D., Samek, W., Müller, K.R., Lapuschkin, S.: Finding and removing clever hans: Using explanation methods to debug and improve deep models. Inf. Fusion 77, 261–295 (2022)
Arrieta, A.B., et al.: Explainable artificial intelligence (XAI): concepts, taxonomies, opportunities and challenges toward responsible AI. Inf. Fusion 58, 82–115 (2020)
Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.R., Samek, W.: On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLoS ONE 10(7), e0130140 (2015)
Bengio, Y., Courville, A., Vincent, P.: Representation learning: a review and new perspectives. IEEE Trans. Pattern Anal. Mach. Intell. 35(8), 1798–1828 (2013)
Doshi-Velez, F., Kim, B.: Considerations for evaluation and generalization in interpretable machine learning. In: Escalante, H.J., Escalera, S., Guyon, I., Baró, X., Güçlütürk, Y., Güçlü, U., van Gerven, M. (eds.) Explainable and Interpretable Models in Computer Vision and Machine Learning. TSSCML, pp. 3–17. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-98131-4_1
Fung, A., Wang, L.Y., Zhang, K., Nejat, G., Benhabib, B.: Using deep learning to find victims in unknown cluttered urban search and rescue environments. Curr. Robot. Rep. 1(3), 105–115 (2020)
Girshick, R., Donahue, J., Darrell, T., Malik, J.: Rich feature hierarchies for accurate object detection and semantic segmentation. In: 2014 IEEE Conference on Computer Vision and Pattern Recognition, pp. 580–587. IEEE (2013)
Goodfellow, I.J., Bengio, Y., Courville, A.: Deep Learning. MIT Press, Cambridge (2016)
Jocher, G., et al.: ultralytics/yolov5: v6.1 - tensorrt, tensorflow edge TPU and openvino export and inference (2022). https://doi.org/10.5281/zenodo.6222936
Karasmanoglou, A., Antonakakis, M., Zervakis, M.: Heatmap-based explanation of YOLOv5 object detection with layer-wise relevance propagation. In: 2022 IEEE International Conference on Imaging Systems and Techniques (IST), pp. 1–6. IEEE (2022)
Lapuschkin, S., Wäldchen, S., Binder, A., Montavon, G., Samek, W., Müller, K.R.: Unmasking clever hans predictors and assessing what machines really learn. Nat. Commun. 10(1), 1–8 (2019)
Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollar, P.: Focal loss for dense object detection. IEEE Trans. Pattern Anal. Mach. Intell. 42(2), 318–327 (2020)
Lin, T.Y., et al.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10602-1_48
Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.): ECCV 2016. LNCS, vol. 9906. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46475-6
Miyakawa, K., Kanda, T., Ohya, J., Ogata, H., Hashimoto, K., Takanishi, A.: Automatic estimation of the position and orientation of stairs to be reached and climbed by a disaster response robot by analyzing 2D image and 3D point cloud. Int. J. Mech. Eng. Rob. Res. 9(9), 1312–1321 (2020)
Nishikawa, K., Ohya, J., Matsuzawa, T., Takanishi, A., Ogata, H., Hashimoto, K.: Automatic detection of valves with disaster response robot on basis of depth camera information. In: 2018 Digital Image Computing: Techniques and Applications (DICTA). IEEE (2019)
Pearl, J., Mackenzie, D.: The Book of Why. Basic Books, New York (2018)
Redmon, J., Farhadi, A.: YOLOv3: An incremental improvement. arXiv preprint arXiv:1804.02767 (2018). https://arxiv.org/abs/1804.02767v1
Refaeilzadeh, P., Tang, L., Liu, H.: Cross-validation. Encycl. Database Syst. 1–7 (2016)
Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Trans. Pattern Anal. Mach. Intell. 39(6), 1137–1149 (2017)
Schneider, F.E.: ELROB - the European land robot trial (2022). https://www.elrob.org/
Schneider, F.E., Wildermuth, D.: Assessing the search and rescue domain as an applied and realistic benchmark for robotic systems. In: 2016 17th International Carpathian Control Conference (ICCC), pp. 657–662. IEEE (2016)
Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-CAM: visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 618–626 (2017)
Shim, V.A., Yuan, M., Tan, B.H.: Automatic object searching by a mobile robot with single RGB-D camera. In: 2017 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), pp. 56–62. IEEE (2018)
Tjoa, E., Guan, C.: A survey on explainable artificial intelligence (XAI): towards medical XAI. IEEE Trans. Neural Netw. Learn. Syst. 32(11), 4793–4813 (2019)
Turk, M.A., Pentland, A.P.: Face recognition using eigenfaces. In: Proceedings of the 1991 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 586–587. IEEE Computer Society (1991)
Wang, J., Jing, X., Yan, Z., Fu, Y., Pedrycz, W., Yang, L.T.: A survey on trust evaluation based on machine learning. ACM Comput. Surv. 53(5), 1–36 (2020)
Wöber, W.: Identifying geographically di erentiated features of ethopian nile tilapia (oreochromis niloticus) morphology with machine learning. PLoS ONE 16, 1–30 (2021)
Wöber, W., Mehnen, L., Curto, M., Tibihika, P.D., Tesfaye, G., Meimberg, H.: Investigating shape variation using generalized procrustes analysis and machine learning. Appl. Sci. 12(6) (2022)
Xiang, Y., Schmidt, T., Narayanan, V., Fox, D.: PoseCNN: a convolutional neural network for 6D object pose estimation in cluttered scenes. arXiv preprint arXiv:1711.00199 (2017). https://arxiv.org/abs/1711.00199v3
Zhang, J., Yin, B., Xiao, X., Yang, H.: 3D detection and 6D pose estimation of texture-less objects for robot grasping. In: 2021 6th International Conference on Control and Robotics Engineering, ICCRE 2021, pp. 33–38. IEEE (2021)
Acknowledgments
We gratefully acknowledge the support of the Austrian Research Promotion Agency (FFG), project SmartDis, and the Austrian Science Fund (FWF), under project No. I 6114, project iChores.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Hönig, P., Wöber, W. (2023). Explainable Object Detection in the Field of Search and Rescue Robotics. In: Petrič, T., Ude, A., Žlajpah, L. (eds) Advances in Service and Industrial Robotics. RAAD 2023. Mechanisms and Machine Science, vol 135. Springer, Cham. https://doi.org/10.1007/978-3-031-32606-6_5
Download citation
DOI: https://doi.org/10.1007/978-3-031-32606-6_5
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-32605-9
Online ISBN: 978-3-031-32606-6
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)