Abstract
This paper highlights the significance of maintaining and enhancing situational awareness in Urban Search and Rescue (USAR) missions. It focuses specifically on investigating the capabilities of Unmanned Aerial Vehicles (UAV) equipped with limited sensing capabilities and onboard computational resources to perform visual inspections of apriori unknown fractured and collapsed structures in unfamiliar environments. The proposed approach, referred to as First Look Inspect-Explore (FLIE), employs a flexible bifurcated behavior tree that leverages real-time RGB image and depth cloud data. By employing a recursive and reactive synthesis of safe view pose within the inspection module, FLIE incorporates a novel active visual guidance scheme for identifying previously inspected surfaces. Furthermore, the integration of a tiered hierarchical exploration module with the visual guidance system enables the UAV to navigate towards new and unexplored structures without relying on a map. This decoupling reduces memory overhead and computational effort by eliminating the need to plan based on an incrementally built, error-prone global map. The proposed autonomy is extensively evaluated through simulation and experimental verification under various scenarios and compared against state-of-art approaches, demonstrating its performance and effectiveness.
Article PDF
Similar content being viewed by others
Avoid common mistakes on your manuscript.
Availability of data and materials
ROSbags of sensor and submodule ouput data from the experiments can be made available at the suggestion of the reviewers and editors.
Code Availability
Not applicable
References
Viswanathan, V.K., Satpute, S., Banerjee, A., Nikolakopoulos, G.: Nonlinear model predictive control based cooperative stereo-visual coverage of an asteroid. In: 2022 American control conference (ACC), pp. 5360–5367 (2022). IEEE
Viswanathan, V.K., Papadimitriou, A., Banerjee, A., Mansouri, S.S., Nikolakopolous, G.: Exogenous disturbance estimation for autonomous navigation around small celestial bodies. In: 2022 IEEE 61st conference on decision and control (CDC), pp. 3760–3766 (2022). IEEE
Khaloo, A., Lattanzi, D., Jachimowicz, A., Devaney, C.: Utilizing uav and 3d computer vision for visual inspection of a large gravity dam. Front. Built Environ. 31 (2018)
Chan, B., Guan, H., Jo, J., Blumenstein, M.: Towards uav-based bridge inspection systems: A review and an application perspective. Struct. Monit. Maint. 2(3), 283–300 (2015)
Kanellakis, C., Mansouri, S.S., Fresk, E., Kominiak, D., Nikolakopoulos, G.: Cooperative uavs as a tool for aerial inspection of large scale aging infrastructure. In: 2018 IEEE/RSJ international conference on intelligent robots and systems (IROS), pp. 5040–5040 (2018). IEEE
Addabbo, P., Angrisano, A., Bernardi, M.L., Gagliarde, G., Mennella, A., Nisi, M., Ullo, S.L.: Uav system for photovoltaic plant inspection. IEEE Aerosp. Electron. Syst. Mag. 33(8), 58–67 (2018)
Zhang, Y., Yuan, X., Li, W., Chen, S.: Automatic power line inspection using uav images. Remote Sens. 9(8), 824 (2017)
Patel, A., Lindqvist, B., Kanellakis, C., Nikolakopoulos, G.: Fast planner for mav navigation in unknown environments based on adaptive search of safe look-ahead poses. In: 2022 30th Mediterranean conference on control and automation (MED), pp. 545–550 (2022). IEEE
Mansouri, S.S., Kanellakis, C., Fresk, E., Lindqvist, B., Kominiak, D., Koval, A., Sopasakis, P., Nikolakopoulos, G.: Subterranean mav navigation based on nonlinear mpc with collision avoidance constraints. IFAC-PapersOnLine 53(2), 9650–9657 (2020)
Dang, T., Tranzatto, M., Khattak, S., Mascarich, F., Alexis, K., Hutter, M.: Graph-based subterranean exploration path planning using aerial and legged robots. J. Field Robot. 37(8), 1363–1388 (2020). Wiley Online Library
Lindqvist, B., Kanellakis, C., Mansouri, S.S., Agha-mohammadi, A.-A., Nikolakopoulos, G.: Compra: A compact reactive autonomy framework for subterranean mav based search-and-rescue operations. J. Intell. Robot. Syst. 105(3), 1–21 (2022)
Kumar, A.: Real-time performance comparison of vision-based autonomous landing of quadcopter on a ground moving target. IETE J. Res. pp. 1–18 (2021)
Viswanathan, V.K., Satpute, S.G., Lindqvist, B., Kanellakis, C., Nikolakopoulos, G.: Experimental evaluation of a geometry-aware aerial visual inspection framework in a constrained environment. In: 2022 30th Mediterranean conference on control and automation (MED), pp. 468–474 (2022). IEEE
Kerle, N., Nex, F., Gerke, M., Duarte, D., Vetrivel, A.: Uav-based structural damage mapping: A review. ISPRS Int. J. Geo-inform. 9(1), 14 (2019)
Lauterbach, H.A., Koch, C.B., Hess, R., Eck, D., Schilling, K., Nüchter, A.: The eins3d project-instantaneous uav-based 3d mapping for search and rescue applications. In: 2019 IEEE international symposium on safety, security, and rescue robotics (SSRR), pp. 1–6 (2019). IEEE
Cheng, M.-L., Matsuoka, M., Liu, W., Yamazaki, F.: Near-real-time gradually expanding 3d land surface reconstruction in disaster areas by sequential drone imagery. Autom. Constr. 135, 104105 (2022)
Bircher, A., Kamel, M., Alexis, K., Oleynikova, H., Siegwart, R.: Receding horizon path planning for 3d exploration and surface inspection. Auton. Robot. 42(2), 291–306 (2018)
Brogaard, R.Y., Andersen, R.E., Kovac, L., Zajaczkowski, M., Boukas, E.: Towards an autonomous, visual inspection-aware 3d exploration and mapping system for water ballast tanks of marine vessels. In: 2021 IEEE international conference on imaging systems and techniques (IST), pp. 1–6 (2021). IEEE
Yoder, L., Scherer, S.: Autonomous exploration for infrastructure modeling with a micro aerial vehicle. In: Field and service robotics, pp. 427–440. Springer, (2016)
Faria, M., Ferreira, A.S., Pérez-Leon, H., Maza, I., Viguria, A.: Autonomous 3d exploration of large structures using an uav equipped with a 2d lidar. Sens. 19(22), 4849 (2019)
Song, S., Kim, D., Jo, S.: Online coverage and inspection planning for 3d modeling. Auton. Robot. 44(8), 1431–1450 (2020)
Yamauchi, B.: A frontier-based approach for autonomous exploration. In: Proceedings 1997 IEEE international symposium on computational intelligence in robotics and automation CIRA’97.’Towards new computational principles for robotics and automation’, pp. 146–151 (1997). IEEE
Song, S., Jo, S.: Surface-based exploration for autonomous 3d modeling. In: 2018 IEEE international conference on robotics and automation (ICRA), pp. 4319–4326 (2018). IEEE
Schmid, L., Pantic, M., Khanna, R., Ott, L., Siegwart, R., Nieto, J.: An efficient sampling-based method for online informative path planning in unknown environments. IEEE Robot. Autom. Lett. 5(2), 1500–1507 (2020)
Smith, N., Moehrle, N., Goesele, M., Heidrich, W.: Aerial path planning for urban scene reconstruction: A continuous optimization method and benchmark (2018)
Hepp, B., Nießner, M., Hilliges, O.: Plan3d: Viewpoint and trajectory optimization for aerial multi-view stereo reconstruction. ACM Trans. Graph. (TOG) 38(1), 1–17 (2018)
Song, S., Kim, D., Choi, S.: View path planning via online multiview stereo for 3-d modeling of large-scale structures. IEEE Trans. Robot. 38(1), 372–390 (2021)
Viswanathan, V.K., Satpute, S.G., Lindqvist, B., Nikolakopoulos, G.: First-look enabled autonomous aerial visual inspection of geometrically fractured objects in constrained environments. In: 2022 IEEE 31st international symposium on industrial electronics (ISIE), pp. 295–300 (2022). IEEE
Viswanathan, V.K., Satpute, S.G., Nikolakopoulos, G.: Flie: First-look enabled inspect-explore autonomy towards visual inspection of unknown distributed and discontinuous structures. IEEE Access (2023)
Lowe, D.G.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60, 91–110 (2004)
Muja, M., Lowe, D.G.: Fast approximate nearest neighbors with automatic algorithm configuration. VISAPP (1) 2(331-340), 2 (2009)
Lindqvist, B., Haluska, J., Kanellakis, C., Nikolakopoulos, G.: An adaptive 3d artificial potential field for fail-safe uav navigation. In: 2022 30th Mediterranean conference on control and automation (MED), pp. 362–367 (2022). IEEE
Furrer, F., Burri, M., Achtelik, M., Siegwart, R.: RotorS—A modular gazebo mav simulator framework. Robot operating system (ROS): The complete reference (Volume 1), pp. 595–625. Springer, Cham (2016)
Quigley, M., Conley, K., Gerkey, B., Faust, J., Foote, T., Leibs, J., Wheeler, R., Ng, A.Y., et al.: Ros: an open-source robot operating system. In: ICRA workshop on open source software, vol.3, p. 5. Kobe, Japan (2009)
Design and use paradigms for gazebo, an open-source multi-robot simulator. In: 2004 IEEE/RSJ international conference on intelligent robots and systems (IROS)(IEEE Cat. No. 04CH37566), vol. 3, pp. 2149–2154 (2004). IEEE
Labbé, M., Michaud, F.: Rtab-map as an open-source lidar and visual simultaneous localization and mapping library for large-scale and long-term online operation. J. Field Robot. 36(2), 416–446 (2019)
Oleynikova, H., Taylor, Z., Fehr, M., Siegwart, R., Nieto, J.: Voxblox: Incremental 3d euclidean signed distance fields for on-board mav planning. In: IEEE/RSJ international conference on intelligent robots and systems (IROS) (2017)
Zhou, B., Zhang, Y., Chen, X., Shen, S.: Fuel: Fast uav exploration using incremental frontier structure and hierarchical planning. IEEE Robot. Autom. Lett. 6(2), 779–786 (2021)
Selin, M., Tiger, M., Duberg, D., Heintz, F., Jensfelt, P.: Efficient autonomous exploration planning of large-scale 3-d environments. IEEE Robot. Autom. Lett. 4(2), 1699–1706 (2019). https://doi.org/10.1109/LRA.2019.2897343
Feng, C., Li, H., Gao, F., Zhou, B., Shen, S.: Predrecon: A prediction-boosted planning framework for fast and high-quality autonomous aerial reconstruction. arXiv preprint arXiv:2302.04488 (2023)
Epic Games: Unreal Engine. https://www.unrealengine.com (2023)
Shah, S., Dey, D., Lovett, C., Kapoor, A.: Airsim: High-fidelity visual and physical simulation for autonomous vehicles. In: Field and service robotics. arXiv:1705.05065 (2017)
Papadimitriou, A., Jafari, H., Mansouri, S.S., Nikolakopoulos, G.: External force estimation and disturbance rejection for micro aerial vehicles. Expert Syst. Appl. 200, 116883 (2022)
Acknowledgements
This work has been partially funded by the European Union’s Horizon 2020 Research and Innovation Programme under the Grant Agreement No. 101003591 NEXGEN SIMS. The authors acknowledge Ilias Tevetzidis contribution towards this work by assuming the role of safety pilot during experimental trials.
Funding
Open access funding provided by Lulea University of Technology. This work has been partially funded by the European Union’s Horizon 2020 Research and Innovation Programme under the Grant Agreement No. 101003591 NEXGEN SIMS.
Author information
Authors and Affiliations
Contributions
Vignesh Kottayam Viswanathan: Development of manuscript, primary research contributor and experimental work; Björn Lindqvist: Experimental work, manuscript contributions and advisory role; Sumeet Gajanan Satpute, Christoforos Kanellakis and George Nikolakopoulos: Manuscript contributions and advisory.
Corresponding author
Ethics declarations
Ethics approval
Not applicable
Consent to participate
All authors have read and agreed to publish this work.
Consent for publication
All authors have read and agreed to publish this work.
Conflict of interest
The authors declare that they have no conflict of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Viswanathan, V.K., Lindqvist, B., Satpute, S.G. et al. Towards Visual Inspection of Distributed and Irregular Structures: A Unified Autonomy Approach. J Intell Robot Syst 109, 32 (2023). https://doi.org/10.1007/s10846-023-01961-9
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s10846-023-01961-9