Abstract
In this paper, we propose a data-driven testing and evaluation framework for multi-UAVs to evaluate their performance in executing missions in the physical world. Seven micro-behaviors, termed here as modes of operation, are leveraged to describe the autonomous functionalities of the UAVs. These functionalities are then used to design five scenarios for model training, validation and testing of the proposed framework. Each scenario includes a distinct sequence of behaviors for the UAVs in order for the different autonomous functionalities to be evaluated. We develop and implement a simulation environment using the Robot Operating System (ROS), Gazebo, and the Pixhawk autopilot to generate synthetic data for the training of a classification model. This trained model is then utilized to evaluate the behaviors of the UAVs while performing real-world missions. Finally, the proposed framework is tested using synthetic data generated from a simulation environment and validated using real-world data.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Abadi, M., et al.: TensorFlow: a system for large-scale machine learning. In: 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 2016), pp. 265–283 (2016)
Chaki, S., Dolan, J.M., Giampapa, J.A.: Toward a quantitative method for assuring coordinated autonomy. In: Proceedings of ARMS Workshop (2013)
Chollet, F., et al.: Keras (2015)
Cowart, K., Valerdi, R., Kenley, C.R.: Development, validation and implementation considerations of a decision support system for unmanned & autonomous system of systems test & evaluation (2010)
Cui, Z., Ke, R., Pu, Z., Wang, Y.: Deep bidirectional and unidirectional LSTM recurrent neural network for network-wide traffic speed prediction. arXiv preprint arXiv:1801.02143 (2018)
Djang, P.A., Lopez, F.: Unmanned and autonomous systems mission based test and evaluation. In: Proceedings of the 9th Workshop on Performance Metrics for Intelligent Systems, pp. 81–85 (2009)
Gers, F.A., Schmidhuber, J., Cummins, F.: Learning to forget: continual prediction with LSTM (1999)
Girma, A., et al.: IoT-enabled autonomous system collaboration for disaster-area management. IEEE/CAA J. Automatica Sin. 7(5), 1249–1262 (2020)
Girma, A., Yan, X., Homaifar, A., Driver identification based on vehicle telematics data using LSTM-recurrent neural network. In: 2019 IEEE 31st International Conference on Tools with Artificial Intelligence (ICTAI), pp. 894–902. IEEE (2019)
Gonda, N.D.: A framework for test & evaluation of autonomous systems along the virtuality-reality spectrum. Master’s thesis, Old Dominion University, Norfolk, VA, USA (2019)
Graves, A., Schmidhuber, J.: Framewise phoneme classification with bidirectional LSTM and other neural network architectures. Neural Netw. 18(5–6), 602–610 (2005)
Greff, K., Srivastava, R.K., Koutník, J., Steunebrink, B.R., Schmidhuber, J.: LSTM: a search space odyssey. IEEE Trans. Neural Netw. Learn. Syst. 28(10), 2222–2232 (2016)
Holden, J., Goel, N.: Fast-forwarding to a future of on-demand urban air transportation, San Francisco, CA (2016)
Huang, Z., Xu, W., Yu, K.: Bidirectional LSTM-CRF models for sequence tagging. arXiv preprint arXiv:1508.01991 (2015)
Keane, J., Joiner, K.: Experimental test and evaluation of autonomous underwater vehicles. Aust. J. Multi-Discip. Eng. 16(1), 67–79 (2020)
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
Koopman, P., Wagner, M.: Challenges in autonomous vehicle testing and validation. SAE Int. J. Transp. Saf. 4(1), 15–24 (2016)
Leathrum, J.F., Mielke, R.R., Shen, Y., Johnson, H.: Academic/industry educational lab for simulation-based test & evaluation of autonomous vehicles. In: 2018 Winter Simulation Conference (WSC), pp. 4026–4037. IEEE (2018)
Michelson, R.C.: Test and evaluation for fully autonomous micro air vehicles. ITEA J. 29(4), 367–374 (2008)
NASA: Advanced air mobility studies/reports/presentations (2019)
Reitz, B.C., Wilkerson, J.L.: Test and evaluation of autonomous surface vehicles: a case study. In: 2020 IEEE/ION Position, Location and Navigation Symposium (PLANS), pp. 839–850. IEEE (2020)
Roske, V.P., Kohlberg, I., Wagner, R.: Autonomous systems challenges to test and evaluation. In: 28th Conference of National Defense Industrial Association (2012)
Sarkar, M., Homaifar, A., Erol, B.A., Behniapoor, M., Tunstel, E.: PIE: a tool for data-driven autonomous UAV flight testing. J. Intell. Robot. Syst. 98(2), 421–438 (2019). https://doi.org/10.1007/s10846-019-01078-y
Sun, Y., Xiong, G., Song, W., Gong, J., Chen, H.: Test and evaluation of autonomous ground vehicles. Adv. Mech. Eng. 6 (2014). https://doi.org/10.1155/2014/681326
Thompson, M.: Testing the intelligence of unmanned autonomous systems. Technical report, Trideum Corp., Aberdeen, MD (2008)
Shi, X., Chen, Z., Wang, H., Yeung, D.-Y., Wong, W.-K., Woo, W.: Convolutional LSTM network: a machine learning approach for precipitation nowcasting. In: Advances in Neural Information Processing Systems, pp. 802–810 (2015)
Acknowledgment
The authors would like to thank the Office of the Secretary of Defense (OSD) for the financial support under agreement number FA8750-15-2-0116. This work is also partially funded through the National Institute of Aerospace’s Langley Distinguished Professor Program under grant number C16-2B00-NCAT, and by the NASA University Leadership Initiative (ULI) under grant number 80NSSC20M0161. Also, this work is supported in parts by NSF under grant Nos. CAREER CPS-1851588, S&AS 1849198, and SATC-1801611.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Sarkar, M., Yan, X., Nateghi, S., Holmes, B.J., Vamvoudakis, K.G., Homaifar, A. (2022). A Framework for Testing and Evaluation of Operational Performance of Multi-UAV Systems. In: Arai, K. (eds) Intelligent Systems and Applications. IntelliSys 2021. Lecture Notes in Networks and Systems, vol 294. Springer, Cham. https://doi.org/10.1007/978-3-030-82193-7_24
Download citation
DOI: https://doi.org/10.1007/978-3-030-82193-7_24
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-82192-0
Online ISBN: 978-3-030-82193-7
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)