Skip to main content
SpringerLink
Log in

Three Dimensional Intruder Closest Point of Approach Estimation Based-on Monocular Image Parameters in Aircraft Sense and Avoid

Motto: ’Almost Everything from Almost Nothing’

  • Published:
Journal of Intelligent & Robotic Systems Aims and scope Submit manuscript

Abstract

The paper deals with monocular image-based sense and avoid assuming constant aircraft velocities and straight flight paths. From very limited two dimensional image information it finally characterizes the whole three dimensional collision situation by estimating the time to closest point of approach, the horizontal relative distance and its direction and the vertical relative distance also. The distances are relative to the intruder aircraft horizontal and vertical sizes. The overall estimated relative distance is the closest between the two aircraft in three dimension. So finally, every important information can be extracted to be used in a collision decision. The applicability of the developed method is presented in software-in-the-loop simulation test runs. Several intruder size and speed values are considered together with trajectories covering the whole three dimensional space. The horizontal intruder flight directions relative to the own aircraft cover 360∘ and the intruder can come from below ar above also. Detailed evaluation and discussion of the results is also included. Finally, the missed detection rate results to be superior (below 3% in every test scenario) though the false alarm rate results a bit high between 7–14%.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. airliners.net: aircraft-data. http://www.airliners.net/aircraft-data/ (2015)

  2. Bauer, P., Hiba, A.: Vision only collision detection with omnidirectional multi-camera system. In: Proc. of the 20th World Congress of the International Federation of Automatic Control, pp. 15,780–15,785. IFAC, Toulouse, France (2017)

  3. Bauer, P., Hiba, A., Bokor, J.: Monocular image-based intruder direction estimation at closest point of approach. In: Proc. of the International Conference on Unmanned Aircraft Systems (ICUAS) 2017, pp. 1108–1117, ICUAS Association, Miami, FL, USA (2017)

  4. Bauer, P., Hiba, A., Vanek, B., Zarandy, A., Bokor, J.: Monocular image-based time to collision and closest point of approach estimation. In: proceedings of 24th Mediterranean Conference on Control and Automation (MED’16). Athens, Greece (2016)

  5. Bauer, P., Vanek, B., Peni, T., Futaki, A., Pencz, B., Zarandy, A., Bokor, J.: Monocular image parameter-based aircraft sense and avoid. In: proceedings of 23rd Mediterranean Conference on Control and Automation (MED’15). Torremolinos, Spain (2015)

  6. Bauer, P., Vanek, B., Pni, T., Zsedrovits, T., Pencz, B., Zarndy, K., Bokor, J.: Aircraft trajectory tracking with large sideslip angles for sense and avoid intruder state estimation. In proceedings of 22nd Mediterranean Conference on Control and Automation (MED’14). Palermo, Italy (2014)

  7. Byrne, J., Taylor, C.J.: Expansion segmentation for visual collision detection and estimation. In: Proc. of IEEE International Conference on Robotics and Automation (2009)

  8. Choi, H., Kim, Y., Hwang, I.: Reactive collision avoidance of unmanned aerial vehicles using a single vision sensor. J. Guid. Control. Dyn. 36(4), 1234–1240 (2013). https://doi.org/10.2514/1.57131

    Article  Google Scholar 

  9. Degen, S.: Reactive image-based collision avoidance system for unmanned aircraft systems. Master’s thesis, Australian Research Centre for Aerospace Automation (2011)

  10. Dempsey, M.: U.s. army unmanned aircraft systems roadmap 2010-2035. Tech. rep., U.S. Army UAS Center of Excellence (2010)

  11. EU: Roadmap for the integration of civil Remotely-Piloted Aircraft Systems into the European Aviation System. Tech. rep., European RPAS Steering Group (2013)

  12. FAA: Federal Aviation Regulation 14 CFR Part 91. Federal Aviation Administration (FAA) (2016)

  13. Fasano, G., Accardo, D., Forlenza, L., Moccia, A., Rispoli, A.: A multi-sensor obstacle detection and tracking system for autonomousuav sense and avoid. In: XX Congresso Nazionale AIDAA, Milano (2009)

  14. Fasano, G., Accardo, D., Tirri, A.E., Moccia, A.: Feature article: Experimental analysis of onboard non-cooperative sense and avoid solutions based on radar, optical sensors, and data fusion. IEEE Aerosp. Electron. Syst. Mag. 31(7), 6–14 (2016). https://doi.org/10.1109/MAES.2016.150164

    Article  Google Scholar 

  15. Forlenza, L.: Vision based strategies for implementing Sense and Avoid capabilities onboard Unmanned Aerial Systems. Ph.D. thesis, UNIVERSITÁ DEGLI STUDI DI NAPOLI FEDERICO II (2012)

  16. Frew, E.W.: Observer trajectory generation for target-motion estimation using monocular vision. Ph.D. thesis, Stanford University (2003)

  17. Hutchings, T., Jeffryes, S., Farmer, S.J.: Architecting uav sense & avoid systems. In: Proc. Institution of Engineering and Technology Conf. Autonomous Systems, pp. 1–8 (2007)

  18. Jamoom, M.B., Joerger, M., Pervan, B.: Unmanned aircraft system sense-and-avoid integrity and continuity risk. J. Guid. Control. Dyn. 39(3), 498–509 (2015). https://doi.org/10.2514/1.G001468

    Article  Google Scholar 

  19. Lyu, Y., Pan, Q., Zhao, C., Zhang, Y., Hu, J.: Feature article: Vision-based UAV collision avoidance with 2D dynamic safety envelope. IEEE Aerosp. Electron. Syst. Mag. 31(7), 16–26 (2016). https://doi.org/10.1109/MAES.2016.150155

    Article  Google Scholar 

  20. Mejias, L., McFadyen, A., Ford, J.J.: Feature article: Sense and avoid technology developments at Queensland university of technology. IEEE Aerosp. Electron. Syst. Mag. 31(7), 28–37 (2016). https://doi.org/10.1109/MAES.2016.150157

    Article  Google Scholar 

  21. Melnyk, R., Schrage, D., Volovoi, V., Jimenez, H.: Sense and avoid requirements for unmanned aircraft systems using a target level of safety approach. Risk Anal. 34(10), 1894–1906 (2014)

    Article  Google Scholar 

  22. Meyer, F., Bouthemy, P.: Estimation of time-to-collision maps from first order motion models and normal flows. In: Proc. of 11th IAPR International Conference on Pattern Recognition (1992)

  23. Mori, T., Scherer, S.: First results in detecting and avoiding frontal obstacles from a monocular camera for micro unmanned aerial vehicles. In: International Conference on Robotics and Automation (2013)

  24. Nussberger, A., Grabner, H., Gool, L.V.: Feature article: Robust Aerial Object Tracking from an Airborne platform. IEEE Aerosp. Electron. Syst. Mag. 31(7), 38–46 (2016). https://doi.org/10.1109/MAES.2016.150126

    Article  Google Scholar 

  25. Shakernia, O., Chen, W.-Z., Raska, V.: Passive ranging for uav sense and avoid applications. In: fotech Aerospace (2005)

  26. Salazar, L.R., Sabatini, R., Ramasamy, S., Gardi, A.: A novel system for non-cooperative uav sense-and-avoid. In: Proceedings of european navigation conference 2013 (ENC 2013) (2013)

  27. Schaub, A., Burschka, D.: Spatio-temporal prediction of collision candidates for static and dynamic objects in monocular image sequences. In Proc. of IEEE Intelligent Vehicles Symposium (IV 2013) (2013)

  28. Speijker, L., Verstraeten, J., Kranenburg, C., van der Geest, P.: Scoping improvements to ’See And Avoid’ for general aviation (SISA). Tech. rep., European Aviation Safety Agency (EASA (2012)

  29. Vanek, B., Peni, T., Zarandy, A., Bokor, J., Zsedrovits, T., Roska, T.: Performance characteristics of a complete vision only sense and avoid system. In: Proceedings of AIAA GNC 2012 (Guidance, Navigation and Control Conference), AIAA 2012-4703, pp. 1–15, Minneapolis, Minnesota (2012)

  30. Watanabe, Y.: Stochastically optimized monocular vision-based navigation and guidance. Ph.D. thesis, Georgia Institute of Technology (2008)

  31. Zarandy, A., Nagy, Z., Vanek, B., Zsedrovits, T., Kiss, A., Nemeth, M.: A five-camera vision system for uav visual attitude calculation and collision warning. In: Computer Vision Systems, Lecture Notes in Computer Science, pp. 11–20. Saint Petersburg, Russia (2013)

  32. Zsedrovits, T., Bauer, P., Pencz, B.J.M., Hiba, A., Gőzse, I., Kisantal, M., Németh, M., Nagy, Z., Vanek, B., Zarándy, A., Bokor, J.: Onboard visual sense and avoid system for small aircraft. IEEE A&E Syst. Mag. 31, 18–27 (2016)

    Article  Google Scholar 

  33. Zsedrovits, T., Zarandy, A., Vanek, B., Peni, T., Bokor, J., Roska, T.: Collision avoidance for UAV using visual detection. In: Proceedings of IEEE ISCAS 2011 (International Symposium on Circuits and Systems), pp. 2173–2176. IEEE, Rio de Janeiro (2011)

    Book  Google Scholar 

Download references

Acknowledgments

The research leading to these results has received funding from the European Union’s Horizon 2020 research and innovation program under grant agreement No. 690811 and the Japan New Energy and Industrial Technology Development Organization under grant agreement No. 062600 as a part of the EU/Japan joint research project entitled ’Validation of Integrated Safety-enchanced Intelligent flight cONtrol (VISION)’ This work was also supported by the Institute for Computer Science and Control (SZTAKI) Grant Number 008. The authors greatly appreciate the work of the reviewers which helped to improve the overall quality of the paper.

Author information

Authors and Affiliations

  1. Systems and Control Laboratory, Institute for Computer Science and Control, Hungarian Academy of Sciences (MTA SZTAKI), H-1111, Budapest, Kende utca 13-17, Hungary

    Peter Bauer & Jozsef Bokor

  2. Computational Optical Sensing and Processing Laboratory, Institute for Computer Science and Control, Hungarian Academy of Sciences (MTA SZTAKI), H-1111, Budapest, Kende utca 13-17, Hungary

    Antal Hiba & Akos Zarandy

Authors
  1. Peter Bauer
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Antal Hiba
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jozsef Bokor
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Akos Zarandy
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Peter Bauer.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix A: Intruder Characterization Based-on Real Aircraft Data

Appendix A: Intruder Characterization Based-on Real Aircraft Data

The possible airspace categories where the proposed S&A system can be used are selected based on [21] which makes an important effort to set S&A system effectiveness standards considering different class of UAVs and airspaces. The targeted airspaces by current development are Class D/E and G which does not require on-board transponder or ATC link. The targeted own aircraft categories are Group 1 to 4 (micro to tactical). Considering the possible threats, in Class D/E airspaces the intruder aircraft can range from micro UAVs through general aviation (GA) aircraft until large airliners / transporters on their approach to airports. So, these types should be characterized. [21] characterizes aircraft based-on their weight and speed, however from a vision sensor point of view it is better to use size and speed.

Wingspan (b), fuselage length (L), height (H) and cruise speed characteristics were collected from [1] ranging from CAP-10 to Airbus A380 and AN-225 including also helicopters. Three characteristic diagrams were obtained. The first is the horizontal size-speed diagram in Fig. 1 which shows the cruise speeds of the aerial vehicles against their average (mean of wingspan (rotor diameter) and length) horizontal size (R). Minimum, mean and maximum size-cruise speed curves are fitted to the data (crosses in the plot) which can be used in the generation of intruder aircraft parameters in Monte-Carlo simulation tests.

Fig. 1
figure 1

Possible intruder aircraft sizes and related cruise speeds

Full size image

The second is the histogram for the wingspan (b) / length (L) ratios in Fig. 2. This shows that most of the ratios are close to 1 and so the horizontal disk geometrical model presented in Section ?? can be valid.

Fig. 2
figure 2

Histogram of wingspan (b) / length (L) ratios

Full size image

The third is the histogram for the height (H) / average size (R) ratios in Fig. 3. This shows that height / horizontal size can range from 0.2 to 0.4 and most of the data is around 0.3.

Fig. 3
figure 3

Histogram of height (H) / average size (R) ratios

Full size image

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bauer, P., Hiba, A., Bokor, J. et al. Three Dimensional Intruder Closest Point of Approach Estimation Based-on Monocular Image Parameters in Aircraft Sense and Avoid. J Intell Robot Syst 93, 261–276 (2019). https://doi.org/10.1007/s10846-018-0816-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10846-018-0816-6

Keywords

Mathematics Subject Classification (2010)

Search

Navigation