Skip to main content
SpringerLink
Log in
Book cover

International Conference on Runtime Verification

RV 2018: Runtime Verification pp 409–416Cite as

Evaluating Perception Systems for Autonomous Vehicles Using Quality Temporal Logic

Part of the Lecture Notes in Computer Science book series (LNPSE,volume 11237)

Abstract

For reliable situation awareness in autonomous vehicle applications, we need to develop robust and reliable image processing and machine learning algorithms. Currently, there is no general framework for reasoning about the performance of perception systems. This paper introduces Timed Quality Temporal Logic (TQTL) as a formal language for monitoring and testing the performance of object detection and situation awareness algorithms for autonomous vehicle applications. We demonstrate that it is possible to describe interesting properties as TQTL formulas and detect cases where the properties are violated.

Keywords

  • Temporal logic
  • Monitoring
  • Autonomous vehicles
  • Perception
  • Image processing
  • Machine Learning

This is a preview of subscription content, access via your institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   64.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   84.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

References

  1. Autonomous vehicle disengagement reports 2016. https://www.dmv.ca.gov/portal/dmv/detail/vr/autonomous/disengagement_report_2016

  2. IEEE Connected Vehicles: Google reports self-driving car disengagements. http://sites.ieee.org/connected-vehicles/2015/12/15/google-reports-self-driving-car-disengagements

  3. S-TaLiRo Toolbox. https://sites.google.com/a/asu.edu/s-taliro/s-taliro

  4. Alur, R., Henzinger, T.A.: A really temporal logic. J. ACM 41(1), 181–204 (1994)

    CrossRef  MathSciNet  Google Scholar 

  5. Annpureddy, Y., Liu, C., Fainekos, G., Sankaranarayanan, S.: S-TaLiRo: a tool for temporal logic falsification for hybrid systems. In: Abdulla, P.A., Leino, K.R.M. (eds.) TACAS 2011. LNCS, vol. 6605, pp. 254–257. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-19835-9_21

    CrossRef  MATH  Google Scholar 

  6. Bartocci, E., Deshmukh, J., Donzé, A., Fainekos, G., Maler, O., Ničković, D., Sankaranarayanan, S.: Specification-based monitoring of cyber-physical systems: a survey on theory, tools and applications. In: Bartocci, E., Falcone, Y. (eds.) Lectures on Runtime Verification - Introductory and Advanced Topics. LNCS, vol. 10457, pp. 135–175. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-75632-5_5

    CrossRef  Google Scholar 

  7. Deshmukh, J.V., Majumdar, R., Prabhu, V.S.: Quantifying conformance using the skorokhod metric. In: 27th International Conference on Computer Aided Verification (CAV), pp. 234–250 (2015)

    CrossRef  Google Scholar 

  8. Dokhanchi, A., Hoxha, B., Tuncali, C.E., Fainekos, G.: An efficient algorithm for monitoring practical TPTL specifications. In: The ACM/IEEE International Conference on Formal Methods and Models for System Design (MEMOCODE), pp. 184–193 (2016)

    Google Scholar 

  9. Dreossi, T., Ghosh, S., Sangiovanni-Vincentelli, A.L., Seshia, S.A.: Systematic testing of convolutional neural networks for autonomous driving. In: ICML Workshop on Reliable Machine Learning in the Wild (RMLW) (2017)

    Google Scholar 

  10. Fainekos, G., Pappas, G.J.: Robustness of temporal logic specifications for continuous-time signals. Theor. Comput. Sci. 410(42), 4262–4291 (2009)

    CrossRef  MathSciNet  Google Scholar 

  11. Geiger, A., Lenz, P., Stiller, C., Urtasun, R.: Vision meets robotics: the KITTI dataset. Int. J. Robot. Res. (IJRR) 32, 1229–1235 (2013)

    Google Scholar 

  12. Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT Press (2016). http://www.deeplearningbook.org

  13. Maler, O., Nickovic, D.: Monitoring temporal properties of continuous signals. In: Lakhnech, Y., Yovine, S. (eds.) FORMATS/FTRTFT -2004. LNCS, vol. 3253, pp. 152–166. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-30206-3_12

    CrossRef  MATH  Google Scholar 

  14. Nguyen, L.V., Kapinski, J., Jin, X., Deshmukh, J.V., Johnson, T.T.: Hyperproperties of real-valued signals. In: The ACM-IEEE International Conference on Formal Methods and Models for System Design (MEMOCODE), pp. 104–113 (2017)

    Google Scholar 

  15. Tuncali, C.E., Fainekos, G., Ito, H., Kapinski, J.: Simulation-based adversarial test generation for autonomous vehicles with machine learning components. In: IEEE Intelligent Vehicles Symposium (IV) (2018)

    Google Scholar 

  16. Wu, B., Iandola, F., Jin, P.H., Keutzer, K.: Squeezedet: Unified, small, low power fully convolutional neural networks for real-time object detection for autonomous driving (2016)

    Google Scholar 

  17. Xu, H., Gao, Y., Yu, F., Darrell, T.: End-to-end learning of driving models from large-scale video datasets. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017). https://doi.org/10.1109/CVPR.2017.376

Download references

Acknowledgements

This work was partially supported by the NSF I/UCRC Center for Embedded Systems and by NSF grants 1350420, 1361926 and 1446730.

Author information

Authors and Affiliations

  1. Arizona State University, Tempe, AZ, 85281, USA

    Adel Dokhanchi, Heni Ben Amor & Georgios Fainekos

  2. University of Southern California, Los Angeles, CA, 90089, USA

    Jyotirmoy V. Deshmukh

Authors
  1. Adel Dokhanchi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Heni Ben Amor
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jyotirmoy V. Deshmukh
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Georgios Fainekos
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Adel Dokhanchi .

Editor information

Editors and Affiliations

  1. University of Malta, Msida, Malta

    Christian Colombo

  2. University of Lübeck, Lübeck, Germany

    Prof. Dr. Martin Leucker

Rights and permissions

Reprints and Permissions

Copyright information

© 2018 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Dokhanchi, A., Amor, H.B., Deshmukh, J.V., Fainekos, G. (2018). Evaluating Perception Systems for Autonomous Vehicles Using Quality Temporal Logic. In: Colombo, C., Leucker, M. (eds) Runtime Verification. RV 2018. Lecture Notes in Computer Science(), vol 11237. Springer, Cham. https://doi.org/10.1007/978-3-030-03769-7_23

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-03769-7_23

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-03768-0

  • Online ISBN: 978-3-030-03769-7

  • eBook Packages: Computer ScienceComputer Science (R0)

search

Navigation