Skip to main content

Semantic Role Assisted Natural Language Rule Formalization for Intelligent Vehicle

  • Conference paper
  • First Online:
Rules and Reasoning (RuleML+RR 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14244))

Included in the following conference series:

  • 360 Accesses

Abstract

This paper proposes a novel pipeline to translate natural language rules and instructions for intelligent vehicles into temporal logic. The pipeline uses semantic role labeling (SRL), soft rule-based selection restrictions, and large language models (LLMs) to extract predicates, arguments, and temporal aspects from natural language rules and instruction. We then use the language understanding capability of LLMs to generate temporal logic rules from unstructured natural language text and additional information provided by SRL. We envision our model as a human-in-the-loop system that can facilitate the automated rule formalization for planning and verification systems in automated driving and drone planning. We demonstrate that our method can generate semantically correct temporal logic formulas from natural language text and provide implicit explanations of the output by showing the intermediate reasoning steps involved. This paper illustrates the integration of additional semantic knowledge and LLM and its application for the intelligent system domain of automated driving and drone planning. Our generalizable pipeline can easily extend to new logic formalization types, traffic rules, drone planning instructions, and application domains.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 49.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 64.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://www.gesetze-im-internet.de/stvo_2013/.

  2. 2.

    https://unece.org/DAM/trans/conventn/crt1968e.pdf.

  3. 3.

    https://github.com/kumarmanas/TR2MTL.

  4. 4.

    https://germanlawarchive.iuscomp.org/?p=1290.

  5. 5.

    https://github.com/mvcisback/py-metric-temporal-logic.

  6. 6.

    https://github.com/whitemech/LTLf2DFA.

  7. 7.

    https://platform.openai.com/docs/models/gpt-3-5.

  8. 8.

    https://platform.openai.com/docs/models/codex.

References

  1. Aréchiga, N.: Specifying safety of autonomous vehicles in signal temporal logic. In: 2019 IEEE Intelligent Vehicles Symposium (IV), pp. 58–63 (2019). https://doi.org/10.1109/IVS.2019.8813875

  2. Brown, T., et al.: Language models are few-shot learners. In: Advances in Neural Information Processing Systems, vol. 33, pp. 1877–1901 (2020)

    Google Scholar 

  3. Esterle, K., Aravantinos, V., Knoll, A.: From specifications to behavior: maneuver verification in a semantic state space. In: 2019 IEEE Intelligent Vehicles Symposium (IV), pp. 2140–2147 (2019). https://doi.org/10.1109/IVS.2019.8814241

  4. Fuggitti, F., Chakraborti, T.: NL2LTL - a python package for converting natural language (NL) instructions to linear temporal logic (LTL) formulas. In: AAAI (2023). system Demonstration

    Google Scholar 

  5. Gavran, I., Darulova, E., Majumdar, R.: Interactive synthesis of temporal specifications from examples and natural language. Proc. ACM Program. Lang. 4, 1–26 (2020). https://doi.org/10.1145/3428269

    Article  Google Scholar 

  6. Gung, J., Palmer, M.: Predicate representations and polysemy in VerbNet semantic parsing. In: International Conference on Computational Semantics (2021)

    Google Scholar 

  7. Hahn, C., Schmitt, F., Tillman, J.J., Metzger, N., Siber, J., Finkbeiner, B.: Formal specifications from natural language. arXiv:abs/2206.01962 (2022)

  8. He, J., Bartocci, E., Ničković, D., Isakovic, H., Grosu, R.: DeepSTL - from English requirements to signal temporal logic. In: 2022 IEEE/ACM 44th International Conference on Software Engineering (ICSE), pp. 610–622 (2022). https://doi.org/10.1145/3510003.3510171

  9. Hekmatnejad, M., et al.: Encoding and monitoring responsibility sensitive safety rules for automated vehicles in signal temporal logic. In: Proceedings of the 17th ACM-IEEE International Conference on Formal Methods and Models for System Design. Association for Computing Machinery (2019). https://doi.org/10.1145/3359986.3361203

  10. Huang, W., Abbeel, P., Pathak, D., Mordatch, I.: Language models as zero-shot planners: extracting actionable knowledge for embodied agents. arXiv preprint: arXiv:2201.07207 (2022)

  11. Karimi, A., Duggirala, P.S.: Formalizing traffic rules for uncontrolled intersections. In: 2020 ACM/IEEE 11th International Conference on Cyber-Physical Systems (ICCPS), pp. 41–50. IEEE (2020). https://doi.org/10.1109/ICCPS48487.2020.00012

  12. Karlsson, J., Tumova, J.: Intention-aware motion planning with road rules. In: 2020 IEEE 16th International Conference on Automation Science and Engineering (CASE), pp. 526–532 (2020). https://doi.org/10.1109/CASE48305.2020.9217037

  13. Konrad, S., Cheng, B.: Real-time specification patterns. In: Proceedings 27th International Conference on Software Engineering, 2005. ICSE 2005, pp. 372–381 (2005). https://doi.org/10.1109/ICSE.2005.1553580

  14. Li, R., et al.: StarCoder: may the source be with you! arXiv:abs/2305.06161 (2023)

  15. Li, X., et al.: Differentiable logic layer for rule guided trajectory prediction. In: Conference on Robot Learning (2020)

    Google Scholar 

  16. Lin, J., et al.: Road traffic law adaptive decision-making for self-driving vehicles. In: 2022 IEEE 25th International Conference on Intelligent Transportation Systems (ITSC), pp. 2034–2041 (2022). https://doi.org/10.1109/ITSC55140.2022.9922208

  17. Liu, J.X., et al.: Lang2LTL: translating natural language commands to temporal specification with large language models. In: CoRL Workshop on Language and Robot Learning (2022)

    Google Scholar 

  18. Liu, J., et al.: Generated knowledge prompting for commonsense reasoning. In: Annual Meeting of the Association for Computational Linguistics (2021)

    Google Scholar 

  19. Maierhofer, S., Moosbrugger, P., Althoff, M.: Formalization of intersection traffic rules in temporal logic. In: 2022 IEEE Intelligent Vehicles Symposium (IV), pp. 1135–1144 (2022). https://doi.org/10.1109/IV51971.2022.9827153

  20. Maierhofer, S., Rettinger, A.K., Mayer, E.C., Althoff, M.: Formalization of interstate traffic rules in temporal logic. In: 2020 IEEE Intelligent Vehicles Symposium (IV), pp. 752–759 (2020). https://doi.org/10.1109/IV47402.2020.9304549

  21. Muennighoff, N., et al.: Crosslingual generalization through multitask finetuning. arXiv:abs/2211.01786 (2022)

  22. Oh, Y., Patel, R., Nguyen, T., Huang, B., Pavlick, E., Tellex, S.: Planning with state abstractions for non-markovian task specifications. arXiv:abs/1905.12096 (2019)

  23. Palmer, M., Gildea, D., Kingsbury, P.: The proposition bank: an annotated corpus of semantic roles. Comput. Linguist. 31(1), 71–106 (2005). https://doi.org/10.1162/0891201053630264

    Article  Google Scholar 

  24. Palmer, M., Gildea, D., Xue, N.: Semantic Role Labeling. Synthesis Lectures on Human Language Technologies. Springer International Publishing, Cham (2010). https://link.springer.com/10.1007/978-3-031-02135-0

  25. Pan, J., Chou, G., Berenson, D.: Data-efficient learning of natural language to linear temporal logic translators for robot task specification. arXiv e-prints: arXiv:2303.08006 (2023)

  26. Ren, X., Yin, X., Li, S.: Synthesis of controllers for co-safe linear temporal logic specifications using reinforcement learning. In: 2021 40th Chinese Control Conference (CCC), pp. 2304–2309 (2021). https://doi.org/10.23919/CCC52363.2021.9549746

  27. Rizaldi, A., Immler, F., Althoff, M.: A formally verified checker of the safe distance traffic rules for autonomous vehicles. In: Rayadurgam, S., Tkachuk, O. (eds.) NFM 2016. LNCS, vol. 9690, pp. 175–190. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-40648-0_14

    Chapter  Google Scholar 

  28. Rizaldi, A., et al.: Formalising and monitoring traffic rules for autonomous vehicles in Isabelle/HOL. In: Polikarpova, N., Schneider, S. (eds.) IFM 2017. LNCS, vol. 10510, pp. 50–66. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-66845-1_4

    Chapter  Google Scholar 

  29. Schuler, K.K., Palmer, M.S.: VerbNet: a broad-coverage, comprehensive verb Lexicon. Ph.D. thesis, University of Pennsylvania, USA (2005)

    Google Scholar 

  30. Shi, P., Lin, J.: Simple BERT models for relation extraction and semantic role labeling. arXiv preprint: arXiv:1904.05255 (2019)

  31. Thati, P., Roşu, G.: Monitoring Algorithms for metric temporal logic specifications. Electr. Notes Theor. Comput. Sci. 113, 145–162 (2005). https://doi.org/10.1016/j.entcs.2004.01.029

    Article  Google Scholar 

  32. Vaswani, A., et al.: Attention is all you need. In: Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, pp. 6000–6010 (2017)

    Google Scholar 

  33. Wei, J., et al.: Chain-of-thought prompting elicits reasoning in large language models. In: Advances in Neural Information Processing Systems (2022)

    Google Scholar 

  34. Yao, S., et al.: ReAct: synergizing reasoning and acting in language models. In: International Conference on Learning Representations (ICLR) (2023)

    Google Scholar 

  35. Zhang, Q., Hong, D.K., Zhang, Z., Chen, Q.A., Mahlke, S., Mao, Z.M.: A systematic framework to identify violations of scenario-dependent driving rules in autonomous vehicle software. Proc. ACM Meas. Anal. Comput. Syst. 5(2), 1–25 (2021). https://doi.org/10.1145/3460082

    Article  Google Scholar 

Download references

Acknoledgement

This work is partially funded by German Federal Ministry for Economic Affairs and Climate Action within the “KI Wissen” project.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Kumar Manas .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Manas, K., Paschke, A. (2023). Semantic Role Assisted Natural Language Rule Formalization for Intelligent Vehicle. In: Fensel, A., Ozaki, A., Roman, D., Soylu, A. (eds) Rules and Reasoning. RuleML+RR 2023. Lecture Notes in Computer Science, vol 14244. Springer, Cham. https://doi.org/10.1007/978-3-031-45072-3_13

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-45072-3_13

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-45071-6

  • Online ISBN: 978-3-031-45072-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics