Verification and validation meet planning and scheduling

  • Saddek Bensalem
  • Klaus Havelund
  • Andrea Orlandini
Introduction

Abstract

A planning and scheduling (P&S) system takes as input a domain model and a goal, and produces a plan of actions to be executed, which will achieve the goal. A P&S system typically also offers plan execution and monitoring engines. Due to the non-deterministic nature of planning problems, it is a challenge to construct correct and reliable P&S systems, including, for example, declarative domain models. Verification and validation (V&V) techniques have been applied to address these issues. Furthermore, V&V systems have been applied to actually perform planning, and conversely, P&S systems have been applied to perform V&V of more traditional software. This article overviews some of the literature on the fruitful interaction between V&V and P&S.

Keywords

Verification and validation Planning and scheduling  Model checking Theorem proving Testing Monitoring 

References

  1. 1.
    Mars Science Laboratory (MSL) mission website. http://mars.jpl.nasa.gov/msl. Accessed Nov 2013
  2. 2.
    VVPS 2005 workshop website. http://icaps05.uni-ulm.de/workshops.html
  3. 3.
    VVPS 2009 workshop website. http://www-vvps09.imag.fr
  4. 4.
  5. 5.
    Abdedaim, Y., Asarin, E., Gallien, M., Ingrand, F., Lesire, C., Sighireanu, M.: Planning robust temporal plans: a comparison between CBTP and TGA approaches. In: Proceedings of the Seventeenth International Conference on Automated Planning and Scheduling (ICAPS’07), pp. 2–10 (2007)Google Scholar
  6. 6.
    Albarghouthi, A., Baier, J.A., McIlraith, S.A.: On the use of planning technology for verification. In: Proceedings of the ICAPS Workshop on Verification and Validation of Planning and Scheduling Systems (VVPS’09) (2009)Google Scholar
  7. 7.
    Artho, C., Barringer, H., Goldberg, A., Havelund, K., Khurshid, S., Lowry, M., Pasareanu, C., Rosu, G., Sen, K., Visser, W., Washington, R.: Combining test-case generation and runtime verification. Theor. Comput. Sci. 336(2–3), 209–234 (2005)CrossRefMATHMathSciNetGoogle Scholar
  8. 8.
    Artho, C., Havelund, K., Biere. A.: High-level data races. Softw. Test. Verif. Reliab. 13(4), 207–227 (2004)Google Scholar
  9. 9.
    Bakera, M., Edelkamp, S., Kissmann, P., Renner, C.D.: Solving \(\mu \)-calculus parity games by symbolic planning. In: Peled. D., Wooldridge, M. (eds.) MoChArt. Lecture Notes in Computer Science, vol. 5348, pp. 15–33. Springer, New York (2008)Google Scholar
  10. 10.
    Barringer, H., Groce, A., Havelund, K., Smith, M.: Formal analysis of log files. J. Aerosp. Comput. Inf. Commun. 7(11), 365–390 (2010)CrossRefGoogle Scholar
  11. 11.
    Barringer, H., Havelund, K., Kurklu, E., Morris, R.: Checking flight rules with TraceContract: application of a Scala DSL for trace analysis. In: Scala Days 2011. Stanford University, California (2011)Google Scholar
  12. 12.
    Behrmann, G., Cougnard, A., David, A., Fleury, E., Larsen, K., Lime, D.: UPPAAL-TIGA: time for playing games! In: Proceedings of 19th International Conference on Computer Aided Verification (CAV’07), vol. 4590 in LNCS, pp. 121–125. Springer, New York (2007)Google Scholar
  13. 13.
    Behrmann, G., Fehnker, A., Hune, T., Larsen, K., Pettersson, P., Romijn, J.: Efficient Guiding Towards Cost-Optimality in UPPAAL. Springer, New York (2001)Google Scholar
  14. 14.
    Behrmann, G., Larsen, K.G., Rasmussen, J.I.: Optimal scheduling using priced timed automata. In: Proceedings of the ICAPS Workshop on Verification and Validation of Model-Based Planning and Scheduling Systems (VVPS’05) (2005)Google Scholar
  15. 15.
    Bensalem, S., Bozga, M., Krichen, M., Tripakis, S.: Testing conformance of real-time applications: case of planetary rover controller. In: Proceedings of the ICAPS Workshop on Verification and Validation of Model-Based Planning and Scheduling Systems (VVPS’05), pp. 23–32 (2005)Google Scholar
  16. 16.
    Bodik, R., Jobstmann, B.: Algorithmic program synthesis: introduction. Int. J. Softw. Tools Technol. Transf. STTT 15(5–6), 397–411 (October 2013)Google Scholar
  17. 17.
    Bozzano, M., Cimatti, A., Roveri, M., Tchaltsev, A.: A comprehensive approach to on-board autonomy verification and validation. In: Proceedings of the ICAPS Workshop on Verification and Validation of Planning and Scheduling Systems (VVPS’09) (2009)Google Scholar
  18. 18.
    Brat, G., Drusinsky, D., Giannakopoulou, D., Goldberg, A., Havelund, K., Lowry, M., Pasareanu, C., Visser, W., Washington, R.: Experimental evaluation of verification and validation tools on Martian rover software. Form. Methods Syst. Des. 25(2), 167–198 (2004)Google Scholar
  19. 19.
    Brat, G., Gannakopoulou, D., Izygon, M., Alex, E., Wang, L., Frank, J., Molin, A.: Model-based verification and validation for procedure authoring. In: Proceedings of the ICAPS Workshop on Verification and Validation of Planning and Scheduling Systems (VVPS’09) (2009)Google Scholar
  20. 20.
    Cerrito, S., Mayer, M.C.: Using linear temporal logic to model and solve planning problems. In: Giunchiglia, F (ed.) Artificial Intelligence: Methodology, Systems, and Applications. Lecture Notes in Computer Science, vol 1480, pp. 141–152. Springer, New York (1998)Google Scholar
  21. 21.
    Cesta, A., Finzi, A., Fratini, S., Orlandini, A., Tronci, E.: Verifying flexible timeline-based plans. In: Proceedings of the ICAPS Workshop on Verification and Validation of Planning and Scheduling Systems (VVPS’09) (2009)Google Scholar
  22. 22.
    Cesta, A., Finzi, A., Fratini, S., Orlandini, A., Tronci, E.: Flexible plan verification: feasibility results. Fundamenta Informaticae 107, 111–137 (2011)MATHMathSciNetGoogle Scholar
  23. 23.
    Cichy, B., Chien, S., Schaffer, S., Tran, D., Rabideau, G., Sherwood, R.: Validating the autonomous EO-1 science agent. In: Proceedings of the ICAPS Workshop on Verification and Validation of Model-Based Planning and Scheduling Systems (VVPS’05), pp. 75–85 (2005)Google Scholar
  24. 24.
    Cimatti, A., Giunchiglia, F., Giunchiglia, E., Traverso, P.: Planning via model checking: a decision procedure for AR. In: Steel, S., Alami, R. (eds.) ECP. Lecture Notes in Computer Science, vol. 1348, pp. 130–142. Springer, New York (1997)Google Scholar
  25. 25.
    Cimatti, A., Roveri, M., Traverso, P.: Strong planning in non-deterministic domains via model checking. In: Simmons, R.G., Veloso, M.M., Smith, S.F. (eds.) AIPS, pp. 36–43. AAAI, Pittsburgh (1998)Google Scholar
  26. 26.
    Crampton, J., Huth, M., Kuo, J.H.-P.: Authorized workflow schemas : deciding realizability through LTL(F) model checking. Int. J. Softw. Tools Technol. Transf. STTT (2014)Google Scholar
  27. 27.
    Dierks, H.: Finding optimal plans for domains with restricted continuous effects with UPPAAL-CORA. In: Proceedings of the ICAPS Workshop on Verification and Validation of Model-Based Planning and Scheduling Systems (VVPS’05) (2005)Google Scholar
  28. 28.
    Dixon, L., Smaill, A., Bundy, A.: Verified planning by deductive synthesis in intuitionistic linear logic. In: Proceedings of the ICAPS Workshop on Verification and Validation of Planning and Scheduling Systems (VVPS’09) (2009)Google Scholar
  29. 29.
    Edelkamp, S.: Promela planning. In: Ball, T., Rajamani, S.K. (eds.) SPIN. Lecture Notes in Computer Science, vol. 2648, pp. 197–212. Springer, New York (2003)Google Scholar
  30. 30.
    Edelkamp, S., Jabbar, S.: Action planning for directed model checking of Petri nets. Electr. Notes Theor. Comput. Sci. 149(2), 3–18 (2006)CrossRefGoogle Scholar
  31. 31.
    Edelkamp, S., Jabbar, S., Lluch-Lafuente, A.: Cost-algebraic heuristic search. In: Veloso, M.M., Kambhampati, S. (eds.) AAAI, pp. 1362–1367. AAAI Press/The MIT Press, Cambridge (2005)Google Scholar
  32. 32.
    Edelkamp, S., Kellershoff, M., Sulewski, D.: Program model checking via action planning. In: van der Meyden, R., Smaus, J.-G. (eds.) MoChArt. Lecture Notes in Computer Science, vol. 6572, pp. 32–51. Springer, New York (2010)Google Scholar
  33. 33.
    Feather, M., Smith, B.: Automatic generation of test oracles—from pilot studies to application. Autom. Softw. Eng. 8(1), 31–61 (2001)CrossRefMATHGoogle Scholar
  34. 34.
    Fox, M., Howey, R., Long, D.: Exploration of the robustness of plans. In: Proceedings of the ICAPS Workshop on Verification and Validation of Model-Based Planning and Scheduling Systems (VVPS’05), pp. 67–74 (2005)Google Scholar
  35. 35.
    Fox, M., Long, D., Baldwin, L., Wilson, G., Woods, M., Jameux, D., Aylett, R.: On-board timeline validation and repair: a feasibility study. In: Proceedings of 5th International Workshop on Planning and Scheduling for Space (IWPSS’06) (2006)Google Scholar
  36. 36.
    Freitag, B., Margaria, T., Steffen, B.: A pragmatic approach to software synthesis. In: Wing, J.M., Wexelblat, R.L. (eds.) Workshop on Interface Definition Languages, Portland, Oregon, USA, pp. 46–58. ACM Press, New York (1994)Google Scholar
  37. 37.
    Giannakopoulou, D., Pasareanu, C.S., Lowry, M., Washington, R.: Lifecycle verification of the NASA Ames K9 rover executive. In: Proceedings of the ICAPS Workshop on Verification and Validation of Model-Based Planning and Scheduling Systems (VVPS’05), pp. 75–85 (2005)Google Scholar
  38. 38.
    Giunchiglia, F., Traverso, P.: Planning as model checking. In: Biundo, S., Fox, M. (eds.) ECP. Lecture Notes in Computer Science, vol. 1809, pp. 1–20. Springer, New York (1999)Google Scholar
  39. 39.
    Goldberg, A., Havelund, K., McGann, C.: Runtime verification for autonomous spacecraft software. In: Proceedings of IEEE Aerospace Conference, IEEE Computer Society, USA (2005)Google Scholar
  40. 40.
    Goldman, R.P., Kuter, U., Schneider, A.: Using classical planners for plan verification and counterexample generation. In: Proceedings of AAAI Workshop on Problem Solving Using Classical, Planning (2012)Google Scholar
  41. 41.
    Goldman, R.P., Musliner, D.J., Pelican, M.J.: Exploiting implicit representations in timed automaton verification for controller synthesis. In: Proceedings of the Fifth International Workshop on Hybrid Systems: Computation and Control (HSCC’02) (2002)Google Scholar
  42. 42.
    Goldman, R.P., Pelican, M.J., Musliner, D.J.: A loop acceleration technique to speed up verification of automatically-generated plans. Int. J. Softw. Tools Technol. Transf. STTT (2014)Google Scholar
  43. 43.
    Havelund, K., Groce, A., Holzmann, G., Joshi, R., Smith, M.: Automated testing of planning models. In: Proceedings of the Fifth International Workshop on Model Checking and Artificial Intelligence, pp. 90–105 (2008)Google Scholar
  44. 44.
    Havelund, K., Lowry, M., Park, S., Pecheur, C., Penix, J., Visser, W., White, J.L.: Formal analysis of the Remote Agent—before and after flight. In: The Fifth NASA Langley Formal Methods Workshop, Virginia (2001)Google Scholar
  45. 45.
    Havelund, K., Lowry, M., Penix, J.: Formal analysis of a spacecraft controller using SPIN. IEEE Trans. Softw. Eng. 27(8), 749–765 (2001). An earlier version occurred in the proceedings of 4th SPIN, workshop, 1998CrossRefGoogle Scholar
  46. 46.
    Howey, R., Long, D.: VAL’s progress: the automatic validation tool for PDDL2.1 used in the international planning competition. In: Proceedings of the ICAPS Workshop on The Competition: Impact, Organization, Evaluation, Benchmarks, pp. 28–37, Trento (2003)Google Scholar
  47. 47.
    Johnston, M.D., Tran, D.: Verification and validation of a deep space network scheduling application. In: Proceedings of the ICAPS Workshop on Verification and Validation of Planning and Scheduling Systems (VVPS’09) (2009)Google Scholar
  48. 48.
    Jonsson, A., Morris, P., Muscettola, N., Rajan, K., Smith, B.: Planning in interplanetary space: theory and practice. In: Proceedings of the Fifth International Conference on Artificial Intelligence Planning and Scheduling (AIPS’00), pp. 177–186 (2000)Google Scholar
  49. 49.
    Kautz, H., Selman, B.: BLACKBOX: a new approach to the application of theorem proving to problem solving. In: AIPS’98 Workshop on Planning as Combinatorial, Search, pp. 58–60 (1998)Google Scholar
  50. 50.
    Kautz, H., Selman, B.: Unifying sat-based and graph-based planning. In: Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI-99), vol. 99, pp. 318–325 (1999)Google Scholar
  51. 51.
    Kautz, H.A., Selman, B., et al.: Planning as satisfiability. In: Proceedings of the International European Conference on Artificial Intelligence (ECAI-92), vol. 92, pp. 359–363 (1992)Google Scholar
  52. 52.
    Khatib, L., Muscettola, N., Havelund, K.: Verification of plan models using UPPAAL. In: First International Workshop on Formal Approaches to Agent-Based Systems, NASA’s Goddard Space center, Maryland. Lecture Notes in Artificial Intelligence, vol. 1871. Springer, New York (2000)Google Scholar
  53. 53.
    Khatib, L., Muscettola, N., Havelund, K.: Mapping temporal planning constraints into timed automata. In: The Eigth International Symposium on Temporal Representation and Reasoning (TIME’01), pp. 21–27 (2001)Google Scholar
  54. 54.
    Kupferman, O., Vardi, M.Y.: Safraless decision procedures. In: 46th Annual IEEE Symposium on Foundations of Computer Science (FOCS’05), Pittsburgh, pp. 531–542 (2005)Google Scholar
  55. 55.
    Lindsey, T., Pecheur, C.: Simulation-based verification of autonomous controllers with Livingstone PathFinder. In: Proceedings of the 10th International Conference on Tools and Algorithms for the Construction and Analysis of Systems (TACAS’04), Barcelona, Spain. Lecture Notes in Computer Science, vol. 2988 (2004)Google Scholar
  56. 56.
    Long, D., Fox, M., Howey, R.: Planning domains and plans: validation, verification and analysis. In: Proceedings of the ICAPS Workshop on Verification and Validation of Planning and Scheduling Systems (VVPS’09) (2009)Google Scholar
  57. 57.
    Lowry, M.R., Havelund, K., Penix, J.: Verification and validation of AI systems that control deep-space spacecraft. In: Foundations of Intelligent Systems, Proceedings of 10th International Symposium, ISMIS’97, Charlotte. Lecture Notes in Computer Science, vol. 1325, pp. 35–47. Springer, New York (1997)Google Scholar
  58. 58.
    Mayer, M.C., Limongelli, C., Orlandini, A., Poggioni, V.: Linear temporal logic as an executable semantics for planning languages. J. Log. Lang. Inf. 16(1), 63–89 (2007)CrossRefMATHMathSciNetGoogle Scholar
  59. 59.
    McCarthy, J., Hayes, P.: Some Philosophical Problems from the Standpoint of Artificial Intelligence. Stanford University, USA (1968)Google Scholar
  60. 60.
    McDermott, D., Ghallab, M., Howe, A., Knoblock, C., Ram, A., Veloso, M., Weld, D., Wilkins, D.: PDDL-the planning domain definition language. Technical Report CVC TR98003/DCS TR1165, New Haven, CT. Yale Center for Computational Vision and Control (1998)Google Scholar
  61. 61.
    Menzies, T., Pecheur, C.: Verification and validation and artificial intelligence. Adv. Comput. 65, 5–45 (2005)Google Scholar
  62. 62.
    Morris, P.H., Muscettola, N.: Temporal dynamic controllability revisited. In: Proceedings of the 20th National Conference on Artificial Intelligence (AAAI-05) (2005)Google Scholar
  63. 63.
    Muscettola, N.: HSTS: integrating planning and scheduling. In: Zweben, M., Fox, M.S. (eds.) Intelligent Scheduling. Morgan Kauffmann, Burlington (1994)Google Scholar
  64. 64.
    Musliner, D.J., Pelican, M.J.S., Schlette, P.J.: Verifying equivalence of procedures in different languages: preliminary results. In: Proceedings of the ICAPS Workshop on Verification and Validation of Planning and Scheduling Systems (VVPS’09) (2009)Google Scholar
  65. 65.
    Nayak, P.P., Bernard, D.E., Dorais, G., Gamble, E.B., Kanefsky, B., Kurien, J., Millar, W., Muscettola, N., Rajan, K., Rouquette, N., Smith, B.D., Taylor, W.: Validating the DS1 Remote Agent experiment. In: Proceeedings of the Fifth International Symposium on Artificial Intelligence, Robotics and Automation in Space (iSAIRAS’99) (1999)Google Scholar
  66. 66.
    Orlandini, A., Finzi, A., Cesta, A., Fratini, S.: TGA-based controllers for flexible plan execution. In: Advances in Artificial Intelligence (KI 2011), 34th Annual German Conference on AI. Lecture Notes in Computer Science, vol. 7006, pp. 233–245. Springer, New York (2011)Google Scholar
  67. 67.
    Patron, P., Birch, A.: Plan proximity: an enhanced metric for plan stability. In: Proceedings of the ICAPS Workshop on Verification and Validation of Planning and Scheduling Systems (VVPS’09) (2009)Google Scholar
  68. 68.
    Penix, J., Pecheur, C., Havelund, K.: Using model checking to validate AI planner domain models. In: Proceedings of the 23\(^{rd}\) Annual Software Engineering, Workshop (1998)Google Scholar
  69. 69.
    Piterman, N., Pnueli, A., Sa’ar, Y.: Synthesis of reactive(1) designs. In: 7th International Conference on Verification, Model Checking and Abstract Interpretation (VMCAI’06). Lecture Notes in Computer Science, vol. 3855, pp. 364–380. Springer, New York (2006)Google Scholar
  70. 70.
    Pnueli, A.: The temporal logic of programs. In: 18th Annual Symposium on Foundations of Computer Science, pp. 46–57. IEEE Computer Society, USA (1977)Google Scholar
  71. 71.
    Pnueli, A., Rosner, R.: On the synthesis of a reactive module. In: Symposium on Principles of Programming Languages (POPL’89), pp. 179–190 (1989)Google Scholar
  72. 72.
    Preece, A.: Evaluating verification and validation methods in knowledge engineering. In: Roy, R. (ed.) Micro-Level Knowledge Management, pp. 123–145. Morgan-Kaufman, Burlington (2001)Google Scholar
  73. 73.
    R-Moreno, M.D., Brat, G., Muscettola, N., Rijsman, D.: Validation of a multi-agent architecture for planning and execution. In: Proceedings of 18th International Workshop on Principles of Diagnosis (DX’07) (2007)Google Scholar
  74. 74.
    Raimondi, F., Pecheur, C., Brat, G.: PDVer, a tool to verify PDDL planning domains. In: Proceedings of the ICAPS Workshop on Verification and Validation of Planning and Scheduling Systems (VVPS’09) (2009)Google Scholar
  75. 75.
    Razavi, N., Farzan, A., McIlraith, S.A.: Generating effective tests for concurrent programs via AI automated planning techniques. Int. J. Softw. Tools Technol. Transf. STTT (2014)Google Scholar
  76. 76.
    Shah, M., Chrpa, L., Jimoh, F., Kitchin, D., McCluskey, T., Parkinson, S., Vallati, M.: Knowledge engineering tools in planning: state-of-the-art and future challenges. In: Proceedings of the ICAPS Workshop on Knowledge Engineering for Planning and Scheduling (KEPS 2013) (2013)Google Scholar
  77. 77.
    Siminiceanu, R.I., Butler, R.W., Munoz, C.A.: Experimental evaluation of a planning language suitable for formal verification. In: Proceedings of the Fifth International Workshop on Model Checking and Artificial Intelligence, pp. 18–34 (2008)Google Scholar
  78. 78.
    Smith, B., Feather, M., Muscettola, N.: Challenges and methods in testing the Remote Agent planner. In: Proceedings of the Fifth International Conference on Artificial Intelligence Planning and Scheduling (AIPS’00), pp. 254–263 (2000)Google Scholar
  79. 79.
    Smith, B., Millar, W., Dunphy, J., Tung, Y.-W., Nayak, P., Gamble, E., Clark, M.: Validation and verification of the Remote Agent for spacecraft autonomy. In: Proceedings of IEEE Aerospace Conference (1999)Google Scholar
  80. 80.
    Smith, B., Rajan, K., Muscettola, N.: Knowledge acquisition for the onboard planner of an autonomous spacecraft. In: 10th European Workshop on Knowledge Acquisition, Modeling and Management (EKAW’97). Lecture Notes in Computer Science, vol. 1319, pp. 253–268 (1997)Google Scholar
  81. 81.
    Smith, M.H., Holzmann, G.J., Cucullu, G.C., Smith, B.D.: Model checking autonomous planners: even the best laid plans must be verified. In: Proceedings of IEEE Aerospace Conference, pp. 1–11. IEEE Computer Society, USA (2005)Google Scholar
  82. 82.
    Srivastava, S., Immerman, N., Zilberstein, S.: Finding plans with branches, loops and preconditions. In: Proceedings of the ICAPS Workshop on Verification and Validation of Planning and Scheduling Systems (VVPS’09) (2009)Google Scholar
  83. 83.
    Steffen, B., Isberner, M., Naujokat, S., Margaria, T., Geske, M.: Property-driven benchmark generation. In: Bartocci, E., Ramakrishnan, C. (eds.) Model Checking Software, vol. 7976, pp. 341–357. Lecture Notes in Computer Science. Springer, Berlin (2013)Google Scholar
  84. 84.
    Steffen, B., Margaria, T., Braun, V.: The electronic tool integration platform: concepts and design. Int. J. Softw. Tools Technol. Transf. STTT 1(1–2), 9–30 (1997)CrossRefMATHGoogle Scholar
  85. 85.
    Vaquero, T., Silva, J., Beck, J.: A brief review of tools and methods for knowledge engineering for planning and scheduling. In: Proceedings of the ICAPS Workshop on Knowledge Engineering for Planning and Scheduling (KEPS 2011) (2011)Google Scholar
  86. 86.
    Vidal, T.: A unified dynamic approach for dealing with temporal uncertainty and conditional planning. In: Proceedings of the Fifth International Conference on Artificial Intelligence Planning and Scheduling (AIPS’00) (2000)Google Scholar
  87. 87.
    Wehrle, M. Helmert, M.: The causal graph revisited for directed model checking. In: Palsberg, J., Su, Z. (eds.) SAS. Lecture Notes in Computer Science, vol. 5673, pp. 86–101. Springer, New York (2009)Google Scholar
  88. 88.
    Williams, B.C., Nayak, P.P.: A model-based approach to reactive self-configuring systems. AAAI/IAAI 2, 971–978 (1996)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Saddek Bensalem
    • 1
  • Klaus Havelund
    • 2
  • Andrea Orlandini
    • 3
  1. 1.Verimag LaboratoryGrenobleFrance
  2. 2.Jet Propulsion LaboratoryCalifornia Institute of TechnologyPasadenaUSA
  3. 3.ISTC-CNR, National Research CouncilRomeItaly

Personalised recommendations