Formal Methods in System Design

, Volume 51, Issue 1, pp 154–199 | Cite as

Predictive runtime enforcement

  • Srinivas PinisettyEmail author
  • Viorel Preoteasa
  • Stavros Tripakis
  • Thierry Jéron
  • Yliès Falcone
  • Hervé Marchand


Runtime enforcement (RE) is a technique to ensure that the (untrustworthy) output of a black-box system satisfies some desired properties. In RE, the output of the running system, modeled as a sequence of events, is fed into an enforcer. The enforcer ensures that the sequence complies with a certain property, by delaying or modifying events if necessary. This paper deals with predictive runtime enforcement, where the system is not entirely black-box, but we know something about its behavior. This a priori knowledge about the system allows to output some events immediately, instead of delaying them until more events are observed, or even blocking them permanently. This in turn results in better enforcement policies. We also show that if we have no knowledge about the system, then the proposed enforcement mechanism reduces to standard (non-predictive) runtime enforcement. All our results related to predictive RE of untimed properties are also formalized and proved in the Isabelle theorem prover. We also discuss how our predictive runtime enforcement framework can be extended to enforce timed properties.


Runtime monitoring Runtime enforcement Automata Timed automata Monitor synthesis 



This work was supported in part by the Academy of Finland and the U.S. National Science Foundation (Awards #1329759 and #1139138). Thierry Jéron and Yliès Falcone acknowledge the support of the COST Action ARVI IC1402, which is supported by COST (European Cooperation in Science and Technology).


  1. 1.
    Alur R, Dill DL (1994) A theory of timed automata. Theor Comput Sci 126:183–235. doi: 10.1016/0304-3975(94)90010-8 MathSciNetCrossRefzbMATHGoogle Scholar
  2. 2.
    Baier C, Bertrand N, Bouyer P, Brihaye T (2009) When are timed automata determinizable? In: Albers S, Marchetti-Spaccamela A, Matias Y, Nikoletseas SE, Thomas W (eds) 36th international colloquium on automata, languages and programming, ICALP 2009, Rhodes, Greece, July 5–12, 2009, proceedings, part II. Lecture notes in computer science, vol 5556. Springer, pp 43–54. doi: 10.1007/978-3-642-02930-1_4
  3. 3.
    Bloem R, Könighofer B, Könighofer R, Wang C (2015) Shield synthesis: runtime enforcement for reactive systems. In: TACAS. LNCS, vol 9035. Springer, BerlinGoogle Scholar
  4. 4.
    Cassandras CG, Lafortune S (2006) Introduction to discrete event systems. Springer, SecaucuszbMATHGoogle Scholar
  5. 5.
    Chabot H, Khoury R, Tawbi N (2011) Extending the enforcement power of truncation monitors using static analysis. Comput Secur 30(4):194–207CrossRefGoogle Scholar
  6. 6.
    Dolzhenko E, Ligatti J, Reddy S (2015) Modeling runtime enforcement with mandatory results automata. Int J Inf Secur 14(1):47–60. doi: 10.1007/s10207-014-0239-8 CrossRefGoogle Scholar
  7. 7.
    D’Silva V, Kroening D, Weissenbacher G (2008) A survey of automated techniques for formal software verification. IEEE Trans CAD Integr Circuits Syst 27(7):1165–1178. doi: 10.1109/TCAD.2008.923410 CrossRefGoogle Scholar
  8. 8.
    Evans D, Larochelle D (2002) Improving security using extensible lightweight static analysis. IEEE Softw 19(1):42–51. doi: 10.1109/52.976940 CrossRefGoogle Scholar
  9. 9.
    Falcone Y, Fernandez J, Mounier L (2012) What can you verify and enforce at runtime? STTT 14(3):349–382CrossRefGoogle Scholar
  10. 10.
    Falcone Y, Mounier L, Fernandez JC, Richier JL (2011) Runtime enforcement monitors: composition, synthesis, and enforcement abilities. Form Methods Syst Des 38(3):223–262CrossRefzbMATHGoogle Scholar
  11. 11.
    Falcone Y, Jéron T, Marchand H, Pinisetty S (2016) Runtime enforcement of regular timed properties by suppressing and delaying events. Sci Comput Program 123:2–41CrossRefGoogle Scholar
  12. 12.
    Finkel O (2006) Undecidable problems about timed automata. In: Asarin E, Bouyer P (ed) Formal modeling and analysis of timed systems: 4th international conference, FORMATS 2006, Paris, France, September 25–27, 2006. Springer, Berlin, pp 187–199Google Scholar
  13. 13.
    Ligatti J, Bauer L, Walker D (2009) Run-time enforcement of nonsafety policies. ACM Trans Inf Syst Secur 12(3):19:1–19:41CrossRefGoogle Scholar
  14. 14.
    Malan GR, Watson D, Jahanian F, Howell P (2000) Transport and application protocol scrubbing. In: Proceedings IEEE INFOCOM 2000, Israel, pp 1381–1390Google Scholar
  15. 15.
    Nipkow T, Paulson LC, Wenzel M (2002) Isabelle/HOL—a proof assistant for higher-order logic. In: LNCS, vol 2283. Springer, BerlinGoogle Scholar
  16. 16.
    Pinisetty S, Falcone Y, Jéron T, Marchand H (2014) Runtime enforcement of parametric timed properties with practical applications. In: Lesage J, Faure J, Cury JER, Lennartson B (eds) 12th international workshop on discrete event systems, WODES 2014. International federation of automatic control, pp. 420–427.
  17. 17.
    Pinisetty S, Falcone Y, Jéron T, Marchand H (2014) Runtime enforcement of regular timed properties. In: Proceedings of the ACM symposium on applied computing (SAC-SVT). ACM, pp 1279–1286Google Scholar
  18. 18.
    Pinisetty S, Falcone Y, Jéron T, Marchand H, Rollet A, Timo OLN (2012) Runtime enforcement of timed properties. In: Qadeer S, Tasiran S (eds) Proceedings of the third international conference on runtime verification (RV 2012). Lecture notes in computer science, vol 7687. Springer, Berlin, pp 229–244Google Scholar
  19. 19.
    Pinisetty S, Falcone Y, Jéron T, Marchand H, Rollet A, Nguena Timo O (2014) Runtime enforcement of timed properties revisited. Form Methods Syst Des 45(3):381–422CrossRefzbMATHGoogle Scholar
  20. 20.
    Pinisetty S, Preoteasa V, Tripakis S, Jéron T, Falcone Y, Marchand H (2016) Predictive runtime enforcement. In: Proceedings of the ACM symposium on applied computing (SAC-SVT). ACM (to appear)Google Scholar
  21. 21.
    Pitt J, Mamdani EH (1999) A protocol-based semantics for an agent communication language. In: Dean T (ed) Proceedings of the sixteenth international joint conference on artificial intelligence, IJCAI 99, Stockholm, Sweden, July 31–August 6, 1999, 2 vol. Morgan Kaufmann, pp 486–491Google Scholar
  22. 22.
    Raffelt H, Steffen B, Berg T, Margaria T (2009) Learnlib: a framework for extrapolating behavioral models. Int J Softw Tools Technol Transf 11(5):393–407. doi: 10.1007/s10009-009-0111-8 CrossRefGoogle Scholar
  23. 23.
    Renard M, Falcone Y, Rollet A, Pinisetty S, Jéron T, Marchand H (2015) Enforcement of (timed) properties with uncontrollable events. In: Leucker M, Rueda C, Valencia FD (eds) 12th international colloquium on theoretical aspects of computing–ICTAC 2015, Cali, Colombia, October 29–31, 2015, proceedings. Lecture notes in computer science, vol 9399. Springer, Berlin, pp 542–560Google Scholar
  24. 24.
    Rosu G (2012) On safety properties and their monitoring. Sci Ann Comput Sci 22(2):327–365MathSciNetGoogle Scholar
  25. 25.
    Schneider FB (2000) Enforceable security policies. ACM Trans Inf Syst Secur 3(1):30–50. doi: 10.1145/353323.353382 MathSciNetCrossRefGoogle Scholar
  26. 26.
    Tripakis S (2006) Folk theorems on the determinization and minimization of timed automata. Inf Process Lett 99(6):222–226. doi: 10.1016/j.ipl.2006.04.015 MathSciNetCrossRefzbMATHGoogle Scholar
  27. 27.
    Tuglular T, Belli F (2009) Protocol-based testing of firewalls. In: 2009 fourth south-east European workshop on formal methods (SEEFM), pp 53–59Google Scholar
  28. 28.
    Wooldridge M (2009) An introduction to multiagent systems, 2nd edn. Wiley Publishing, New YorkGoogle Scholar
  29. 29.
    Zhang X, Leucker M, Dong W (2012) Runtime verification with predictive semantics. In: 4th international symposium on NASA formal methods. LNCS, vol 7226. Springer, Berlin, pp 418–432Google Scholar

Copyright information

© Springer Science+Business Media New York 2017

Authors and Affiliations

  1. 1.Aalto UniversityEspooFinland
  2. 2.University of California, BerkeleyBerkeleyUSA
  3. 3.INRIA Rennes - Bretagne AtlantiqueRennesFrance
  4. 4.Laboratoire d’Informatique de GrenobleUniv. Grenoble Alpes, Inria, LIGGrenobleFrance

Personalised recommendations