Quantifying Conformance Using the Skorokhod Metric

  • Jyotirmoy V. Deshmukh
  • Rupak Majumdar
  • Vinayak S. Prabhu
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9207)


The conformance testing problem for dynamical systems asks, given two dynamical models (e.g., as Simulink diagrams), whether their behaviors are “close” to each other. In the semi-formal approach to conformance testing, the two systems are simulated on a large set of tests, and a metric, defined on pairs of real-valued, real-timed trajectories, is used to determine a lower bound on the distance. We show how the Skorokhod metric on continuous dynamical systems can be used as the foundation for conformance testing of complex dynamical models. The Skorokhod metric allows for both state value mismatches and timing distortions, and is thus well suited for checking conformance between idealized models of dynamical systems and their implementations. We demonstrate the robustness of the metric by proving a transference theorem: trajectories close under the Skorokhod metric satisfy “close” logical properties in the timed linear time logic TLTL augmented with a rich class of temporal and spatial constraint predicates. We provide an efficient window-based streaming algorithm to compute the Skorokhod metric, and use it as a basis for a conformance testing tool for Simulink. We experimentally demonstrate the effectiveness of our tool in finding discrepant behaviors on a set of control system benchmarks, including an industrial challenge problem.


  1. 1.
    Abbas, H., Fainekos, G.E.: Formal property verification in a conformance testing framework. In: MEMOCODE (2014, To appear)Google Scholar
  2. 2.
    Abbas, H., Hoxha, B., Fainekos, G.E., Deshmukh, J.V., Kapinski, J., Ueda, K.: Conformance testing as falsification for cyber-physical systems. CoRR, abs/1401.5200 (2014)Google Scholar
  3. 3.
    Althoff, M.: Reachability analysis of nonlinear systems using conservative polynomialization and non-convex sets. HSCC 13, 173–182 (2013)MathSciNetCrossRefGoogle Scholar
  4. 4.
    Alur, R., Henzinger, T.A.: A really temporal logic. J. ACM 41(1), 181–204 (1994)MathSciNetCrossRefMATHGoogle Scholar
  5. 5.
    Anderson, B.D.O.: Optimal Control: Linear Quadratic Methods Dover Books on Engineering. Dover Publications, Mineola (2007) Google Scholar
  6. 6.
    Bouyer, P., Chevalier, F., Markey, N.: On the Expressiveness of TPTL and MTL. In: Sarukkai, S., Sen, S. (eds.) FSTTCS 2005. LNCS, vol. 3821, pp. 432–443. Springer, Heidelberg (2005) CrossRefGoogle Scholar
  7. 7.
    Branicky, M.S.: Studies in hybrid systems: modeling, analysis, and control. Ph.D. thesis, Massachusetts Institute of Technology, Cambridge, MA, USA (1995)Google Scholar
  8. 8.
    Broucke, M.: Regularity of solutions and homotopic equivalence for hybrid systems. IEEE Conf. Decis. Control 4, 4283–4288 (1998)Google Scholar
  9. 9.
    Caspi, P., Benveniste, A.: Toward an approximation theory for computerised control. In: Sangiovanni-Vincentelli, A.L., Sifakis, J. (eds.) EMSOFT 2002. LNCS, vol. 2491. Springer, Heidelberg (2002) CrossRefGoogle Scholar
  10. 10.
    Chen, X., Ábrahám, E., Sankaranarayanan, S.: Flow*: an analyzer for non-linear hybrid systems. In: Sharygina, N., Veith, H. (eds.) CAV 2013. LNCS, vol. 8044, pp. 258–263. Springer, Heidelberg (2013) CrossRefGoogle Scholar
  11. 11.
    Crossley, P.R., Cook, J.A.: A nonlinear engine model for drivetrain system development. In: International Conference on Control, pp. 921–925. IET (1991)Google Scholar
  12. 12.
    Davoren, J.M.: Epsilon-tubes and generalized skorokhod metrics for hybrid paths spaces. In: Majumdar, R., Tabuada, P. (eds.) HSCC 2009. LNCS, vol. 5469, pp. 135–149. Springer, Heidelberg (2009) CrossRefGoogle Scholar
  13. 13.
    Deshmukh, J.V., Majumdar, R., Prabhu, V.S.: Quantifying conformance using the Skorokhod metric. CoRR, abs/1505.05832 (2015)Google Scholar
  14. 14.
    Jinand, X., Deshmukh, J., Kapinski, J., Ueda, K., Butts, K.: Benchmarks for model transformations and conformance checking. In: ARCH (2014)Google Scholar
  15. 15.
    Donzé, A., Maler, O.: Robust satisfaction of temporal logic over real-valued signals. In: Chatterjee, K., Henzinger, T.A. (eds.) FORMATS 2010. LNCS, vol. 6246, pp. 92–106. Springer, Heidelberg (2010) CrossRefGoogle Scholar
  16. 16.
    Duggirala, P.S., Mitra, S., Viswanathan, M.: Verification of annotated models from executions. In: EMSOFT, p. 26 (2013)Google Scholar
  17. 17.
    Girard, A., Pola, G., Tabuada, P.: Approximately bisimilar symbolic models for incrementally stable switched systems. IEEE Trans. Automat. Contr. 55(1), 116–126 (2010)MathSciNetCrossRefGoogle Scholar
  18. 18.
    Haghverdi, E., Tabuada, P., Pappas, G.J.: Bisimulation relations for dynamical, control, and hybrid systems. Theor. Comput. Sci. 342(2–3), 229–261 (2005)MathSciNetCrossRefGoogle Scholar
  19. 19.
    Hennessy, M., Milner, R.: Algebraic laws for nondeterminism and concurrency. J. ACM 32(1), 137–161 (1985)MathSciNetCrossRefMATHGoogle Scholar
  20. 20.
    Henzinger, M.R., Henzinger, T.A., Kopke, P.W.: Computing simulations on finite and infinite graphs. In: FOCS: Foundations of Computer Science, pp. 453–462. IEEE Computer Society (1995)Google Scholar
  21. 21.
    Jin, X., Deshmukh, J.V., Kapinski, J., Ueda, K., Butts, K.: Powertrain control verification benchmark. In: HSCC, pp. 253–262 (2014)Google Scholar
  22. 22.
    Kapinski, J., Deshmukh, J.V., Sankaranarayanan, S., Arechiga, N.: Simulation-guided lyapunov analysis for hybrid dynamical systems. In: HSCC 2014, pp 133–142. ACM (2014)Google Scholar
  23. 23.
    Koymans, R.: Specifying real-time properties with metric temporal logic. Real Time Syst. 2(4), 255–299 (1990)CrossRefGoogle Scholar
  24. 24.
    Majumdar, R., Prabhu, V.S.: Computing the Skorokhod distance between polygonal traces. CoRR, abs/1410.6075 (2014)Google Scholar
  25. 25.
    Majumdar, R., Prabhu, V.S.: Computing the Skorokhod distance between polygonal traces. In: HSCC (2015)Google Scholar
  26. 26.
    The Mathworks. Engine timing model with closed loop controlGoogle Scholar
  27. 27.
    Messner, W., Tilbury, D.: Control tutorials for matlab and simulinkGoogle Scholar
  28. 28.
    Milner, R. (ed.): A Calculus of Communicating Systems. LNCS. Springer, Heidelberg (1980) Google Scholar
  29. 29.
    Sangiorgi, D., Rutten, J.: Advanced Topics in Bisimulation and Coinduction. Cambridge University Press, Cambridge (2011)CrossRefGoogle Scholar
  30. 30.
    Tabuada, P.: Verification and Control of Hybrid Systems - A Symbolic Approach. Springer, Heidelberg (2009)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Jyotirmoy V. Deshmukh
    • 1
  • Rupak Majumdar
    • 2
  • Vinayak S. Prabhu
    • 2
    • 3
  1. 1.Toyota Technical CenterLos AngelesUSA
  2. 2.MPI-SWSKaiserslauternGermany
  3. 3.University of PortoPortoPortugal

Personalised recommendations