Applied Mathematics & Optimization

, Volume 74, Issue 2, pp 375–421 | Cite as

A Piecewise Deterministic Markov Toy Model for Traffic/Maintenance and Associated Hamilton–Jacobi Integrodifferential Systems on Networks

  • Dan GoreacEmail author
  • Magdalena Kobylanski
  • Miguel Martinez


We study optimal control problems in infinite horizon whxen the dynamics belong to a specific class of piecewise deterministic Markov processes constrained to star-shaped networks (corresponding to a toy traffic model). We adapt the results in Soner (SIAM J Control Optim 24(6):1110–1122, 1986) to prove the regularity of the value function and the dynamic programming principle. Extending the networks and Krylov’s “shaking the coefficients” method, we prove that the value function can be seen as the solution to a linearized optimization problem set on a convenient set of probability measures. The approach relies entirely on viscosity arguments. As a by-product, the dual formulation guarantees that the value function is the pointwise supremum over regular subsolutions of the associated Hamilton–Jacobi integrodifferential system. This ensures that the value function satisfies Perron’s preconization for the (unique) candidate to viscosity solution.


Piecewise deterministic Markov process Viscosity solutions Network constraints Linear programming Traffic and maintenance 

Mathematics Subject Classification

49L25 93E20 60J25 49L20 



The authors would like to thank the anonymous referees for constructive remarks allowing to improve the manuscript. The work of the first author has been partially supported by the French National Research Agency project PIECE, Number ANR-12-JS01-0006.


  1. 1.
    Achdou, Y., Camilli, F., Cutrì, A., Tchou, N.: Hamilton–Jacobi equations constrained on networks. Nonlinear Differ. Equ. Appl. NoDEA 20(3), 413–445 (2013)MathSciNetCrossRefzbMATHGoogle Scholar
  2. 2.
    Achdou, Y., Oudet, S., Tchou, N.: Hamilton–Jacobi equations for optimal control on junctions and networks. ESAIM Control Optim. Calc. Var. 21(3), 876–899 (2015)MathSciNetCrossRefzbMATHGoogle Scholar
  3. 3.
    Bardet, J.-B., Christen, A., Guillin, A., Malrieu, F., Zitt, P.-A.: Total variation estimates for the TCP process. Electron. J. Probab. 18(10), 1–21 (2013)MathSciNetzbMATHGoogle Scholar
  4. 4.
    Barles, G., Jakobsen, E.R.: On the convergence rate of approximation schemes for Hamilton–Jacobi–Bellman equations. ESAIM Math. Model. Numer. Anal. 36(1), 33–54 (2002)MathSciNetCrossRefzbMATHGoogle Scholar
  5. 5.
    Barles, G., Briani, A., Chasseigne, E.: A Bellman approach for two-domains optimal control problems in R–N. ESAIM Control Optim. Calc. Var. 19, 710–739 (2013)MathSciNetCrossRefzbMATHGoogle Scholar
  6. 6.
    Barles, G., Briani, A., Chasseigne, E.: A Bellman approach for regional optimal control problems in \(\mathbb{R}^N\). SIAM J. Control Optim. 52(3), 1712–1744 (2014)MathSciNetCrossRefzbMATHGoogle Scholar
  7. 7.
    Barnard, R.C., Wolenski, P.R.: Flow invariance on stratified domains. Set-Valued Var. Anal. 21(2), 377–403 (2013)MathSciNetCrossRefzbMATHGoogle Scholar
  8. 8.
    Biswas, I.H., Jakobsen, E.R., Karlsen, K.H.: Viscosity solutions for a system of integro-PDEs and connections to optimal switching and control of jump-diffusion processes. Appl. Math. Optim. 62(1), 47–80 (2010)MathSciNetCrossRefzbMATHGoogle Scholar
  9. 9.
    Boxma, O., Kaspi, H., Kella, O., Perry, D.: On/off storage systems with state-dependent input, output, and switching rates. Probab. Eng. Inf. Sci. 19, 1–14 (2005)MathSciNetCrossRefzbMATHGoogle Scholar
  10. 10.
    Bressan, A., Hong, Y.: Optimal control problems on stratified domains. Netw. Heterog. Media 2(2), 313–331 (2007)MathSciNetCrossRefzbMATHGoogle Scholar
  11. 11.
    Camilli, F., Marchi, C.: A comparison among various notions of viscosity solution for Hamilton–Jacobi equations on networks. J. Math. Anal. Appl. 407(1), 112–118 (2013)MathSciNetCrossRefzbMATHGoogle Scholar
  12. 12.
    Cook, D.L., Gerber, A.N., Tapscott, S.J.: Modelling stochastic gene expression: implications for haploinsufficiency. Proc. Natl. Acad. Sci. USA 95, 15641–15646 (1998)CrossRefGoogle Scholar
  13. 13.
    Crudu, A., Debussche, A., Radulescu, O.: Hybrid stochastic simplifications for multiscale gene networks. BMC Syst. Biol. 3, 89 (2009)CrossRefGoogle Scholar
  14. 14.
    Crudu, A., Debussche, A., Muller, A., Radulescu, O.: Convergence of stochastic gene networks to hybrid piecewise deterministic processes. Ann. Appl. Probab. 22(5), 1822–1859 (2012)MathSciNetCrossRefzbMATHGoogle Scholar
  15. 15.
    Davis, M.H.A.: Piecewise-deterministic Markov-processes—a general-class of non-diffusion stochastic-models. J. R. Stat. Soc. Ser. B Methodol. 46(3), 353–388 (1984)zbMATHGoogle Scholar
  16. 16.
    Davis, M.H.A.: Control of piecewise-deterministic processes via discrete-time dynamic-programming. Stochastic Differential Systems. Lecture Notes in Control and Information Sciences, vol. 78, pp. 140–150. Springer, Berlin (1986)Google Scholar
  17. 17.
    Davis, M.H.A.: Markov Models and Optimization. Monographs on Statistics and Applied Probability, vol. 49. Chapman & Hall, London (1993)CrossRefGoogle Scholar
  18. 18.
    Dufour, F., Stockbridge, R.H.: On the existence of strict optimal controls for constrained, controlled Markov processes in continuous time. Stochastics 84(1), 55–78 (2012)MathSciNetzbMATHGoogle Scholar
  19. 19.
    Dufour, F., Dutuit, Y., Gonzalez, K., Zhang, H.: Piecewise deterministic Markov processes and dynamic reliability. Proc. Inst. Mech. Eng. Part O J. Risk Reliab. 222, 545–551 (2008)Google Scholar
  20. 20.
    Finlay, L., Gaitsgory, V., Lebedev, I.: Linear programming solutions of periodic optimization problems: approximation of the optimal control. J. Ind. Manag. Optim. 3(2), 399–413 (2007)MathSciNetCrossRefzbMATHGoogle Scholar
  21. 21.
    Frankowska, H., Plaskacz, S.: Semicontinuous solutions of Hamilton–Jacobi–Bellman equations with state constraints. In: Differential Inclusions and Optimal Control. Lecture Notes in Nonlinear Analysis, vol. 2, pp. 145–161 (1998)Google Scholar
  22. 22.
    Frankowska, H., Vinter, R.: Existence of neighbouring trajectorie: applications to dynamic programming for state constraints optimal control problems. J. Optim. Theory Appl. 104(1), 20–40 (2000)MathSciNetCrossRefGoogle Scholar
  23. 23.
    Goreac, D.: Viability, invariance and reachability for controlled piecewise deterministic Markov processes associated to gene networks. ESAIM Control Optim. Calc. Var. 18(2), 401–426 (2012)MathSciNetCrossRefzbMATHGoogle Scholar
  24. 24.
    Goreac, D., Serea, O.-S.: Linearization techniques for controlled piecewise deterministic Markov processes: application to Zubov’s method. Appl. Math. Optim. 66, 209–238 (2012)MathSciNetCrossRefzbMATHGoogle Scholar
  25. 25.
    Goreac, D., Serea, O.-S.: Optimality issues for a class of controlled singularly perturbed stochastic systems. J. Optim. Theory Appl. 1–31 (2015). doi: 10.1007/s10957-015-0738-4
  26. 26.
    Graham, C., Robert, P.: Interacting multi-class transmissions in large stochastic networks. Ann. Appl. Probab. 19(6), 2334–2361 (2009)MathSciNetCrossRefzbMATHGoogle Scholar
  27. 27.
    Imbert, C., Monneau, R.: Flux-Limited Solutions for quasi-convex Hamilton–Jacobi Equations on Networks (2014) (submitted)Google Scholar
  28. 28.
    Imbert, C., Monneau, R., Zidani, H.: A Hamilton–Jacobi approach to junction problems and application to traffic flows. ESAIM Control Optim. Calc. Var. 19, 129–166 (2013)MathSciNetCrossRefzbMATHGoogle Scholar
  29. 29.
    Krylov, N.V.: On the rate of convergence of finite-difference approximations for Bellman’s equations with variable coefficients. Probab. Theory Relat. Fields 117(1), 1–16 (2000)CrossRefzbMATHGoogle Scholar
  30. 30.
    Plaskacz, S., Quincampoix, M.: Discontinuous Mayer control problem under state-constraints. Topol. Methods Nonlinear Anal. 15, 91–100 (2000)MathSciNetzbMATHGoogle Scholar
  31. 31.
    Rao, Z., Siconolfi, A., Zidani, H.: Transmission conditions on interfaces for Hamilton–Jacobi–Bellman equations. J. Differ. Equ. 257(11), 3978–4014 (2014)MathSciNetCrossRefzbMATHGoogle Scholar
  32. 32.
    Rolski, T., Schmidli, H., Schmidt, V., Teugels, J.: Stochastic Processes for Insurance and Finance. Wiley Series in Probability and Statistics, vol. 505. Wiley, Chichester (2009)zbMATHGoogle Scholar
  33. 33.
    Schieborn, D., Camilli, F.: Viscosity solutions of Eikonal equations on topological networks. Calc. Var. Partial Differ. Equ. 46(3–4), 671–686 (2013)MathSciNetCrossRefzbMATHGoogle Scholar
  34. 34.
    Soner, H.M.: Optimal control with state-space constraint. I. SIAM J. Control Optim. 24(6), 552–561 (1986)MathSciNetCrossRefzbMATHGoogle Scholar
  35. 35.
    Soner, H.M.: Optimal control with state-space constraint. II. SIAM J. Control Optim. 24(6), 1110–1122 (1986)MathSciNetCrossRefzbMATHGoogle Scholar
  36. 36.
    Wainrib, G., Michèle, T., Pakdaman, K.: Intrinsic variability of latency to first-spike. Biol. Cybern. 103(1), 43–56 (2010)MathSciNetCrossRefzbMATHGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2015

Authors and Affiliations

  • Dan Goreac
    • 1
    Email author
  • Magdalena Kobylanski
    • 1
  • Miguel Martinez
    • 1
  1. 1.Université Paris-Est, LAMA (UMR 8050), UPEMLV, UPEC, CNRSMarne-la-ValléeFrance

Personalised recommendations