Abstract
New algorithms for computing power moments of hitting times and accumulated rewards of hitting type for semi-Markov processes are developed. The algorithms are based on special techniques of sequential phase space reduction and recurrence relations connecting moments of rewards. Applications are discussed as well as possible generalizations of presented results and examples.
Article PDF
Similar content being viewed by others
Avoid common mistakes on your manuscript.
References
Anisimov VV (2008) Switching Processes in Queueing Models. Applied Stochastic Methods Series. ISTE, London and Wiley, Hoboken, NJ
Anisimov VV, Zakusilo OK, Donchenko VS (1987) Elements of Queueing and Asymptotical Analysis of Systems. Lybid’, Kiev, pp 246
Barbu V, Boussemart M, Limnios N (2004) Discrete time semi-Markov model for reliability and survival analysis. Commun Stat Theory, Methods 33(11):2833–2868
Biffi G, D’Amico G, Di Biase G, Janssen J, Manca R, Silvestrov D (2008) Monte Carlo semi-Markov methods for credit risk migration and Basel II rules. I, II J Numer Appl Math 1(96):I: 28–58, II: 59–86
Chung KL (1954) Contributions to the theory of Markov chains II. Trans Amer Math Soc 76:397–419
Chung KL (1960) Fundamental principles of mathematical sciences, 104 Markov chains with stationary transition probabilities. Springer, Berlin, p x+278. 1967
Ciardo G, Raymonf MA, Sericola B, Trivedi KS (1990) Performability analysis using semi-Markov reward processes. IEEE Trans Comput 39(10):1251–1264
Cogburn R (1975) A Uniform theory for sums of Markov chain transition probabilities. Ann Probab 3:191–214
Courtois PJ (1977) ACM Monograph Series Decomposability. Queueing and Computer System Applications. Academic Press, New York, p Xiii+201
D’Amico G, Guillen M, Manca R (2013) Semi-markov disability insurance models. Commun Stat Theory Methods 42(16):2172–2188
D’Amico G, Janssen J, Manca R (2005) Homogeneous semi-Markov reliability models for credit risk management. Decis Econ Finance 28(2):79–93
D’Amico G, Petroni F (2012) A semi-Markov model for price returns. Physica A 391:4867–4876
D’Amico G, Petroni F, Prattico F (2013) First and second order semi-Markov chains for wind speed modeling. Physica A 392(5):1194–1201
D’Amico G, Petroni F, Prattico F (2015) Performance analysis of second order semi-Markov chains: an application to wind energy production. Methodol Comput Appl Probab 17(3):781–794
Golub GH, Van Loan CF (2013) Johns Hopkins Studies in the Mathematical Sciences Matrix Computations, 4th edn. Johns Hopkins University Press, Baltimore, p xiv+756. An extended variant of the first 1983 edition
Gyllenberg M, Silvestrov D (2008) De Gruyter Expositions in Mathematics, 44 Quasi-stationary Phenomena in Nonlinearly Perturbed Stochastic Systems. Walter de Gruyter, Berlin, p ix+579
Hunter JJ (2005) Stationary distributions and mean first passage times of perturbed Markov chains. Linear Algebra Appl 410:217–243
Janssen J, Manca R (2006) Applied Semi-Markov Processes. Springer, New York, p xii+309
Janssen J, Manca R (2007) Semi-markov Risk Models for Finance, Insurance and Reliability, Springer, New York
Kemeny J G, Snell JL (1961a) Potentials for denumerable Markov chains. J Math Anal Appl 6:196–260
Kemeny JG, Snell JL (1961b) Finite continuous time Markov chains. Theor Probab Appl 6:110–115
Korolyuk VS, Brodi SM, Turbin AF (1974) Theoretical Semi-Markov processes 11 Cybernetics, their application. Probability Theory. Mathematical Statistics, vol 1974. VINTI, Moscow, pp 47–97
Korolyuk VS, Korolyuk VV (1999) Mathematics and its Applications, 469 Stochastic Models of Systems. Kluwer, Dordrecht, p xii+185
Koroliuk VS, Limnios N (2005) Stochastic Systems in Merging Phase Space. World Scientific, Singapore, p xv+331
Korolyuk VS, Turbin AF (1976) Semi-Markov Processes and its Applications. Naukova Dumka, Kiev
Korolyuk VS, Turbin AF (1978) Mathematical Foundations of the State Lumping of Large Systems. Naukova Dumka, Kiev. (English edition: Mathematics and its Applications, 264, Kluwer, Dordrecht, 1993, x+278 pp.)
Kovalenko IN (1975) Studies in the Reliability Analysis of Complex Systems. Naukova Dumka, Kiev
Kovalenko IN, Kuznetsov NYU, Pegg PA (1997) Wiley Series in Probability and Statistics Mathematical Theory of Reliability of Time Dependent Systems with Practical Applications. Wiley, New York, p 316
Lamperty J (1963) Criteria for stochastic processes II: passage-time moments. J Math Anal Appl 7:127–145
Limnios N, Oprişan G (2001) Statistics for Industry and Technology Semi-markov Processes and Reliability. Birkhäuser, Boston, p xii+222
Limnios N, Oprişan G (2003) An introduction to Semi-Markov processes with application to reliability. In: Shanbhag D N, Rao C R (eds). handbook of statistics, vol 21, pp 515–556
Nummelin E (1984) Cambridge Tracts in Mathematics, 83 General Irreducible Markov Chains and Nonnegative Operators. Cambridge University Press, Cambridge, p xi+172
Papadopoulou AA, Tsaklidis G, McClean S, Garg L (2012) On the moments and the distribution of the cost of a semi-Markov model for healthcare systems. Methodol Comput Appl Probab 14(3):717–737
Papadopoulou AA (2013) Some results on modeling biological sequences and web navigation with a semi- Markov chain. Comm Stat Theory, Methods 42(16):2153–2171
Pitman JW (1974a) Studies in probability and statistics. In: Williams E J (ed) An identity for stopping times of a Markov process. Academic press, Jerusalem, pp 41–57
Pitman JW (1974b) Uniform rates of convergence for Markov chain transition probabilities. Z Wahrscheinlichkeitsth. 29:193–227
Pitman JW (1977) Occupation measures for Markov chains. Adv Appl Probab 9:69–86
Silvestrov DS (1974) Limit Theorems for Composite Random Functions. Vysshaya Shkola and Izdatel’stvo Kievskogo Universiteta, Kiev
Silvestrov DS (1980a) Mean hitting times for semi-Markov processes, and queueing networks. Elektron Infor Kybern 16:399–415
Silvestrov DS (1980b) Library for an Engineer in Reliability Semi-markov Processes with a Discrete State Space. Sovetskoe Radio, Moscow, p 272
Silvestrov DS (1983a) Method of a single probability space in ergodic theorems for regenerative processes I. Math Operationsforsch Statist, Ser Optimization 14:286–299
Silvestrov DS (1983b) Invariance principle for the processes with semi-Markov switch-overs with an arbitrary state space Proceedings of the 4th USSR-Japan symposium on probability theory and mathematical statistics, tbilisi 1983. Lecture notes in math., vol 1021, pp 617–628
Silvestrov DS (1994) Coupling for Markov renewal processes and the rate of convergence in ergodic theorems for processes with semi-Markov switchings. Acta Applic Math 34:109–124
Silvestrov DS (1996) Recurrence relations for generalised hitting times for semi-Markov processes. Ann Appl Probab 6:617–649
Silvestrov DS, Drozdenko MO (2006) Sufficient conditions for weak convergence of first-rare-event times for semi-Markov processes. Theory Stoch. Process., 12(28) no. 3-4 Necessary Part I: 151–186, Part II: 187–202
Silvestrov D, Manca R, Silvestrova E (2014) Computational algorithms for moments of accumulated Markov and semi-Markov rewards. Comm Statist Theory Methods 43(7):1453–1469
Silvestrov D, Silvestrova E, Manca R (2008) Stochastically ordered models for credit rating dynamics. J Numer Appl Math 1(96):206–218
Silvestrov D, Silvestrov S (2016) Asymptotic expansions for stationary distributions of perturbed semi-Markov processes. In: Silvestrov S, Ranc̆ić M (eds) Engineering Mathematics II. Algebraic, Stochastic and Analysis Structures for Networks, Data Classification and Optimization, Chapter 10. Springer Proceedings in Mathematics & Statistics, 179, Springer, Cham, pp 151–222
Stenberg F, Manca R, Silvestrov D (2006) Semi-markov reward models for disability insurance. Theory Stoch Proces 12(28):239–254. 3-4
Stenberg F, Manca R, Silvestrov D (2007) An algorithmic approach to discrete time non-homogeneous backward semi-Markov reward process with an application to disability insurance. Metodol Comput Appl Probab 9:497–519
Yin GG, Zhang Q (2005) Two-time-scale Methods and Applications. Stochastic Modelling and Applied Probability, 55 Discrete-time Markov Chains. Springer, New York, p xix+348
Yin GG, Zhang Q (2013) Stochastic Modelling and Applied Probability, 37 Continuous-Time Markov Chains and Applications. A Two-Time-Scale Approach, 2nd edn. Springer, New York, p xxii+427. (An extended variant of the first 1998 edition)
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
About this article
Cite this article
Silvestrov, D., Manca, R. Reward Algorithms for Semi-Markov Processes. Methodol Comput Appl Probab 19, 1191–1209 (2017). https://doi.org/10.1007/s11009-017-9559-2
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11009-017-9559-2
Keywords
- Semi-Markov process
- Hitting time
- Accumulated reward
- Power moment
- Phase space reduction
- Recurrent algorithm