Methodology and Computing in Applied Probability

, Volume 18, Issue 3, pp 885–900 | Cite as

Estimation for Discrete-time Semi-Markov Reward Processes: Analysis and Inference

  • K. KhorshidianEmail author
  • F. Negahdari
  • H. A. Mardnifard


In this paper, the moments of a class of reward processes defined on a discrete-time semi-Markov process and the asymptotic behaviors of the corresponding empirical estimators have been investigated. Some known results concerning the asymptotic distribution and properties of semi-Markov kernel have been obtained by a different approach. By using the empirical estimator of the semi-Markov kernel and the mentioned approach, the estimators for the moments of the reward process have been introduced and their asymptotic properties have been established. As a consequence of the strong consistency and asymptotic normality, the confidence intervals have also been obtained. A numerical example illustrates the results.


Discrete-time semi-Markov processes Reward process Empirical estimator Delta method Asymptotic properties 

Mathematics Subject Classification (2010)

60K15 62K20 60J0 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Anderson TW, Goodman LA (1957) Statistical inference about Markov chain. Ann math Statist 28:89–110MathSciNetCrossRefzbMATHGoogle Scholar
  2. Barbu V, Boussmart M, Limnios N (2004) Discrete time semi-Markov model for reliability and survival analysis. Communications in Statistics – Theory and Methods 33(11):2833–2868MathSciNetCrossRefzbMATHGoogle Scholar
  3. Barbu V, Limnios N (2006) Empirical estimator for discrete-time semi-Markov processes with applications in reliability. Journal of Nonparametric statistics 18:483–498MathSciNetCrossRefzbMATHGoogle Scholar
  4. Cinlar E (1969) Markov renewal theory. Adv Appl Prob 1:123–187MathSciNetCrossRefzbMATHGoogle Scholar
  5. D’Amico G (2009) Nonparametric estimation of the accumulated reward for semi-Markov chains. SORT 33(2):159–170MathSciNetzbMATHGoogle Scholar
  6. D’Amico G (2010) Measuring the quality of life through Markov reward process: Analysis and inference. Environmetrics 21:208–220MathSciNetGoogle Scholar
  7. D’Amico G, Guillen M, Manca R (2013) Semi-Markov Disability Models. Communications in Statistics: Theory and Methods 42(16):2172–2188MathSciNetzbMATHGoogle Scholar
  8. D’Amico G, Janssen J, Manca R (a 2011) A Non-Homogeneous Semi-Markov Reward Model for the Credit Spread Computation. Int J Theo Appl Finance 14 (2):221–238MathSciNetCrossRefzbMATHGoogle Scholar
  9. D’Amico G, Manca R, Salvi G (2013) A Semi-Markov Modulated Interest Rate Model. Statistics and Probability Letters 83:2094–2102MathSciNetCrossRefzbMATHGoogle Scholar
  10. Janssen J, Manca R (2006) Applied Semi-Markov processes. Springer, New YorkzbMATHGoogle Scholar
  11. Khorshidian K, Soltani AR (2002) Asymptotic behavior of multivariate reward processes with nonlinear reward functions. Bulletin of the Iranian Mathematical society 28(2):1–17MathSciNetzbMATHGoogle Scholar
  12. Khorshidian K (2008) Strong law large numbers of semi-Markov reward processes. Asian Journal of Mathematics and Statistics 1(1):57–62MathSciNetCrossRefGoogle Scholar
  13. Khorshidian K (2009) Central Limit Theorem for Nonlinear Semi-Markov Reward processes. Stoch Anal Appl 27(4):656–670MathSciNetCrossRefzbMATHGoogle Scholar
  14. Limnios N (2004) A functional central limit theorem for the empirical estimator of a semi-Markov kernel. J Nonparametric statistics 16(1–2):13–18MathSciNetCrossRefzbMATHGoogle Scholar
  15. Ouhbi B, Limnios N (1999) Nonparametric estimation for Markov processes based on its hazard rate. Stat Infer Stochat Proc 2(2):151–173MathSciNetCrossRefzbMATHGoogle Scholar
  16. Ouhbi B, Limnios N (2003) Nonparametric reliability estimation of semi-Markov processes. J Statist Plann Infer 109(1–2):155–165MathSciNetCrossRefzbMATHGoogle Scholar
  17. Pyke R (1961a) Markov renewal process: definitions and preliminary properties. Ann Math Statist 32:1231–1242MathSciNetCrossRefzbMATHGoogle Scholar
  18. Sadek A, Limnios N (2002) Asymptotic properties for maximum likelihood estimators for reliability and failure rates of Markov chains. Communication in Statistics-Theory and Methods 31(10):1837–1861MathSciNetCrossRefzbMATHGoogle Scholar
  19. Soltani AR (1996) Reward processes with nonlinear reward function. J Appl Prob 33:1101–1017MathSciNetCrossRefzbMATHGoogle Scholar
  20. Soltani AR, Khorshidian K (1998) Reward processes for semi-Markov processes: Asymptotic behavior. J Appl Prob 35:833–842MathSciNetCrossRefzbMATHGoogle Scholar
  21. Soltani AR, Khorshidian K, Ghafaripour A (2010) Prediction for reward processes. Stoch Model 26:242–255MathSciNetCrossRefzbMATHGoogle Scholar
  22. Van Der Vart AW (2000). Asymptotic statistics, Cambridge series in statistical and probability mathematics, 3, Cambridge University PressGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2015

Authors and Affiliations

  • K. Khorshidian
    • 1
    Email author
  • F. Negahdari
    • 1
  • H. A. Mardnifard
    • 2
  1. 1.Department of StatisticsShiraz UniversityShirazIran
  2. 2.Department of MathematicsYasouj UniversityYasoujIran

Personalised recommendations