Markov Chains pp 107-139 | Cite as

Markov Decision Processes for Customer Lifetime Value

  • Wai-Ki Ching
  • Ximin Huang
  • Michael K. Ng
  • Tak-Kuen Siu
Part of the International Series in Operations Research & Management Science book series (ISOR, volume 189)


In this chapter a stochastic dynamic programming model with a Markov chain is proposed to capture customer behavior. The advantage of using Markov chains is that the model can take into account the customers switching between the company and its competitors. Therefore customer relationships can be described in a probabilistic way, see for instance Pfeifer and Carraway [170]. Stochastic dynamic programming is then applied to solve the optimal allocation of the promotion budget for maximizing the Customer Lifetime Value (CLV). The proposed model is then applied to practical data in a computer services company.


Markov Decision Process Markov Chain Model Infinite Horizon Stochastic Dynamic Programming Finite Horizon 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Adke S, Deshmukh D (1988) Limit distribution of a high order Markov chain. J Roy Stat Soc Ser B 50:105–108Google Scholar
  2. 3.
    Altman E (1999) Constrained markov decision processes. Chapman and Hall/CRC, LondonGoogle Scholar
  3. 13.
    Berger P, Nasr N (1998) Customer lifetime value: marketing models and applications. J Interact Market 12:17–30CrossRefGoogle Scholar
  4. 14.
    Berger P, Nasr N (2001) The allocation of promotion budget to maximize customer equity. Omega 29:49–61CrossRefGoogle Scholar
  5. 20.
    Blattberg R, Deighton J (1996) Manage market by the customer equity. Harv Bus Rev 73:136–144Google Scholar
  6. 34.
    Buzacott J, Shanthikumar J (1993) Stochastic models of manufacturing systems. Prentice-Hall International Editions, New JerseyGoogle Scholar
  7. 35.
    Carpenter P (1995) Customer lifetime value: do the math. Market Comput 15:18–19Google Scholar
  8. 39.
    Chan R, Ng M (1996) Conjugate gradient method for Toeplitz systems. SIAM Rev 38:427–482CrossRefGoogle Scholar
  9. 41.
    Ching W (1997) Circulant preconditioners for failure prone manufacturing systems. Linear Algebra Appl 266:161–180CrossRefGoogle Scholar
  10. 45.
    Ching W (2001) Machine repairing models for production systems. Int J Product Econ 70:257–266CrossRefGoogle Scholar
  11. 47.
    Ching W (2001) Markovian approximation for manufacturing systems of unreliable machines in Tandem. Int J Naval Res Logist 48:65–78CrossRefGoogle Scholar
  12. 48.
    Ching W (2003) Iterative methods for queuing systems with batch arrivals and negative customers. BIT 43:285–296CrossRefGoogle Scholar
  13. 49.
    Ching W, Chan R, Zhou X (1997) Circulant preconditioners for Markov modulated poisson processes and their applications to manufacturing systems. SIAM J Matrix Anal Appl 18:464–481CrossRefGoogle Scholar
  14. 50.
    Ching W, Fung E, Ng M (2002) A multivariate Markov chain model for categorical data sequences and its applications in demand predictions. IMA J Manag Math 13:187–199CrossRefGoogle Scholar
  15. 51.
    Ching W, Fung E, Ng M (2003) A higher-order Markov model for the Newsboy’s problem. JOper Res Soc 54:291–298CrossRefGoogle Scholar
  16. 61.
    Ching W, Ng M, So M (2004) Customer migration, campaign budgeting, revenue estimation: the elasticity of Markov decision process on customer lifetime value. Electron Int J Adv Model Optim 6(2):65–80Google Scholar
  17. 62.
    Ching W, Ng M, Wong K (2003) Higher-order Markov decision process and its applications in customer lifetime values. In: The 32nd international conference on computers and industrial engineering, vol 2, Limerick, Ireland, pp 821–826, 2003Google Scholar
  18. 64.
    Ching W, Ng M, Wong K, Atlman E (2004) Customer lifetime value: a stochastic programming approach. J Oper Res Soc 55:860–868CrossRefGoogle Scholar
  19. 73.
    Ching W, Yuen W, Ng M, Zhang S (2006) A linear programming approach for solving optimal advertising policy. IMA J Manag Math 17:83–96CrossRefGoogle Scholar
  20. 82.
    DuWors R, Haines G (1990) Event history analysis measure of brand loyalty. J Market Res 27:485–493CrossRefGoogle Scholar
  21. 121.
    Hughes A, Wang P (1995) Media selection for database marketers. J Direct Market 9:79–84CrossRefGoogle Scholar
  22. 123.
    Jackson B (1985) Winning and keeping industrial customers. Lexington Books, Lexington, MAGoogle Scholar
  23. 124.
    Jain D, Singh S (2002) Customer lifetime value research in marketing: a review and future directions. J Interact Market 16:34–46CrossRefGoogle Scholar
  24. 134.
    Kotler P, Armstrong G (1995) Principle of marketing, 7th edn. Prentice Hall, Englewood CliffsGoogle Scholar
  25. 145.
    Leung H, Ching W, Leung I (2008) A stochastic optimization model for consecutive promotion. Qual Technol Quant Manag 5:403–414Google Scholar
  26. 148.
    Lilien L, Kotler P, Moorthy K (1992) Marketing models. Prentice Hall, Englewood CliffsGoogle Scholar
  27. 150.
    Lin C, Lin Y (2007) Robust analysis on promotion duration for two competitive brands. J Oper Res Soc 1:1–8Google Scholar
  28. 159.
    Mesak H (2003) On deriving and validating comparative statics of a symmetric model of advertising competition. Comput Oper Res 30:1791–1806CrossRefGoogle Scholar
  29. 160.
    Mesak H, Calloway J (1999) Hybrid subgames and copycat games in a pulsing model of advertising competition. J Oper Res Soc 50:837–849Google Scholar
  30. 161.
    Mesak H, Means T (1998) Modelling advertising budgeting and allocation decisions using modified multinomial logit market share models. J Oper Res Soc 49:1260–1269Google Scholar
  31. 162.
    Mesak H, Zhang H (2001) Optimal advertising pulsation policies: a dynamic programming approach. J Oper Res Soc 11:1244–1255CrossRefGoogle Scholar
  32. 170.
    Pfeifer P, Carraway R (2000) Modeling customer relationships as Markov chain. J Int Market 14:43–55Google Scholar
  33. 181.
    Ross S (2000) Introduction to probability models, 7th edn. Academic, New YorkGoogle Scholar
  34. 207.
    White D (1993) Markov decision processes. Wiley, ChichesterGoogle Scholar
  35. 208.
    Winston W (1994) Operations research: applications and algorithms, Belmont Calif., 3rd edn. Duxbury, North ScituateGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2013

Authors and Affiliations

  • Wai-Ki Ching
    • 1
  • Ximin Huang
    • 2
  • Michael K. Ng
    • 3
  • Tak-Kuen Siu
    • 4
  1. 1.Department of MathematicsThe University of Hong KongHong KongHong Kong, SAR
  2. 2.College of ManagementGeorgia Institute of TechnologyAtlantaUSA
  3. 3.Department of MathematicsHong Kong Baptist UniversityKowloon TongHong Kong SAR
  4. 4.Cass Business SchoolCity University LondonLondonUK

Personalised recommendations