Abstract
This paper provides a step-by-step guide to estimating infinite horizon discrete choice dynamic programming (DDP) models using a new Bayesian estimation algorithm (Imai et al., Econometrica 77:1865–1899, 2009a) (IJC). In the conventional nested fixed point algorithm, most of the information obtained in the past iterations remains unused in the current iteration. In contrast, the IJC algorithm extensively uses the computational results obtained from the past iterations to help solve the DDP model at the current iterated parameter values. Consequently, it has the potential to significantly alleviate the computational burden of estimating DDP models. To illustrate this new estimation method, we use a simple dynamic store choice model where stores offer “frequent-buyer” type rewards programs. Our Monte Carlo results demonstrate that the IJC method is able to recover the true parameter values of this model quite precisely. We also show that the IJC method could reduce the estimation time significantly when estimating DDP models with unobserved heterogeneity, especially when the discount factor is close to 1.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
Geweke and Keane (2000) proposed to use a flexible polynomial to approximate the future component of the Bellman equation. Their approach allowed them to conduct Bayesian inference on the structural parameters of the current payoff functions and the reduced form parameters of the polynomial approximations. However, since it completely avoids solving and fully specifying the DDP model, their estimation results are not efficient and in general policy experiments cannot be conducted under their approach.
Given that we assume s evolves according to f(s′|s, a; θ s ), one can estimate θ s based on the observed transition of s alone, without using the DDP model.
Walsh (2004) provides an excellent introduction to MCMC methods.
Norets (2010) provides a set of general model assumptions under which the implied value function is continuous in θ.
Formally, the convergence results require three more assumptions: (i) Θ is compact; (ii) the return function, R(s, a, ϵ; θ R ) and the initial guess of the value function, \(\mathcal{V}^0(s,\epsilon;\theta)\), are continuous in ε and θ; (iii) the prior distribution π(θ) is positive and bounded for any given θ ∈ Θ.
Strictly speaking, parameter vector draws obtained from the IJC algorithm are not a Markov chain because the pseudo-expected value function depends on the past pseudo-value functions, which are evaluated at \(\{\theta^l\}_{l=r-N}^{r-2}\) in addition to θ r − 1. As a result, the proof of convergence is non-standard (Imai et al. 2009b).
Norets (2009) derives the convergence rates under the nearest neighbor kernel.
Brown and Flinn (2011) extend the implementation of this key step in estimating a dynamic model of marital status choice and investment in children using the method of simulated moments.
It is important to note that Bernstein and von Mises Theorem states that Bayesian posterior mean and the ML estimators are asymptotically equivalent.
A stochastic optimization algorithm, simulated annealing, has recently gained some attention to handle complicated objective functions. This algorithm is an adaptation of the M-H algorithm (Černý 1985; Kirkpatrick et al. 1983). The approximation step proposed by IJC should also be well-suited when researchers use simulated annealing to maximize/minmize the objective function in classical approaches (e.g., ML and GMM). However, we should note that before a researcher starts the estimation, this method requires him/her to choose a “cooling” rate. The ideal cooling rate cannot be determined a priori. In the MCMC-based Bayesian algorithm, one does not need to deal with this nuisance parameter.
Imai et al. (2009b) only proved convergence for the algorithm where the value functions for the candidate parameter draws were stored. This is because it is easier to prove convergence when the stochastic variations of the parameters are controlled by the candidate generating function than jointly by the candidate generating function and the acceptance rate of the M-H algorithm.
Note that in this setup, the return function is unbounded because ϵ a has unbounded support. Therefore, to show that the Bellman operator is a contraction mapping, one needs to apply a generalized version of Blackwell’s Theorem provided in Rust (1988).
In general, the state space can consist of a mixture of discrete and continuous state variables. In such a case, readers can combine the results in the base case and in this subsection to obtain the nonparametric approximation of the expected value function. See Section 4.4 for an example.
In the example that we will discuss later, we assume g is a normal distribution and μ includes parameters for mean and standard deviation. Assuming that the prior on the mean parameters is normal and that for standard deviation parameters is inverse Wishart (or inverse Gamma if θ R1i is a scalar), the posterior distribution for mean parameters is normal and that for standard deviation parameter is inverse Wishart. There are simple procedures for making a draw from both distributions (e.g., see Train 2003).
We will discuss how to estimate an extension where p ijt is serially correlated in Section 4.4.
Suppose that the gift is a vase. Some consumers may value it highly, but others who already have several vases at home, may not.
With a slight abuse of notation, we use G j to denote the mean value of the gift at store j = 1, 2, and G i = (G i1, G i2) to denote the vector of the values of the gift for consumer i.
For the identification issue of this model, see Ching et al. (2012).
Here we propose to make one draw of price vector in each iteration. However, in practice, we find it useful to draw several price vectors in each iteration and store the average of pseudo-E ϵ max functions evaluated at these draws of price vectors. We will discuss this procedure in Appendix A.
In practice, however, it may not be worthwhile to compute the pseudo-likelihood at θ r − 1 in every iteration because the set of past pseudo-E ϵ max functions is updated by only one element in each iteration. Therefore, the pseudo-likelihood based on H r − 1 could be a good approximation for the pseudo-likelihood based on H r. We will discuss more details in Appendix A.
In terms of the notation in Section 2.5, μ = μ, G i = θ R1i , and θ c = θ R2.
Note that if q(.,.) is symmetric, the expression of the acceptance probability will be simplified to \(\lambda = \min\left\{\frac{\pi(G_{i}^{*r}|\mu^r) \tilde{L}_i^r(\mathsf{b}_i|\mathsf{s}_i,\mathsf{p}_i;G_{i}^{*r},\theta_c^{r-1})} {\pi(G_{i}^{r-1}|\mu^r) \tilde{L}_i^{r}(\mathsf{b}_i|\mathsf{s}_i,\mathsf{p}_i;G_{i}^{r-1},\theta_c^{r-1})},1\right\}\).
Note that both the common and individual-specific parts of the weights have already been computed separately in steps 4 and 5, and can thus be re-used here.
This curse of dimensionality problem is different from that of solving for a dynamic programming model, where it refers to the size of the state space increasing exponentially with the number of state variables and linearly with the number of values for each state variable.
In this exercise, we computed the pseudo-likelihood conditional on previously accepted parameter vector every time a candidate parameter vector was rejected.
References
Ackerberg, D. A. (2003). Advertising, learning, and consumer choice in experience good markets: An empirical examination. International Economic Review, 44(3), 1007–1040.
Ackerberg, D. A. (2009). A new use of importance sampling to reduce computational burden in simulation estimation. Quantitative Marketing and Economics, 7(4), 343–376.
Aguirregabiria, V., & Mira, P. (2002). Swapping the nested fixed point algorithm: A class of estimators for discrete Markov decision models. Econometrica, 70(4), 1519–1543.
Albert, J. H., & Chib, S. (1993). Bayesian analysis of binary and polychotomous response data. Journal of the American Statistical Association, 88, 669–679.
Allenby, G. M. (1994). An introduction to hierarchical Bayesian modeling. Tutorial Notes, Advanced Research Techniques Forum, American Marketing Association.
Allenby, G. M., & Lenk, P. J. (1994). Modeling household purchase behavior with logistic normal regression. Journal of the American Statistical Association, 89, 1218–1231.
Allenby, G. M., & Rossi, P. E. (2006). Hierarchical Bayes models: A practitioner’s guide. In R. Grover, & M. Vriens (Eds.), The handbook of marketing research. Newbury Park: Sage Publications.
Berry, S. T., Levinshohn, J., & Pakes, A. (1995). Automobile prices in market equilibrium. Econometrica, 63(4), 841–890.
Brown, M., & Flinn, C. J. (2011). Family law effects on divorce, fertility and child investment. Working paper, Department of Economics, New York University.
Černý, V. (1985). Thermodynamical approach to the travelling salesman problem: An efficient simulation algorithm. Journal of Optimization Theory and Applications, 45(1), 41–51.
Ching, A. T. (2010). A dynamic oligopoly structural model for the prescription drug market after patent expiration. International Economic Review, 51(4), 1175–1207.
Ching, A. T., Imai, S., Ishihara, M., & Jain, N. (2009). A dynamic model of consumer learning and forgetting. Work-in-progress, Rotman School of Management, University of Toronto.
Ching, A. T., Imai, S., Ishihara, M., & Jain, N. (2012). Identification of dynamic models of rewards program. Working paper, Rotman School of Management, University of Toronto.
Crawford, G. S., & Shum, M. (2005). Uncertainty and learning in pharmaceutical demand. Econometrica, 73(4), 1137–1174.
Diermeier, D., Keane, M. P., & Merlo, A. M. (2005). A political economy model of congressional careers. American Economic Review, 95, 347–373.
Erdem, T., Imai, S., & Keane, M. P. (2003). Brand and quality choice dynamics under price uncertainty. Quantitative Marketing and Economics, 1(1), 5–64.
Erdem, T., & Keane, M. P. (1996). Decision making under uncertainty: Capturing dynamic brand choice processes in turbulent consumer goods markets. Marketing Science, 15(1), 1–20.
Geweke, J., Houser, D., & Keane, M. P. (2001). Simulation based inference for dynamic multinomial choice models. In B. H. Baltagi (Ed.), A companion to theoretical econometrics (pp. 466–493). London: Blackwell.
Geweke, J. F., & Keane, M. P. (2000). Bayesian inference for dynamic discrete choice models without the need for dynamic programming. In R. Mariano, T. Schuermann, & M. J. Weeks (Eds.), Simulation based inference and econometrics: Methods and applications. Cambridge: Cambridge University Press.
Gönül, F., & Srinivasan, K. (1996). Estimating the impact of consumer expectations of coupons on purchase behavior: A dynamic structural model. Marketing Science, 15(3), 262–279.
Hartmann, W. R. (2006). Intertemporal effects of consumption and their implications for demand elasticity estimates. Quantitative Marketing and Economics, 4(4), 325–349.
Hendel, I., & Nevo, A. (2006). Measuring the implications of sales and consumer inventory behavior. Econometrica, 74(6), 1637–1673.
Hitsch, G. (2006). An empirical model of optimal dynamic product launch and exit under demand uncertainty. Marketing Science, 25(1), 25–50.
Hotz, J. V., & Miller, R. (1993). Conditional choice probabilities and the estimation of dynamic models. Review of Economic Studies, 60(3), 497–529.
Imai, S., Jain, N., & Ching, A. (2009a). Bayesian estimation of dynamic discrete choice models. Econometrica, 77(6), 1865–1899.
Imai, S., Jain, N., & Ching, A. (2009b). Supplement to ‘Bayesian estimation of dynamic discrete choice models’. Econometrica (Supplementary Material), 77. http://www.econometricsociety.org/ecta/Supmat/5658_proofs.pdf.
Imai, S., & Krishna, K. (2004). Employment, deterrence and crime in a dynamic model. International Economic Review, 45(3), 845–872.
Ishihara, M. (2011). Dynamic demand for new and used durable goods without physical depreciation. Ph.D. dissertation, Rotman School of Management, University of Toronto.
Keane, M. P., & Wolpin, K. I. (1994). The solution and estimation of discrete choice dynamic programming models by simulation and interpolation: Monte Carlo evidence. Review of Economics and Statistics, 76(4), 648–672.
Keane, M. P., & Wolpin, K. I. (1997). The career decisions of young men. Journal of Political Economy, 105, 473–521.
Kirkpatrick, S., Gelatt, C. D., & Vecchi, M. P. (1983). Optimization by simulated annealing. Science, 220, 671–680.
Lancaster, T. (1997). Exact structural inference in optimal job search models. Journal of Business and Economic Statistics, 15(2), 165–179.
McCulloch, R., & Rossi, P. E. (1994). An exact likelihood analysis of the multinomial probit model. Journal of Econometrics, 64, 207–240.
Norets, A. (2009). Inference in dynamic discrete choice models with serially correlated unobserved state variables. Econometrica, 77(5), 1665–1682.
Norets, A. (2010). Continuity and differentiability of expected value functions in dynamic discrete choice models. Quantitative Economics, 1(2), 305–322.
Osborne, M. (2011). Consumer learning, switching costs, and heterogeneity: A structural examination. Quantitative Marketing and Economics, 9(1), 25–70.
Pantano, J. (2008). Essays in applied microeconomics. Ph.D. dissertation, UCLA.
Roos, J. M. T., Mela, C. F., & Shachar, R. (2011). Hyper-media search and consumption. Working paper, Fuqua School of Business, Duke University.
Rossi, P. E., & Allenby, G. M. (1999). Marketing models of consumer heterogeneity. Journal of Econometrics, 89, 57–78.
Rossi, P. E., Allenby, G. M., & McCulloch, R. (2005). Bayesian statistics and marketing. Chichester: Wiley.
Rossi, P. E., McCulloch, R., & Allenby, G. M. (1996). The value of purchase history data in target marketing. Marketing Science, 15, 321–340.
Rust, J. (1987). Optimal replacement of gmc bus engines: An empirical model of Harold Zurcher. Econometrica, 55(5), 999–1033.
Rust, J. (1988). Maximum likelihood estimation of discrete control processes. SIAM Journal on Control and Optimization, 26(5), 1006–1024.
Rust, J. (1997). Using randomization to break the curse of dimensionality. Econometrica, 65(3), 487–516.
Santos, M. S., & Rust, J. (2004). Convergence properties of policy iteration. SIAM Journal on Control and Optimization, 42(6), 2094–2115.
Silverman, B. W. (1986). Density estimation for statistics and data analysis. London: Chapman and Hall.
Song, I., & Chintagunta, P. K. (2003). A micromodel of new product adoption with heterogeneous and forward looking consumers: Application to the digital camera category. Quantitative Marketing and Economics, 1(4), 371–407.
Sun, B. (2005). Promotion effect on endogenous consumption. Marketing Science, 24(3), 430–443.
Train, K. E. (2003). Discrete choice methods with simulation. Cambridge: Cambridge University Press. Available at http://elsa.berkeley.edu/books/choice2.html.
Walsh, B. (2004). Markov Chain Monte Carlo and Gibbs sampling. Lecture Notes for EEB 581, University of Arizona. http://nitro.biosci.arizona.edu/courses/EEB581-2004/handouts/Gibbs.pdf.
Yang, B., & Ching, A. (2010). Dynamics of consumer adoption of financial innovation: The case of ATM cards. Working paper, Rotman School of Management, University of Toronto. Available at SSRN: http://ssrn.com/abstract=1434722.
Acknowledgements
We thank Martin Burda, Monica Meireles, Matthew Osborne, Peter Rossi, Andrei Strijnev, K. Sudhir, S. Siddarth and two anonymous referees for their helpful comments. We also thank the participants of the UCLA Marketing Camp, SBIES conference, Marketing Science Conference, Marketing Dynamics Conference, UTD-FORMS Conference, Canadian Economic Association Meeting, Econometric Society Meeting and Ph.D. seminars at OSU’s Fisher College of Business, Yale School of Management, University of Groningen, University of Zurich and University of Southern California for their useful feedback. Hyunwoo Lim provided excellent research assistance. All remaining errors are ours. Andrew Ching and Susumu Imai acknowledge the financial support from SSHRC.
Author information
Authors and Affiliations
Corresponding author
Additional information
The computer codes (in C and Matlab) for implementing the Monte Carlo exercises are available upon request.
Appendices
Appendix A
In this appendix, we discuss some techniques that one can use in practice to reduce the computational burden further. While we will use the model without unobserved heterogeneity for illustration purpose, the same ideas apply to the model with unobserved heterogeneity.
1.1 A.1 Integration of iid price shocks
In the base model specification of the store choice model with reward programs, we assume that prices are iid normal random variable. When implementing the IJC algorithm, we propose to make one draw of price vector, \(\tilde{p}^r\), and store \(\tilde{\mathcal{W}}^r(s,\tilde{p}^r;\theta^{*r})\) in each iteration. Alternatively, we may draw a number of price vector in each iteration, \(\{\tilde{p}^m\}_{m=1}^M\), evaluate \(\bar{E}_{p'}\tilde{\mathcal{W}}^r(s,p';\theta^r)\) using
and store \(\bar{E}_{p'}\tilde{\mathcal{W}}^r(s,p';\theta^{*r})\) instead of \(\tilde{\mathcal{W}}^r(s,\tilde{p}^r;\theta^{*r})\). The expected value function can then be approximated as follows (correspond to step 3 in Section 4.3.1).
In this alternative approach, we integrate out price first, before using the kernel regression to obtain the pseudo expected value function \(\tilde{E}_{p'}^r\mathcal{W}(s,p';\theta^{*r})\). So this approach should allow us to achieve the same level of precision by using a smaller N. One potential advantage is that it saves us some memory when computing the weighted average. The additional cost is that we need to compute \(\bar{E}_{p'}\tilde{\mathcal{W}}^r\) in each MCMC iteration. In terms of computational time, we find that these two approaches are roughly the same in our example.
We should also note that in the present example where we assume prices are observed, one can use the observed prices as random realizations in computing \(\bar{E}_{p'}\tilde{\mathcal{W}}^r(s,p';\theta^{*r})\), provided that there are a sufficient number of observations for each s. The advantage of using this approach is that the pseudo-E ϵ max functions of the observed prices, \(\tilde{W}_j^r(s,\mathsf{p};\theta^{*r})\), are by-products of the likelihood function computation. So we can skip step 4(a) and (b) in Section 4.3.1.
1.2 A.2 Computation of \(\tilde{L}^r(\mathsf{b}|\mathsf{s},\mathsf{p};\theta^{r-1})\)
In Section 4.3.1, we propose to compute the pseudo-likelihood at previously accepted parameter vector, \(\tilde{L}^r(\mathsf{b}|\mathsf{s},\mathsf{p};\theta^{r-1})\), in each iteration. This is mainly because in IJC, the set of past pseudo-E ϵ max functions is updated in each iteration, and thus the pseudo-likelihood computed in the previous iteration, \(\tilde{L}^{r-1}(\mathsf{b}|\mathsf{s},\mathsf{p};\theta^{r-1})\), is different from \(\tilde{L}^r(\mathsf{b}|\mathsf{s},\mathsf{p};\theta^{r-1})\). However, in practice, the computation of pseudo-likelihood is the most time-consuming part in the algorithm. Moreover, the set of past pseudo-E ϵ max functions is updated only by one element in each iteration. Thus, we propose the following procedure, which avoids computing \(\tilde{L}^r(\mathsf{b}|\mathsf{s},\mathsf{p};\theta^{r-1})\) in every iteration.
Suppose that we are in step 3 of iteration r (Section 4.3.1). If we have accepted the candidate parameter value in iteration r − 1 (i.e., θ r − 1 = θ *(r − 1)), then use \(\tilde{L}^{r-1}(\mathsf{b}|\mathsf{s},\mathsf{p};\theta^{*(r-1)})\) as a proxy for \(\tilde{L}^r(\mathsf{b}|\mathsf{s},\mathsf{p};\theta^{r-1})\). Note that the calculations of \(\tilde{L}^r(\mathsf{b}|\mathsf{s},\mathsf{p};\theta^{r-1})\) and \(\tilde{L}^{r-1}(\mathsf{b}|\mathsf{s},\mathsf{p};\theta^{*(r-1)})\) only differ in one past pseudo-E ϵ max function, and \(\tilde{L}^{r-1}(\mathsf{b}|\mathsf{s},\mathsf{p};\theta^{*(r-1)})\) has already been computed in iteration r − 1. If we have rejected the candidate parameter vector (i.e., θ r − 1 = θ r − 2), then we could use \(\tilde{L}^{r-1}(\mathsf{b}|\mathsf{s},\mathsf{p};\theta^{r-2})\) as a proxy for \(\tilde{L}^{r}(\mathsf{b}|\mathsf{s},\mathsf{p};\theta^{r-1})\), and only compute \(\tilde{L}^{r}(\mathsf{b}|\mathsf{s},\mathsf{p};\theta^{r-1})\) once every several successive rejections. This procedure avoids using the pseudo-likelihood that is based on an old set of past pseudo-E ε max functions as a proxy for \(\tilde{L}^r(\mathsf{b}|\mathsf{s},\mathsf{p};\theta^{r-1})\). According to our experience, one can obtain a fairly decent reduction in computational time when using this approach.
Appendix B
In this appendix, we explain an alternative way to implement IJC when estimating the model with unobserved heterogeneity. The main goal of this alternative approach is to reduce the memory requirement and computational burden further. Instead of storing \(\{\theta_c^{*l}, \{G_{i}^{*l}, \tilde{\mathcal{W}}^l(.,p^l;G_i^{*l},\theta_c^{*l})\}_{i=1}^I\}_{l=r-N}^{r-1}\), one can store \(\{\theta_c^{*l}, G_{i'}^{*l}, \tilde{\mathcal{W}}^l(.,p^l;G_{i'}^{*l},\theta_c^{*l})\}_{l=r-N}^{r-1}\), where \(i' = r-I*int(\frac{r-1}{I})\); int(.) is an integer function that converts any real number to an integer by discarding its value after the decimal place. i′ is simply one way to “randomly” select a consumer’s pseudo-E ϵ max function to be stored in each iteration. When approximating the expected value function in, say step 4(b) in Section 4.3.2, we can then set
Note that we are using the same set of past pseudo-E ϵ max functions for all consumers here. If there is a large number of consumers in the sample, this approach, which is also independently adopted by Osborne (2011), can dramatically reduce the memory requirement and computational burden for implementing IJC.
This approach works because \(G_{i'}^{*l}\) is a random realization from a distribution that covers the support of the parameter space. This is one important requirement that ensures the pseudo-E ϵ max functions converge to the true ones in the proof of IJC.
Rights and permissions
About this article
Cite this article
Ching, A.T., Imai, S., Ishihara, M. et al. A practitioner’s guide to Bayesian estimation of discrete choice dynamic programming models. Quant Mark Econ 10, 151–196 (2012). https://doi.org/10.1007/s11129-012-9119-6
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11129-012-9119-6