Embedding a state space model into a Markov decision process
- 216 Downloads
In agriculture Markov decision processes (MDPs) with finite state and action space are often used to model sequential decision making over time. For instance, states in the process represent possible levels of traits of the animal and transition probabilities are based on biological models estimated from data collected from the animal or herd.
State space models (SSMs) are a general tool for modeling repeated measurements over time where the model parameters can evolve dynamically.
In this paper we consider methods for embedding an SSM into an MDP with finite state and action space. Different ways of discretizing an SSM are discussed and methods for reducing the state space of the MDP are presented. An example from dairy production is given.
KeywordsState space model Markov decision process Sequential decision making Stochastic dynamic programming
Unable to display preview. Download preview PDF.
- Ausiello, G., Franciosa, P. G., & Frigioni, D. (2001). Directed hypergraphs: Problems, algorithmic results, and a novel decremental approach. In Lecture notes in computer science. Vol. 2202: Theoretical computer science: 7th Italian conference, ICTCS (pp. 312–328). Torino, Italy. Berlin: Springer. Google Scholar
- de Mol, R. M., Keen, A., Kroeze, G. H., & Achten, J. M. F. H. (1999). Description of a detection model for oestrus and diseases in dairy cattle based on time series analysis combined with a Kalman filter. Computers and Electronics in Agriculture, 22(2–3), 171–185. doi: 10.1016/s0168-1699(99)00016-2. CrossRefGoogle Scholar
- Feng, Z., Dearden, R., Meuleau, N., & Washington, R. (2004). Dynamic programming for structured continuous Markov decision problems. In AUAI ’04: Proceedings of the 20th conference on uncertainty in artificial intelligence (pp. 154–161). Arlington, Virginia, USA. Google Scholar
- Guestrin, C., Koller, D., Parr, R., & Venkataraman, S. (2003). Efficient solution algorithms for factored MDPs. Journal of Artificial Intelligence Research, 19, 399–468. Google Scholar
- Guestrin, C., Hauskrecht, M., & Kveton, B. (2004). Solving factored MDPs with continuous and discrete variables. In AUAI ’04: Proceedings of the 20th conference on uncertainty in artificial intelligence (pp. 235–242). Arlington, Virginia, USA. Google Scholar
- Jørgensen, E. (1992). Sow replacement: Reduction of state space in dynamic programming model and evaluation of benefit from using the model (Dina Research Report 6). National Institute of Animal Science. Google Scholar
- Kozlov, A. V., & Koller, D. (1997). Nonuniform dynamic discretization in hybrid networks. In The thirteenth conference on uncertainty in artificial intelligence (UAI-97) (pp. 314–325). 1–3 August 1997. Google Scholar
- Nielsen, L. R., Pretolani, D., & Andersen, K. A. (2009). Bicriterion shortest paths in stochastic time-dependent networks. In Lecture notes in economics and mathematical systems. Vol. 618: Multiobjective programming and goal programming (pp. 57–67). Berlin: Springer. doi: 10.1007/978-3-540-85646-7_6. CrossRefGoogle Scholar
- Stott, A. W., Jones, G. M., Humphry, R. W., & Gunn, G. J. (2005). Financial incentive to control paratuberculosis (Johne’s disease) on dairy farms in the United Kingdom. Veterinary Record, 156(26), 825–831. Google Scholar
- West, M., & Harrison, J. (1997). Bayesian forecasting and dynamic models (2nd ed). New York: Springer. Google Scholar