Skip to main content
SpringerLink
Log in

Robust Dynamics and Control of a Partially Observed Markov Chain

  • Published:
Applied Mathematics and Optimization Submit manuscript

Abstract

In a seminal paper, Martin Clark (Communications Systems and Random Process Theory, Darlington, 1977, pp. 721–734, 1978) showed how the filtered dynamics giving the optimal estimate of a Markov chain observed in Gaussian noise can be expressed using an ordinary differential equation. These results offer substantial benefits in filtering and in control, often simplifying the analysis and an in some settings providing numerical benefits, see, for example Malcolm et al. (J. Appl. Math. Stoch. Anal., 2007, to appear).

Clark’s method uses a gauge transformation and, in effect, solves the Wonham-Zakai equation using variation of constants. In this article, we consider the optimal control of a partially observed Markov chain. This problem is discussed in Elliott et al. (Hidden Markov Models Estimation and Control, Applications of Mathematics Series, vol. 29, 1995). The innovation in our results is that the robust dynamics of Clark are used to compute forward in time dynamics for a simplified adjoint process. A stochastic minimum principle is established.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Bismut, J.-M.: An introductory approach to duality in optimal stochastic control. SIAM Rev. 20(1), 62–78 (1978)

    Article  MATH  MathSciNet  Google Scholar 

  2. Clark, J.M.C.: The design of robust approximations to the stochastic differential equations for nonlinear filtering. In: Skwirzynski, J.K. (ed.) Communications Systems and Random Process Theory, Darlington, 1977, pp. 721–734. Sijthoff and Noorhoff, Alphen aan den Rijn (1978)

    Google Scholar 

  3. Davis, M.H.A.: Martingale methods in stochastic control. In: Stochastic Control Theory and Stochastic Differential Systems, pp. 85–117. Springer, New York (1979)

    Chapter  Google Scholar 

  4. Elliott, R.J.: A stochastic minimum principle. Bull. Am. Math. Soc. 82, 944–946 (1976)

    Article  MATH  Google Scholar 

  5. Elliott, R.J.: Stochastic Calculus and its Applications. Springer, Berlin (1982)

    Google Scholar 

  6. Elliott, R.J.: A partially observed control problem for Markov chains. Appl. Math. Optim. 25, 151–169 (1992)

    Article  MATH  MathSciNet  Google Scholar 

  7. Elliott, R.J., Moore, J.P.: Adjoint processes for Markov chains observed in Gaussian Noise. In: 26th Asilomar Conference on Signals Systems and Computing, vol. 1, pp. 396–399. California, USA, October 1992

  8. Elliott, R.J., Aggoun, L., Moore, J.P.: Hidden Markov Models Estimation and Control. Applications of Mathematics Series, vol. 29. Springer, Berlin (1995)

    MATH  Google Scholar 

  9. Fleming, W.H.: Optimal continuous-parameter stochastic control. SIAM Rev. 11(4) (October 1969)

  10. Haussmann, U.G.: Some examples of optimal stochastic controls or: the stochastic maximum principle at work. SIAM Rev. 23(3), 292–307 (1981)

    Article  MATH  MathSciNet  Google Scholar 

  11. James, M.R., Krishnamurthy, V., Le Gland, F.: Time discretization of continuous-time filters and smoothers for HMM parameter estimation. IEEE Trans. Inform. Theory 42(2), 593–605 (1996)

    Article  MATH  Google Scholar 

  12. Malcolm, W.P., Elliott, R.J., van der Hoek, J.: A deterministic discretisation-step upper bound for state estimation via Clark transformations. J. Appl. Math. Stoch. Anal. 371–384 (2004)

  13. Malcolm, W.P., Elliott, R.J., van der Hoek, J.: On the numerical stability of time-discretised state estimation via Clark transformations. In: 42nd IEEE Conference on Decision and Control, pp. 1406–1412, Mauii, Hawaii, December 2003

Download references

Author information

Authors and Affiliations

  1. Haskayne School of Business, Scurfield Hall, University of Calgary, 2500 University Drive NW, Calgary, AB, Canada, T2N 1N4

    R. J. Elliott

  2. National ICT Australia, Locked Bag 8001, Canberra, ACT, 2601, Australia

    W. P. Malcolm & J. P. Moore

Authors
  1. R. J. Elliott
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. W. P. Malcolm
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. J. P. Moore
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to R. J. Elliott.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Elliott, R.J., Malcolm, W.P. & Moore, J.P. Robust Dynamics and Control of a Partially Observed Markov Chain. Appl Math Optim 56, 303–311 (2007). https://doi.org/10.1007/s00245-007-9007-8

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00245-007-9007-8

Keywords

Search

Navigation