Skip to main content

Ergodicity, Decisions, and Partial Information

Part of the Lecture Notes in Mathematics book series (SEMPROBAB,volume 2123)

Abstract

In the simplest sequential decision problem for an ergodic stochastic process X, at each time n a decision u n is made as a function of past observations \(X_{0},\ldots,X_{n-1}\), and a loss l(u n , X n ) is incurred. In this setting, it is known that one may choose (under a mild integrability assumption) a decision strategy whose pathwise time-average loss is asymptotically smaller than that of any other strategy. The corresponding problem in the case of partial information proves to be much more delicate, however: if the process X is not observable, but decisions must be based on the observation of a different process Y, the existence of pathwise optimal strategies is not guaranteed. The aim of this paper is to exhibit connections between pathwise optimal strategies and notions from ergodic theory. The sequential decision problem is developed in the general setting of an ergodic dynamical system \((\varOmega,\mathcal{B},\mathbf{P},T)\) with partial information \(\mathcal{Y}\subseteq \mathcal{B}\). The existence of pathwise optimal strategies grounded in two basic properties: the conditional ergodic theory of the dynamical system, and the complexity of the loss function. When the loss function is not too complex, a general sufficient condition for the existence of pathwise optimal strategies is that the dynamical system is a conditional K-automorphism relative to the past observations \(\bigvee _{n\geq 0}T^{n}\mathcal{Y}\). If the conditional ergodicity assumption is strengthened, the complexity assumption can be weakened. Several examples demonstrate the interplay between complexity and ergodicity, which does not arise in the case of full information. Our results also yield a decision-theoretic characterization of weak mixing in ergodic theory, and establish pathwise optimality of ergodic nonlinear filters.

Keywords

  • Pathwise Optimality
  • Complexity Assumptions
  • Weak Mixing
  • Stochastic Process Setting
  • Full Information Setting

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

This is a preview of subscription content, access via your institution.

Buying options

Chapter
USD   29.95
Price excludes VAT (Canada)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   39.99
Price excludes VAT (Canada)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   54.99
Price excludes VAT (Canada)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Notes

  1. 1.

    To be precise, our definitions are time-reversed with respect to the textbook definitions; however, T is a K-automorphism if and only if T −1 is a K-automorphism [31, p. 110], and the corresponding statement for weak mixing is trivial. Therefore, our definitions are equivalent to those in [31].

  2. 2.

    Non-dominated loss functions may also be of significant interest, see [24] for example. We will restrict attention to dominated loss functions, however, which suffice in many cases of interest.

  3. 3.

    In [1, Appendix II.B] it is shown that under a continuity assumption on the loss function l, the optimal asymptotic loss in the full information setting is given by E[inf u E[l(u, X 1) | X 0, X −1, ]]. However, a counterexample is given of a discontinuous loss function for which this expression does not yield the optimal asymptotic loss. The key difference with the expression for L given in Theorem 2.6 is that in the latter the essential infimum runs over \(u \in \mathbb{U}_{0}\), while it is implicit in [1] that the infimum in the above expression is an essential infimum over \(u \in \mathbb{U}_{-\infty,0}\). As the counterexample in [1] shows, these quantities need not coincide in the absence of continuity assumptions.

  4. 4.

    The pointwise separability assumption in [17, Corollary 1.4(2\(\Rightarrow \) 7)] is not needed here, as the essential supremum can be reduced to a countable supremum as in the proof of Lemma 2.5.

  5. 5.

    Some of the statements in [27] are time-reversed as compared to their counterparts stated here. However, as both the absolute regularity and the nondegeneracy assumptions are invariant under time reversal (cf. [30] for the former; the latter is trivial), the present statements follow immediately.

  6. 6.

    As particle filters employ a random sampling mechanism, the strategy \(\mathbf{\tilde{u}}^{N}\) is technically speaking not admissible in the sense of this paper: Π k N (and therefore \(\tilde{u}_{k}^{N}\)) depends also on auxiliary sampling variables ξ 0, , ξ k that are independent of Y 0, , Y k . However, it is easily seen that all our results still hold when such randomized strategies are considered. Indeed, it suffices to condition on (ξ k ) k ≥ 0, so that all our results apply immediately under the conditional distribution.

References

  1. P.H. Algoet, The strong law of large numbers for sequential decisions under uncertainty. IEEE Trans. Inform. Theory 40(3), 609–633 (1994)

    CrossRef  MathSciNet  MATH  Google Scholar 

  2. A. Bellow, V. Losert, The weighted pointwise ergodic theorem and the individual ergodic theorem along subsequences. Trans. Am. Math. Soc. 288(1), 307–345 (1985)

    CrossRef  MathSciNet  MATH  Google Scholar 

  3. D. Berend, V. Bergelson, Mixing sequences in Hilbert spaces. Proc. Am. Math. Soc. 98(2), 239–246 (1986)

    CrossRef  MathSciNet  MATH  Google Scholar 

  4. O. Cappé, E. Moulines, T. Rydén, Inference in Hidden Markov Models (Springer, New York, 2005)

    MATH  Google Scholar 

  5. P. Chigansky, R. van Handel, A complete solution to Blackwell’s unique ergodicity problem for hidden Markov chains. Ann. Appl. Probab. 20(6), 2318–2345 (2010)

    CrossRef  MathSciNet  MATH  Google Scholar 

  6. J.P. Conze, Convergence des moyennes ergodiques pour des sous-suites. In: Contributions au calcul des probabilités. Bull. Soc. Math. France, Mém. No. 35 (Soc. Math. France, Paris, 1973), pp. 7–15

    Google Scholar 

  7. D. Crisan, B. Rozovskiĭ (eds.), The Oxford Handbook of Nonlinear Filtering (Oxford University Press, Oxford, 2011)

    MATH  Google Scholar 

  8. P. Del Moral, M. Ledoux, Convergence of empirical processes for interacting particle systems with applications to nonlinear filtering. J. Theor. Probab. 13(1), 225–257 (2000)

    CrossRef  MATH  Google Scholar 

  9. C. Dellacherie, P.A. Meyer, Probabilities and Potential. C (North-Holland, Amsterdam, 1988)

    Google Scholar 

  10. R.M. Dudley, Uniform Central Limit Theorems (Cambridge University Press, Cambridge, 1999)

    CrossRef  MATH  Google Scholar 

  11. N. Etemadi, An elementary proof of the strong law of large numbers. Z. Wahrsch. Verw. Gebiete 55(1), 119–122 (1981)

    CrossRef  MathSciNet  MATH  Google Scholar 

  12. A. Grothendieck, Produits tensoriels topologiques et espaces nucléaires. Mem. Am. Math. Soc. 1955(16), 140 (1955)

    Google Scholar 

  13. P.R. Halmos, In general a measure preserving transformation is mixing. Ann. Math. (2) 45, 786–792 (1944)

    Google Scholar 

  14. R. van Handel, The stability of conditional Markov processes and Markov chains in random environments. Ann. Probab. 37(5), 1876–1925 (2009)

    CrossRef  MathSciNet  MATH  Google Scholar 

  15. R. van Handel, Uniform time average consistency of Monte Carlo particle filters. Stoch. Process. Appl. 119(11), 3835–3861 (2009)

    CrossRef  MATH  Google Scholar 

  16. R. van Handel, On the exchange of intersection and supremum of σ-fields in filtering theory. Isr. J. Math. 192, 763–784 (2012)

    CrossRef  MATH  Google Scholar 

  17. van Handel, R.: The universal Glivenko-Cantelli property. Probab. Theor. Relat. Fields 155(3–4), 911–934 (2013)

    CrossRef  MATH  Google Scholar 

  18. J. Hoffmann-Jørgensen, Uniform Convergence of Martingales. In: Probability in Banach spaces, 7 (Oberwolfach, 1988), Progr. Probab., vol. 21 (Birkhäuser Boston, Boston, 1990), pp. 127–137

    Google Scholar 

  19. O. Kallenberg, Foundations of Modern Probability, 2nd edn. (Springer, New York, 2002)

    CrossRef  MATH  Google Scholar 

  20. H. Kunita, Asymptotic behavior of the nonlinear filtering errors of Markov processes. J. Multivariate Anal. 1, 365–393 (1971)

    CrossRef  MathSciNet  Google Scholar 

  21. T. Lindvall, Lectures on the Coupling Method (Dover Publications, Mineola, 2002). Corrected reprint of the 1992 original

    Google Scholar 

  22. S. Meyn, R.L. Tweedie, Markov Chains and Stochastic Stability, 2nd edn. (Cambridge University Press, Cambridge, 2009)

    CrossRef  MATH  Google Scholar 

  23. J. Neveu, Discrete-Parameter Martingales (North-Holland, Amsterdam, 1975)

    MATH  Google Scholar 

  24. A.B. Nobel, On optimal sequential prediction for general processes. IEEE Trans. Inform. Theory 49(1), 83–98 (2003)

    CrossRef  MathSciNet  MATH  Google Scholar 

  25. D. Pollard, A User’S Guide to Measure Theoretic Probability (Cambridge University Press, Cambridge, 2002)

    MATH  Google Scholar 

  26. D.J. Rudolph, Pointwise and L 1 mixing relative to a sub-sigma algebra. Ill. J. Math. 48(2), 505–517 (2004)

    MathSciNet  MATH  Google Scholar 

  27. X.T. Tong, R. van Handel, Conditional ergodicity in infinite dimension (2012). Preprint

    Google Scholar 

  28. H. Totoki, On a class of special flows. Z. Wahrscheinlichkeitstheorie und Verw. Gebiete 15, 157–167 (1970)

    CrossRef  MathSciNet  Google Scholar 

  29. A.W. van der Vaart, J.A. Wellner, Weak Convergence and Empirical Processes (Springer, New York, 1996)

    CrossRef  MATH  Google Scholar 

  30. V.A. Volkonskiĭ, Y.A. Rozanov, Some limit theorems for random functions. I. Theor. Probab. Appl. 4, 178–197 (1959)

    CrossRef  Google Scholar 

  31. P. Walters, An Introduction to Ergodic Theory (Springer, New York, 1982)

    CrossRef  MATH  Google Scholar 

  32. T. Weissman, N. Merhav, Universal prediction of random binary sequences in a noisy environment. Ann. Appl. Probab. 14(1), 54–89 (2004)

    CrossRef  MathSciNet  MATH  Google Scholar 

  33. H. von Weizsäcker, Exchanging the order of taking suprema and countable intersections of σ-algebras. Ann. Inst. H. Poincaré Sect. B (N.S.) 19(1), 91–100 (1983)

    Google Scholar 

Download references

Acknowledgements

This work was partially supported by NSF grant DMS-1005575.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Ramon van Handel .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and Permissions

Copyright information

© 2014 Springer International Publishing Switzerland

About this chapter

Cite this chapter

van Handel, R. (2014). Ergodicity, Decisions, and Partial Information. In: Donati-Martin, C., Lejay, A., Rouault, A. (eds) Séminaire de Probabilités XLVI. Lecture Notes in Mathematics(), vol 2123. Springer, Cham. https://doi.org/10.1007/978-3-319-11970-0_18

Download citation