Improper versus finitely additive distributions as limits of countably additive probabilities

Abstract

The Bayesian paradigm with proper priors can be extended either to improper distributions or to finitely additive probabilities (FAPs). Improper distributions and diffuse FAPs can be seen as limits of proper distribution sequences for specific convergence modes. In this paper, we compare these two kinds of limits. We show that improper distributions and FAPs represent two distinct features of the limit behavior of a sequence of proper distribution. More specifically, an improper distribution characterizes the behavior of the sequence inside the domain, whereas diffuse FAPs characterizes how the mass concentrates on the boundary of the domain. Therefore, a diffuse FAP cannot be seen as the counterpart of an improper distribution. As an illustration, we consider several approach to define uniform FAP distributions on natural numbers as an equivalent of improper flat prior. We also show that expected logarithmic convergence may depend on the chosen sequence of compact sets.

Appendices

Appendix 1

We establish some lemmas useful to prove Theorem 1. The first one is straightforward.

Lemma 3

Let $\{\pi _n^{(1)}\}_n$ and $\{\pi _n^{(2)}\}_n$ be two sequences of proper distributions and $0\le \gamma _n\le 1$ be a sequence of scalars that converges to 0. Then, the sequence defined by ${\widetilde{\pi }}_n=\gamma _n\pi _n^{(1)}+(1-\gamma _n)\pi _n^{(2)}$ has the same FAP limit points as $\{\pi _n^{(2)}\}_n$.

Proof

For any $f_1,\ldots ,f_p\in {\mathcal {F}}_b$, then $(\pi _{n_k}^{(2)}(f_1),\ldots ,\pi _{n_k}^{(2)}(f_p) )$ converges to $(\pi (f_1),\ldots ,\pi (f_p))$ iff $({\widetilde{\pi }}_{n_k}(f_1),\ldots ,{\widetilde{\pi }}_{n_k}(f_p) )$ converges to $(\pi (f_1),\ldots ,\pi (f_p))$. The result follows. $\square$

Lemma 4

Let $\{\pi _n\}_n$ be a sequence of proper priors and $K_n$ be a non-decreasing sequence of compact sets such that $\lim _n \pi _n(K_n)=0$, then the sequence defined by ${\widetilde{\pi }}_n=\frac{1}{\pi _n(K_n^c)}\mathbb {1}_{K_n^c}\pi _n$ has the same FAP limit points as $\{\pi _n\}_n$.

Proof

First, note that $\{\pi _n\}_n$ is not defined when $\pi _n(K_n)=1$, but this cannot occur more than a finite number of times. For any $f\in {\mathcal {F}}_b$, $\pi _n(f)=$ $\mathbb {1}_{K_n}\pi _n(f)+\mathbb {1}_{K_n^c}\pi _n(f)=$ $\pi _n(\mathbb {1}_{K_n}f)+\pi _n(K_n^c){\widetilde{\pi }}_n(f)$. Since f is bounded, $\lim _n\pi _n(\mathbb {1}_{K_n}f)$ $=$ 0. Moreover, $\lim _n(K_n^c)=1$. Thus, for any $f_1,\ldots ,f_p\in {\mathcal {F}}_b$, $(\pi _{n_k}(f_1),\ldots ,\pi _{n_k} (f_p) )$ converges to $(\pi (f_1),\ldots ,\pi (f_p))$ iff $({\widetilde{\pi }}_{n_k}(f_1),\ldots ,{\widetilde{\pi }}_{n_k}(f_p) )$ converges to $(\pi (f_1),\ldots ,\pi (f_p))$. $\square$

At the opposite of Lemma 4, the following lemma shows that if we consider the restriction of a sequence $\{\pi _n\}_n$ of a proper or improper distribution on a exhaustive increasing sequence $\{K_n\}_n$ of compact sets, we preserve the q-vague limits.

Lemma 5

Let $K_n$ be a non-decreasing sequence of compact sets such that $\cup _n K_n=\varTheta$ and such that, for any compact K, there exists N such that $K\subset K_N$. A sequence $\{\pi _n\}_n$ of Radon measures converges q-vaguely to the Radon measure $\pi$ if and only if ${\widetilde{\pi }}_n=\frac{1}{\pi _n(K_n)} \mathbb {1}_{K_n}\pi _n$ converges q-vaguely to $\pi$.

Proof

Assume that $\pi _n$ converges q-vaguely to $\pi$, then there exists some positive scalars $\{a_n\}_n$ such that for any f in ${\mathcal {C}}_K$, $\lim _n a_n\pi _n(f)=\pi (f)$. Put ${\widetilde{a}}_n= a_n\,\pi _n(K_n)$ and denote by $K_f$ a compact set that includes the support of f. Then, there exists an integer N such that $K_f\subset K_n$ for $n>N$. Therefore, for $n>N$, ${\widetilde{a}}_n{\widetilde{\pi }}_n(f)=a_n\pi _n(f)$. The result and its reciprocal follow.$\square$

Appendix 2

We prove here Proposition 2 of Sect. 4.2.

In order to show that $\pi$ is SI-uniform, we consider $\pi _n$ as a distribution on the set of positive and negative integers, extending it by 0 on the non-positive integers. Define by $\pi ^{(k)}_n$ the shifted distribution: $\pi ^{(k)}_n (A) = \pi _n (A+k)$, for any subset A of the set of integers. One knows that $\Vert \pi ^{(k)}_n - \pi _n \Vert _{TV} \le \frac{k}{\sqrt{2\pi n}}$, where $\Vert \cdot \Vert _{TV}$ is the total variation norm. Therefore, for any subset of ${\mathbb {N}}$, $\lim _{n\rightarrow \infty } |\pi _n(A+k) - \pi _n(A) | =0$. Letting n go to infinity, we deduce that, for any FAP limit point $\pi$ of $\pi _n$, and any integer k: $\pi (A+k) = \pi (A)$.

The fact that $\pi$ is BS-uniform comes from an easy adaptation of the Hoeffding inequality in that context. Let $(X_k)_{k \in {\mathbb {N}}}$ be a Bernoulli scheme, of parameter p, and denote by ${\mathbb {P}}$ the associated probability. Hoeffding inequality gives, that, for any n:

$$\begin{aligned} {\mathbb {P}} \left\{ \bigg | \sum _{k=0}^{\infty } e^{-n}\frac{k^n}{n!} (X_k(\omega ) -p) \bigg | \ge t \right\} \le 2 e^{-2 c \sqrt{2\pi n}\, t^2}, \end{aligned}$$

for some positive constant c. The expected conclusion is then obtained thanks to the Borel-Cantelli lemma.

The fact that some of the limit points $\pi$ of $\{\pi _n\}_n$ are not LRF uniform is a direct consequence of the following lemma.

Lemma 6

For any $0\le p,p'\le 1$, there exists a set A and some FAP limit points ${\pi }$ of $\{{\pi }_n\}_n$ such that $LRF(A)=p$ and ${\pi }(A)=p'$.

Proof

First note that, for any set $A'$, $LRF(A')=p$ if, and only if, $\sharp \{ k \le n, \, k\in A'\} =$ $pn + o(n)$. Therefore, for any set A with $LRF(A)=p$ and for any set B such that $\sharp \{k\le n , \, k\in B\} = o(N)$, one has both $LRF(A\cup B) = p$ and $LRF(A{\setminus} B ) = p$. Take now for set B the following:

$$\begin{aligned} B = \bigcup _{k\in {\mathbb {N}}} \big \{ u\in {\mathbb {N}}{:}\; 4^k -2^k k \le u \le 4^k+2^k k \big \}. \end{aligned}$$

For that B, one has:

$$\begin{aligned} \limsup _{n\rightarrow \infty } \frac{\sharp \{k\le n, \, k \in B\}}{n+1} = \lim _{k \rightarrow \infty } \frac{\sum _{i=0}^{k} 2^{i+1} i }{4^k + 2^k k} \le \lim _{k \rightarrow \infty } \frac{(k+1)2^{k+2}}{4^k } = 0, \end{aligned}$$

and thus $LRF(B) =0$. However, $\pi _{4^k} (B)$ converges to 1. Indeed, if $U_k$ is some random variable with law $\pi _{4^k}$, one has:

$$\begin{aligned} \pi _{4^k} \big ( \big \{ u\in {\mathbb {N}}{:}\; 4^k -2^k k \le u \le 4^k+2^k k \big \} \big ) = {\mathbb {P}} \bigg ( \frac{U_k -4^k}{\sqrt{4^k}} \in \big [ - k{;}\,k \big ] \bigg ). \end{aligned}$$

The right-hand side term above converges to 1 thanks to the central limit theorem. Hence $LRF(A\cup B) = LRF(A{\setminus} B) = p$ while $\pi _{4^k}(A\cup B)$ converges to 1, and $\pi _{4^k}(A{\setminus} B)$ converges to 0. Now, for any $p'\in [0;1]$, choose two numbers $a<b$, so that $p'=\int _a^b \frac{e^{-u^2/2}}{\sqrt{2\pi }} {\mathrm{d}}u$. Take the set $B'$ to be:

$$\begin{aligned} B' = \bigcup _{k\in {\mathbb {N}}} \big \{ u\in {\mathbb {N}}{:}\; 4^k +2^k \text {max}(-k,a) \le u \le 4^k+2^k \text {min}(k,b) \big \}, \end{aligned}$$

then $LRF(B')=0$ again and $\pi _{4^k} (B')$ converges to $p'$ , still thanks to the central limit theorem. Let $A = (A'{\setminus} B)\cup B'$. Then $LRF (A) =p$ and $\lim _{k \rightarrow \infty } \pi _{4^k} (A) =p'$. Now, any FAP limit point $\pi$ of subsequence $\{\pi _{4^k}\}_k$ is also a FAP limit point of $\{\pi _k\}_k$. Hence, $\pi$ is SI-uniform and BS-uniform, but one has $\pi (A)= p'$ and $LRF(A)=p$.$\square$