Recurrence times, waiting times and universal entropy production estimators

Abstract

The universal typical-signal estimators of entropy and cross-entropy based on the asymptotics of recurrence and waiting times play an important role in information theory. Building on their construction, we introduce and study universal typical-signal estimators of entropy production in the context of nonequilibrium statistical mechanics of one-sided shifts over finite alphabets.

Appendix A

Proof of Theorem 6.1

One follows the steps of the proof of Theorem 3.4 in Sect. 5, with the following changes. In Step 1, the ${\mathbb {P}}$-almost sure validity of (4) now relies on Hypothesis i and adaptations of Fekete’s lemma and Kingman’s theorem to the corresponding generalized subadditivity condition which are discussed in [41]. The set $E_{n, \epsilon }$ is defined in the same way, but one takes

$$\begin{aligned} B_{n, \epsilon }:=\left\{ x:\, {{\widehat{R}}}_{n}(x)\le \text {e}^{-\log {{\widehat{{\mathbb {P}}}}}_{n-\tau _n}(x_1^n)-n\epsilon }\right\} . \end{aligned}$$

In Step 2, one starts with

$$\begin{aligned} {\mathbb {P}}( B_{n,\epsilon } ) = \sum _{a \in {{\,\textrm{supp}\,}}{\mathbb {P}}_n} \sum _{j=1}^{\lfloor \text {e}^{- \log {\widehat{{\mathbb {P}}}}_{n-\tau _n}(a_1^{n-\tau _n}) - n \epsilon }\rfloor }{\mathbb {P}}\left( \left\{ x : {\widehat{R}}_n(x) = j, x_1^n=a \right\} \right) , \end{aligned}$$

for n large enough, and estimates

$$\begin{aligned} {\mathbb {P}}(B_{n,\epsilon })\le & {} \sum _{a \in {{\,\textrm{supp}\,}}{\mathbb {P}}_n} \sum _{j=1}^{\lfloor \text {e}^{- \log {\widehat{{\mathbb {P}}}}_{n-\tau _n}(a_1^{n-\tau _n}) - n \epsilon }\rfloor }\sum _{\zeta \in {\mathcal {A}}^{j-1}} {\mathbb {P}}_{n+j-1+n} (a \zeta \, {\widehat{a}})\nonumber \\\le & {} \sum _{a \in {{\,\textrm{supp}\,}}{\mathbb {P}}_{n-\tau _n}}\sum _{b, b^\prime \in {{{\mathcal {A}}}}^{\tau _n}} \sum _{j=1}^{\lfloor \text {e}^{- \log {\widehat{{\mathbb {P}}}}_{n-\tau _n}(a) - n \epsilon }\rfloor } \sum _{\zeta \in {\mathcal {A}}^{j-1}} {\mathbb {P}}_{n+j-1+n} (ab \zeta \, b^\prime {\widehat{a}}) \nonumber \\\le & {} \text {e}^{2c_{n-\tau _n}}\sum _{a \in {{\,\textrm{supp}\,}}{\mathbb {P}}_{n-\tau _n}} \sum _{j=1}^{\lfloor \text {e}^{- \log {\widehat{{\mathbb {P}}}}_{n-\tau _n}(a) - n \epsilon }\rfloor }\sum _{\zeta \in {\mathcal {A}}^{j-1}} {\mathbb {P}}_{j-1} ( \zeta ){\mathbb {P}}_{n-\tau _n}(a){{\widehat{{\mathbb {P}}}}}_{n-\tau _n}( {a})\,\nonumber \\\le & {} \text {e}^{2c_{n-\tau _n}}\sum _{a\in {{\,\textrm{supp}\,}}{\mathbb {P}}_{n-\tau _n}} \text {e}^{-\log {\widehat{{\mathbb {P}}}}_{n-\tau _n}(a)-n\epsilon }{\mathbb {P}}_{n-\tau _n}(a) {\widehat{{\mathbb {P}}}}_{n-\tau _n}(a)\nonumber \\ \,= & {} \text {e}^{2c_{n-\tau _n}-n\epsilon }. \end{aligned}$$

(24)

The final estimate and the assumption that $c_n$ and $\tau _n$ are o(n) give the desired summability condition. In Step 3, one first chooses, for n large enough, a natural number m(a) so that

$$\begin{aligned} \text {e}^{-\log {\widehat{{\mathbb {P}}}}_n(a) + \frac{1}{2} n\epsilon } \le m(a) - n \le m(a) - n + \tau _n <\text {e}^{-\log {\widehat{{\mathbb {P}}}}_n(a) + n\epsilon } \end{aligned}$$

(25)

and replaces (17) with the estimates

$$\begin{aligned} {\mathbb {P}}(E_{n,\epsilon })&\le \sum _{a\in {{\,\textrm{supp}\,}}{\mathbb {P}}_n}\sum _{b^\prime \in {{{\mathcal {A}}}}^{\tau _n}}\sum _{b \in {\mathcal {A}}^{m}} {\mathbb {P}}\left( \left\{ x: {\widehat{R}}_n(x) > m-n+\tau _n \text { and } x_{1}^{n+\tau _n+m}=ab^\prime b \right\} \right) \\ {}&\le \sum _{a\in {{\,\textrm{supp}\,}}{\mathbb {P}}_n}\sum _{b^\prime \in {{{\mathcal {A}}}}^{\tau _n}} \sum _{b \in {\mathcal {A}}^{m}} \\&\quad {\mathbb {P}}\left( \left\{ x: x_1^{n+\tau _n+m}=ab^\prime b \text { and } b_k^{k+n-1}\not ={{\widehat{a}}} \text { for } 1\le k \le m-n\right\} \right) \\\,&\le \text {e}^{c_n}\sum _{a\in {{\,\textrm{supp}\,}}{\mathbb {P}}_n} \sum _{b \in {\mathcal {A}}^m} {\mathbb {P}}_n\times {\mathbb {P}}_m\left( \left\{ (a, b): b_k^{k+n-1}\not ={{\widehat{a}}} \text { for } 1\le k \le m-n \right\} \right) . \end{aligned}$$

At this point, one proceeds in exactly the same way as in Sect. 5 to derive the estimate

$$\begin{aligned} {\mathbb {P}}(E_{n, \epsilon })\le \text {e}^{c_n} {\mathbb {P}}\times {{\widehat{{\mathbb {P}}}}} \left( \left\{ (x, y): W_n(x, y)> \text {e}^{-\log {\widehat{{\mathbb {P}}}}_n(x_1^n) + \frac{1}{2} n\epsilon } \right\} \right) , \end{aligned}$$

which, combined with Hypothesis ii, yields the desired summability. $\square$

Proof of Theorem 6.2

One again follows the same strategy. In Step 1, the ${\mathbb {P}}$-almost sure validity of (4) now relies on Hypothesis i”. In Step 2, the sets $B_{n, \epsilon }$ are the same as in (13) and one starts with the identity

$$\begin{aligned} {\mathbb {P}}( B_{n,\epsilon } )&= \sum _{a \in {{\,\textrm{supp}\,}}{\mathbb {P}}_n} \sum _{j=1}^{\lfloor \text {e}^{- \log {\widehat{{\mathbb {P}}}}_n(a) - n \epsilon }\rfloor }{\mathbb {P}}(\{ x : {\widehat{R}}_n(x) = j \} \cap [a]), \end{aligned}$$

which gives

$$\begin{aligned} {\mathbb {P}}( B_{n,\epsilon } )&\le \sum _{a \in {{\,\textrm{supp}\,}}{\mathbb {P}}_n} \sum _{j=1}^{\lfloor \text {e}^{- \log {\widehat{{\mathbb {P}}}}_n(a) - n \epsilon }\rfloor }{\mathbb {P}}([a_1^{n-\tau _{n,K}}] \cap \{ x : {\widehat{R}}_n(a\sigma ^{n}(x) = j \}) \\&= \sum _{a \in {{\,\textrm{supp}\,}}{\mathbb {P}}_n} {\mathbb {P}}\left( \sigma ^{-\tau _{n, K}}[a_1^{n-\tau _{n,K}}] \cap \bigcup _{j=1}^{\lfloor \text {e}^{- \log {\widehat{{\mathbb {P}}}}_n(a) - n \epsilon } \rfloor }\sigma ^{-\tau _{n, K}}\{ x : {\widehat{R}}_n(a\sigma ^{n}(x)) = j \}\right) \end{aligned}$$

as soon as n is large enough that $n - \tau _{n,K} \ge 1$. (This is possible because $\tau _{n,K}$ is o(n).) Since each of the $\lfloor \text {e}^{- \log {\widehat{{\mathbb {P}}}}_n(a) - n \epsilon }\rfloor$ sets of the form $\sigma ^{-\tau _{n,K}}\{ x: {\widehat{R}}_n(a\sigma ^{n}(x)) = j \}$ is also necessarily of the form $\sigma ^{- n - \tau _{n,K}}(B)$ for some $B \in {\mathcal {F}}_{\text {fin}}$ and has probability bounded above by ${\widehat{{\mathbb {P}}}}_n(a)$, Hypothesis i’ gives that

$$\begin{aligned} \begin{aligned} {\mathbb {P}}( B_{n,\epsilon } )&\le \sum _{a \in {{\,\textrm{supp}\,}}{\mathbb {P}}_n} \left( \text {e}^{c_{n, K}} {\mathbb {P}}_{n-\tau _{n, K}}(a_1^{n-\tau _{n,K}}) \text {e}^{- \log {\widehat{{\mathbb {P}}}}_n(a) - n \epsilon } {\widehat{{\mathbb {P}}}}_n(a) + \text {e}^{-nK} \right) \\\,&\le \text {e}^{c_{n, K} + \tau _{n,K} \log |{\mathcal {A}}| - n\epsilon } + \text {e}^{n\log |{\mathcal {A}}|-nK}, \end{aligned} \end{aligned}$$

where we used ${{\,\textrm{supp}\,}}{\mathbb {P}}_n \subseteq {{\,\textrm{supp}\,}}{\mathbb {P}}_{n-\tau _{n,K}} \times {\mathcal {A}}^{\tau _{n,K}}$ to get the second inequality. If $K > \log |{\mathcal {A}}|$, the last estimate gives the desired summability since both $c_{n,K}$ and $\tau _{n,K}$ are o(n). We now turn to Step 3. With $E_{n, \epsilon }$ as in (13) and

$$\begin{aligned} G_{n,K} := \left\{ x : {\widehat{{\mathbb {P}}}}([x_1^n] )\ge \text {e}^{- nK} \right\} , \end{aligned}$$

it suffices to show that for all $\epsilon > 0$ and all $K \in \mathbb {N}$ large enough, the summability condition

$$\begin{aligned} \sum _{n = 1}^\infty {\mathbb {P}}(E_{n,\epsilon } \cap G_{n,K}) < \infty \end{aligned}$$

(26)

holds.⁶ The identity

$$\begin{aligned} {\mathbb {P}}(E_{n,\epsilon } \cap G_{n,K})&= \sum _{a \in {\mathcal {A}}^n} {\mathbb {P}}\left( \left\{ x \in G_{n,K} : {\widehat{R}}_n(x) \ge \text {e}^{-\log {\widehat{{\mathbb {P}}}}_n(a) + n\epsilon } \right\} \cap [a]\right) \end{aligned}$$

gives

$$\begin{aligned} {\mathbb {P}}(E_{n,\epsilon } \cap G_{n,K}) \le \sum _{\begin{array}{c} a \in {{\,\textrm{supp}\,}}{\mathbb {P}}_n \\ {\widehat{{\mathbb {P}}}}_n(a) \ge \text {e}^{-nK} \end{array}} {\mathbb {P}}\left( [a] \cap \bigcap _{k=1}^{\lfloor \text {e}^{-\log {\widehat{{\mathbb {P}}}}_n(a) + n\epsilon } \rfloor -1}\sigma ^{-n-k+1} ([{{\widehat{a}}}]^{{\textsf{C}}})\right) . \end{aligned}$$

(27)

Setting

$$\begin{aligned} C_{n, K, \epsilon }(a):=\lfloor \text {e}^{-\log {\widehat{{\mathbb {P}}}}_n(a) + n\epsilon } \rfloor -\tau _{n,K}-1, \end{aligned}$$

and using an obvious inclusion, $\sigma$-invariance and then Hypothesis i’, one derives

$$\begin{aligned} \begin{aligned}&{\mathbb {P}}\left( [a] \cap \bigcap _{k=1}^{\lfloor \text {e}^{-\log {\widehat{{\mathbb {P}}}}_n(a) + n\epsilon } \rfloor -1}\sigma ^{-n-k+1} ([{{\widehat{a}}}]^{{\textsf{C}}})\right) \\&\quad \le {\mathbb {P}}\left( [a] \cap \bigcap _{k=\tau _{n,K}+1}^{\lfloor \text {e}^{-\log {\widehat{{\mathbb {P}}}}_n(a) + n\epsilon } \rfloor -1}\sigma ^{-n-k+1} ([{{\widehat{a}}}]^{{\textsf{C}}})\right) \\&\quad \le \text {e}^{c_{n,K}} {\mathbb {P}}([a] ){\mathbb {P}}\left( \bigcap _{k=1}^{C_{n, K, \epsilon }(a)} \sigma ^{-k+1} ([{{\widehat{a}}}]^{{\textsf{C}}})\right) + \text {e}^{-Kn}. \end{aligned} \end{aligned}$$

(28)

The identities

$$\begin{aligned}&{\mathbb {P}}\left( \left\{ y : y_{k}^{k+n-1} \ne {\widehat{a}}, \, 1\le k \le C_{n,K, \epsilon }(a)\right\} \right) \\&\quad = {\mathbb {P}}_{C_{n, K, \epsilon }+ n}\left( \left\{ b : b_k^{k+n-1} \ne {\widehat{a}}, \,1\le k \le C_{n,K, \epsilon }(a)\right\} \right) \\\,&\quad = {\mathbb {P}}_{C_{n, K, \epsilon } + n}\left( \left\{ {\widehat{b}} : b_k^{k+n-1} \ne a, \,1\le k \le C_{n, K, \epsilon }(a)\right\} \right) \\\,&\quad = {\widehat{{\mathbb {P}}}}_{C_{n, K, \epsilon }+ n}\left( \left\{ b : b_k^{k+n-1} \ne a, \,1\le k \le C_{n, k, \epsilon }(a)\right\} \right) , \end{aligned}$$

and

$$\begin{aligned} \begin{aligned} {\mathbb {P}}([a]){\widehat{{\mathbb {P}}}}_{C_{n,K, \epsilon }(a) + n}&\left( \left\{ b: b_k^{k+n-1}\not = a,\,1\le k \le C_{n, K, \epsilon }(a)\right\} \right) \\\,&= ({\mathbb {P}}\times {\widehat{{\mathbb {P}}}})\left( \left\{ (x,y): W_n(x,y) > C_{n,K,\epsilon }(x_1^n)\right\} \cap ([a] \times \Omega )\right) , \end{aligned} \end{aligned}$$

further give

$$\begin{aligned}{} & {} {\mathbb {P}}(B_{n,\epsilon } \cap G_{n,K}) \nonumber \\{} & {} \quad \le \sum _{\begin{array}{c} a \in {{\,\textrm{supp}\,}}{\mathbb {P}}_n \\ {\widehat{{\mathbb {P}}}}_n(a) \ge \text {e}^{- nK} \end{array}} \text {e}^{c_{n,K}} ({\mathbb {P}}\times {\widehat{{\mathbb {P}}}})\left( \left\{ (x,y): W_n(x,y)> C_{n, K, \epsilon }(x_1^n)\right\} \cap ([a] \times \Omega )\right) + \text {e}^{-nK}\nonumber \\{} & {} \quad \le \text {e}^{c_{n,K}}({\mathbb {P}}\times {\widehat{{\mathbb {P}}}})\left( \left\{ (x,y): {\widehat{{\mathbb {P}}}}([x_1^n]) \ge \text {e}^{-nK} \text { and } W_n(x,y) > C_{n,K, \epsilon }(x_1^n) \right\} \right) \nonumber \\{} & {} \qquad + \text {e}^{n\log |{\mathcal {A}}|-nK}. \end{aligned}$$

(29)

Note that for n large enough,

$$\begin{aligned} C_{n, K, \epsilon }(x_1^n) > \text {e}^{-\log {\widehat{{\mathbb {P}}}}([x_1^n]) + \frac{1}{2} n\epsilon } \end{aligned}$$

(30)

for all $x_1^n\in {{{\mathcal {A}}}}^n$. Taking $K>\log |{\mathcal {A}}|$, the estimates (29)–(30) and Hypothesis ii’ give the summability condition (26). $\square$