Turning a Coin over Instead of Tossing It

Given a sequence of numbers (pn)n≥2\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(p_n)_{n\ge 2}$$\end{document} in [0, 1], consider the following experiment. First, we flip a fair coin and then, at step n, we turn the coin over to the other side with probability pn\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p_n$$\end{document}, n≥2\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n\ge 2$$\end{document}, independently of the sequence of the previous terms. What can we say about the distribution of the empirical frequency of heads as n→∞\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n\rightarrow \infty $$\end{document}? We show that a number of phase transitions take place as the turning gets slower (i. e., pn\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p_n$$\end{document} is getting smaller), leading first to the breakdown of the Central Limit Theorem and then to that of the Law of Large Numbers. It turns out that the critical regime is pn=const/n\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p_n=\text {const}/n$$\end{document}. Among the scaling limits, we obtain uniform, Gaussian, semicircle, and arcsine laws.


General Model
In this paper, we examine what happens if, instead of tossing a coin, we turn it over (from heads to tails and from tails to heads), with certain probabilities.
To define the model precisely, let p n , n = 2, 3, . . . be a given deterministic sequence of numbers between 0 and 1. We define a time-dependent "coin turning process" X with X n ∈ {0, 1}, n ≥ 1, as follows. Let X 1 = 1 ("heads") or = 0 ("tails") with probability 1/2. For n ≥ 2, set recursively X n := 1 − X n−1 , with probability p n ; X n−1 , otherwise, that is, we turn the coin over with probability p n and do nothing with probability 1 − p n , independently of the sequence of the previous terms. Consider 1 N N n=1 X n , that is, the empirical frequency of 1's ("heads") in the sequence of X n 's. We are interested in the asymptotic behavior, in law, of this random variable as N → ∞.
Since we are interested in limit theorems, we center the variable X n ; for convenience, we also multiply it by two, thus focus on Y n := 2X n − 1 ∈ {−1, +1} instead of X n . We have and hence if i = 1, then we get E(Y j ) = e 1, j E(Y 1 ). In particular, since we assumed p 1 = 1/2, E(Y 1 ) as well as all consecutive E(Y j ) equal 0 for all j ≥ 2. In fact, for arbitrary { p n , n ≥ 1} satisfying p n = 1/2, n ≥ 2, the entire sequence (Y n ) n≥1 is centered in expectation (equivalently, E(X n ) = 1/2, n ≥ 1) if and only if p 1 = 1/2. Throughout the paper for N ≥ 1, we set T N := X 1 + · · · + X N , S N := Y 1 + · · · + Y N .
Then S N = 2T N − N and hence limit theorems we establish below for S N /N will easily imply analogous results for T N /N = S N /(2N ) + 1/2. At a more elementary level, we first observe that S N is symmetric in distribution around zero (hence its odd moments vanish), and as a result, T N is symmetric about N /2. The symmetry of the law of S N about zero follows from the symmetry in the definition of the model. In fact, a straightforward calculation gives Ee it S N = E cos(U N t), where U N := 1 + (−1) W 2 + (−1) W 2 +W 3 + · · · + (−1) W 2 +W 3 +···+W N . Using Corr and Cov for correlation and covariance, respectively, one also has Corollary 1 (Correlation estimate) Assume that lim k→∞ p k = 0 and let n * ∈ N be such that p k ≤ 1/2 for k ≥ n * . For n * ≤ i < j, where r k := 2 p 2 k e 2 p k , which is tending to zero rapidly. Furthermore, for any given C > 1, there exists an n * ∈ N such that for n * ≤ i < j, Proof Use the Remainder Theorem for Taylor series, yielding and multiply these inequalities, to get the first statement.
For the second statement, use that for sufficiently small positive x, Similarly to (1), if K = 2m is a positive even number, and i 1 < i 2 < · · · < i K then, using the fact that we obtain that We close this section with introducing some frequently used notation. Notation: In the sequel, Bessel I α and Bessel K α will denote the modified Bessel function of the first kind (or Bessel-I function) and the modified Bessel function of the second kind (or Bessel-K function), respectively.
Writing out these functions explicitly, one has if α is not an integer (otherwise it is defined through limits), where is Euler's gamma function. See, e.g., Sections 9-10 in [1], and formula (6.8) in [2].

Review of Literature and Comparison with Our Results
As the Associate Editor kindly pointed out for us, the problem has a history, going back to at least the 1950s. Below we give a review of the relevant achievements in the past and compare them to the results presented in this paper. The case p n = 1/n was already introduced in R. Dobrushin's thesis 1 in the 1950s and it is attributed to Bernstein (see [4]). Dobrushin did not seem to explicitly identify the limiting frequency of heads with the uniform distribution. However, in their 2007 paper [3] Dietz and Sethuraman proved that for the more general p n = a/n case (a > 0), the limiting frequency is Beta(a, a)-see their Theorems 1.3 and 1.4. (They consider a state space consisting of m ≥ 2 points and so they treat the more general Dirichlet distributions). Therefore, in the p n = a/n case, our contribution is providing a different proof only. The proof in [3] is significantly longer and more complicated than ours; however, we only consider m = 2.
The case p n = a/n γ is also treated in [3], albeit only the Weak Law of Large Numbers (SLLN when 0 < γ < 1/2). The authors note that simulations suggest that actually a.s. convergence might hold also on the range 1/2 ≤ γ < 1; in our case, we prove this statement in our Corollary 2. Fluctuations about the mean are not considered in [3] though.
The situation with the Central Limit Theorem is more interesting in this case. First, Dobrushin's Central Limit Theorem for inhomogeneous Markov chains (Theorem 1.1 in [11]) only provides the statement in the sense that, after centering and normalizing with the standard deviation, the limit is standard normal. We did not find, however, any result in the literature identifying the order of the standard deviation, which we do provide. (See the estimates on p. 411 and also Corollary 15 on p. 421 in [10]).
Secondly, and more importantly, Dobrushin's Theorem only applies to the case when 0 < γ < 1/3. Although the condition given in that result (formula (1.3) in [11]) is known to be optimal (see Section 2 in [11]), this is only true in the very general setting in which the theorem is stated. It is therefore interesting, we believe, that we also prove the CLT in the 1/3 ≤ γ < 1 case excluded in Dobrushin's result. (In [10], Dobrushin's condition was improved by Peligrad, but it is still not applicable in our case when 1/3 ≤ γ < 1: formula (7) in [10] is actually more stringent then the Dobrushin condition (9)).
To the best of our knowledge, Theorem 3 is completely new.

Supercritical Cases
First, if n p n < ∞, then by the Borel-Cantelli lemma, only finitely many turns will occur a.s.; therefore, the X j 's will eventually become all ones or all zeros, and hence where ζ ∈ {0, 1}. By the symmetry of the definition with respect to heads and tails (or, by the bounded convergence theorem), ζ is a Bernoulli(1/2) random variable.

The Critical Case
Fix a > 0, and let p n = a n , n ≥ n 0 for some n 0 ∈ N. Denote by Beta(a, a) the symmetric (around the point 1/2) Beta distribution with a > 0, with density on the unit interval (the normalizing constant is B(a, a) := 2 (a)/ (2a), using Euler's Gamma function), and moment generating function (3) Remark 1 It turns out that the convergence in distribution cannot be strengthened to convergence in probability; see, e.g., the example in Section 2 of [7].
Proof We will verify the statement by analyzing the moments of S N . The odd moments are all zero from the symmetry of S N around 0; on the other hand, for even K we can use the multinomial theorem: where I stands for the sum of those products in which not all terms are different. Note that |Y l i | ≤ 1 for any l ≥ 1 (and Y l i ≡ 1 for l even). Therefore, where the numbers i , i 1 , . . . , i K −2 are between 1 and N and are not necessarily distinct. Hence, also using (2), we get Let us now analyze the elements in the sum above. From (1), for j > i > max{2a, n 0 } we have Consequently, (4) can be approximated as (the contribution from the terms where i 1 ≤ max{2a, n 0 } as well as the other remainder terms in the formula for e i, j is of order at most N K −1 ). Since we are working on a compact interval, we may conclude (see, e.g., Section 2, Exercise 3.27 in [5]) that and has the following moments: , K = 2m is even, which, for even moments, can be equivalently written as The moment generating function of ξ a is Let ζ a := (ξ a + 1)/2. We know that 1 N N i X i → ζ a in distribution, and using (3), completing the proof.

Subcritical Case
Now fix γ, a > 0, and let p n = a n γ , n ≥ n 0 for some n 0 . Note that γ > 1 corresponds to the supercritical case studied in Sect. 3, so from now on assume 0 < γ < 1.
In the proof we will use that (see, e.g., [5], Section 2, Exercise 3.15). Let A (A > a) be a given constant. By Corollary 1, there exists an n * = n * (a, Using the fact that x −γ is decreasing and bounding the sum by the integral, we have One can check that sup N Eη 2 a,γ ,N < ∞ (see the computation below with m = 1), and thus Chebyshev's inequality implies that (η a,γ ,N ) N ≥1 is a tight sequence of random variables. Hence, it is enough to show that each subsequential limit is the same.
Assume that (N l ) l≥1 is a subsequence and lim l→∞ Law(η a,γ ,N l ) = L. Since and in fact, this limit must be the same for any A > a (and corresponding n * = n * (a, A, γ )). Informally, this just means that we may throw away a finite chunk of the sequence of Y i 's' (at the beginning) without affecting its limit. Let us denote the even moments of L by M 2m ∈ [0, ∞], m ≥ 1, while we note again that the odd moments must be zero by symmetry. Also, M N l ,A,K will denote the K th moment under L N l ,A .
We will show below that for a fixed A > a and K = 2m, m ≥ 1, Once (11) for all K ≥ 1 and all A > a. Since (11) holds for any A > a, letting A ↓ a and using (11) and (12), one has that in fact In summary, we obtain that for any fixed A > a, At the same time, we recall that the normal distribution is uniquely determined by its moments, and therefore the convergence toward a normal law is implied by the convergence of all the moments (see, e.g., [5], Section 2.3.e). In our case, (13) along with (9) implies L = lim l→∞ L N l ,A = Normal(0, σ 2 ). Therefore, it only remains to prove (11).
Let us start with the upper estimate in (11). For K = 2m, one has where I are lower-order terms, as it will be shown below. Using (2) along with (10), we may continue with where By the calculation in the "Appendix," the RHS of (14) is (1)).
The reason the remaining terms, collected in I , are of lower order is as follows. Apart from the already estimated term, in the expansion for E(Y n * + · · · + Y N ) K for r = 1, 2, . . . , K − 1 we also have to sum up the terms of the type where n * ≤ i 1 < · · · < i r ≤ N , all p j ≥ 1, and p 1 + p 2 + · · · + p r = K .
Since Y i = ±1, and thus Y p i = 1 if p is even and Y p i = Y i if p is odd, it suffices to estimate only the sums where the summation is taken over all sets (i 1 , . . . , i r ) such that i k+1 ≥ i k + k , 1 ≤ k ≤ K , for all k, i 1 ≥ 1 and i r ≤ N . However, since r ≤ K − 1, each of the sums R(r ; 1 , . . . , r ; N ; K ; γ ) is at most of order N r (1+γ )/2 ≤ N (K −1)(1+γ )/2 , precisely by the same arguments which were used to estimate the sum in (14). The number of those sums can be large, as it is the number of integer partitions of K , but it depends only on K and does not increase with N .
Consequently, for m ≥ 1 we have and by similar computation, The proof is complete.

When Does the Law of Large Numbers Hold for General Sequences { p n }?
A natural question to ask is when does S N obey the Strong (Weak) Law of Large Numbers. The following result gives a partial answer. For a positive even number K , introduce the shorthand and note that The first condition in the next theorem may look reminiscent of Kolmogorov's sufficient condition for the Strong Law of Large Numbers.

then the Law of Large Numbers breaks down, and in fact, Law(S N /N ) converges to a law which has zero odd moments and even moments {μ K }.
Note that (16) is the so-called Carleman condition, guaranteeing that the μ K s correspond to at most one probability law (see Theorem 3.11, Section 2, in [5]).
Proof We will use the facts about the method of moments for weak convergence discussed in the proof of Theorem 2, along with the fact that from (4), it follows that (a) Let us consider the two assumptions separately: Under (C1), the statement follows from Theorem 1 in [9], as Under (C2), along the lines of Theorem 6.5 in Section 1 of [5], we note that for ε > 0, one has by the Markov inequality (recall that K is even). Since, by (6), the expression on the left-hand side of (15) is the leading order term in ES K N , by (15), we have N P(|S N /N | > ε) < ∞, and thus, by the Borel-Cantelli lemma, P(|S N /N | > ε i.o.) = 0, which implies the statement.
(b) Since |S N /N | ≤ 1, we know that S N /N converges to zero in law (i.e., in probability, since the limit is deterministic) if and only if all its moments converge to zero. (On direction is to realize that the kth moment is the same as E f K (S N /N ), where f (x) = x on the unit interval, f (x) := 1 for x > 1 and f (x) = −1 for x < −1; then f is bounded and continuous. The other direction is also known, since the deterministically zero distribution is uniquely determined by its moments). By symmetry, it is enough to check the even moments, for which we know (17). The statement then follows from the fact that the remainder term is O (1/N ). (c) Assume that the conditions in (c) hold. Since the moments of S N /N converge (the odd moments are zero by symmetry), the corresponding laws are tight and, by the Carleman condition, all subsequential limits are the same. That is, as N → ∞, Law(S N /N ) converges to a law with moments given by μ K , and since μ K > 0, the limit cannot be deterministically zero.

Giving Up Symmetry
Now we will show that, in the supercritical case as well as in the setups of Theorem 1 and of Theorem 2, the initial condition being symmetric (i.e., X 1 is equally likely to be 0 or 1) is actually not essential for the limiting distributions.
In the supercritical case, we will have again T N /N → ζ ∈ {0, 1} ∼ Bernoulli(q) a.s., but because of the lack of symmetry, we can no longer claim that q = 1/2. Our next statement, the proof of which may already be known, gives the exact value of q for any sequence of { p n }. In particular, if at least one of p i 's is 1/2, then q = 1/2, which is already clear from the symmetry.
In the critical case, Eq. (1) still holds and so does (2) for even K , but for odd The calculation (6) remains valid for even K ; however, if K is odd, one cannot claim any more that ES K N = 0. At the same time, a calculation similar to (6) immediately , and hence the rescaled odd moments tend to zero, while the even moments are the same as in the original model. Hence the limiting distribution must be the same.
Similar argument holds for the subcritical case as well. Indeed, the even moments ES K N remain the same, while the odd moments for K = 2m + 1 will be ES 2m+1 N = O N m(1+γ ) = o N K (1+γ )/2 due to (18) and the result from the "Appendix."

Further Heuristic Arguments and a Conjecture
In this section, we omit the details of some calculations-the reader can find them in the preprint of this paper [6].
To avoid ambiguity, by the "classical CLT" we mean the situation where, after normalizing with the standard deviation, the limit has a standard normal distribution, and the standard deviation itself is of order √ N ; a "nonstandard CLT" will mean that the standard deviation (and thus the fluctuation) is of a different order.
Consider a sum of N ≥ 1 variables having the same law with finite variance. As is well known, the two "extreme cases" for a sum are the independent case, when the variance is linear and one gets the Central Limit Theorem, and the other one is when all the variables are identical and the variance grows like N 2 . By analogy then, (after recalling that in our model 2N 2 E(N , 2) holds), it seems that the first crucial question is whether is still the case. If (19) is true, then Var (S N ) is of order N , and one can expect that the classical CLT holds. This happens when p n ≡ p ∈ (0, 1). In a situation when (19) fails, one should know at least if holds. Indeed, we know from Theorem 3(b) that the exact criterion for WLLN to hold is that In light of this, we make the following conjecture.

Examples Supporting the Discussion and Conjecture 1
In the examples below, the deviations from the classical CLT are becoming more marked as we go from Example 2 to Example 3 to Example 4. Recall that S N := Y 1 + ...Y N and T N := X 1 + · · · + X N with T N = (Y N + 1)/2; the frequency of heads is T N /N .
Therefore, the variance is still of order N , but the constant has changed. Recall that Cov(Y i , Y j ) = e i, j = κ j−i , and, following [8], define σ 2 In this case, since we are dealing with a time homogeneous Markov chain, it is well known (see [8]) that Therefore, unless c = 1/2, the classical CLT is slightly changed, since Var(S n ) ∼ σ 2 c n. It is also clear that the limiting normal variance can be arbitrarily large when c is sufficiently small and thus turns occur very rarely. On the other hand, it can be arbitrarily small if c is sufficiently close to 1 and thus turns occur very frequently. (If the turns were certain, then the limiting variance would vanish of course).
Example 2 (Classical CLT breaks down) Consider the case p n := a/n γ with 0 < γ < 1. Then that is Var(S N ) is of order N γ +1 , and the power is strictly between 1 and 2. Now (21) is true, WLLN is still in force, and S N /N is still around zero (the proportion of heads is around 1/2). But (19) is false. The closer γ to 1, the more the situation differs from the classical CLT. We now have a nonstandard CLT, with larger than classical fluctuations.
Example 3 (LLN breaks down) Consider the case when p n = 1/n. Then e i, j = is of order N 2 , that is, (19) and even (21) are false, causing The Law of Large Numbers to break down, and S N /N is no longer around zero. This means that the correlation is as strong as in the case of identical variables, and the fluctuations are now of order N , destroying LLN. Similar is the situation when p k = a k with a > 0. In terms of the relative frequency of heads, instead of being around the δ 1/2 distribution, now it tends to the Beta(a, a) distribution.
We conclude this section with an open problem.

Problem 1 (Monotonicity for SLLN)
Is it true that if SLLN holds for the sequence { p n }, then it also holds for the sequence {p n }, wheneverp n ≥ p n for all n?
The corresponding statement for WLLN is true by Corollary 3.
For l = 0, it is true. Now assume that we have established (22) for some l ≥ 0. Then where the sum in the second line is taken over i 2l+1 and i 2l+2 only. We shall estimate the sum below.
First, note that where each expression is between 0 and 1, can be very well approximated by the corresponding integral, since, whenever y ≤ x, |x − x| ≤ 1 and |ỹ − y| ≤ 1, the ratio is bounded above by e c 1 [y −γ +x −γ ] where c 1 > 0 is some constant. Hence, outside of the area where x and y are both smaller than √ N , the above ratio is very close to 1, while the double sum over that area can be at most N . Therefore, as N → ∞, To calculate the inner integral, observe Note that ψ(y) ↓ 1 as y → ∞, hence, since y ≥ n * , N n * N y e −cx 1−γ dx e cy 1−γ dy ≤ N n * (1 + ψ(n * ))R(y, N )e cy 1−γ dy Consequently, ( * ) = O(N 1+γ ) and The next step is to compute W.l.o.g. assume that N γ /2 > n * + 2l, then (25) can be split into two terms, depending whether i 2l+1 is smaller or larger than N γ /2 . In the case i 2l+1 ≤ N γ /2 , we have is bounded above by 1 + 2l y e c 2 (x −γ +y −γ ) where c 2 > 0 is some constant. Observing that x ≥ y ≥ N γ /2 yields that this ratio is very close to 1, and thus the double sum in (25) equals From (24), since y ≥ N γ /2 and hence is large, we get that the inner integral equals (1 + o(1)) R(y, b). Therefore, the main expression in (26), up to a factor 1 + o(1) and the remainder term, equals b N γ /2 Now the only remaining step is to show that the integral (**) is of smaller order; then the induction step is finished. To this end, fix some γ < θ < 1, and do the following estimation.