On Algorithmic Statistics for Space-bounded Algorithms

Abstract

Algorithmic statistics looks for models of observed data that are good in the following sense: a model is simple (i.e., has small Kolmogorov complexity) and captures all the algorithmically discoverable regularities in the data. However, this idea can not be used in practice as is because Kolmogorov complexity is not computable. In this paper we develop an algorithmic version of algorithmic statistics that uses space-bounded Kolmogorov complexity. We prove a space-bounded version of a basic result from “classical” algorithmic statistics, the connection between optimality and randomness deficiences. The main tool is the Nisan–Wigderson pseudo-random generator. An extended abstract of this paper was presented at the 12th International Computer Science Symposium in Russia (Milovanov 10).

Appendix

Proposition 13

Letx = y ⋅ zbe a concatenation of strings y and z of length n where$y= \overbrace {000 {\ldots } 00}^{n \text { zeros}}$and C(z) > n − εfor somepositiveε.Assume that x belongs to some Hamming ball B.Then

$$2{\mathrm{C}}(B) + \log |B| - {\mathrm{C}}(x) > \frac{2}{5} n - \varepsilon - O(\log n). $$

If ε is small, then Hamming ball B can not satisfy C(B) ≈ 0 and log |B|≈C(x).

Proof

Denote by r and b the radius and the center of B. Denote byb₀andb₁the first and the secondparts of b. (So, b = b₀ ⋅ b₁and|b₀| = |b₁| = n.) Then z belongs tothe Hamming ball B₁withcenter b₁and radius r.Hence,

$${\mathrm{C}}(B_{1}) + \log |B_{1}| \ge {\mathrm{C}}(z) - O(\log n) = {\mathrm{C}}(x) - O(\log n). $$

Note thatC(B₁) ≤C(B) + O(log n). The log-sizeof B₁can beestimated as $nH(\frac {r}{n}) + O(\log n)$,where H(t) := −t log t − (1 − t) log(1 − t)is the binary Shannon entropy (see, for example, [9]). So, the inequality above can be rewrittenas

$$ {\mathrm{C}}(B) + nH\left( \frac{r}{n}\right) \ge {\mathrm{C}}(x) - O(\log n). $$

(1)

We need to show that $2{\mathrm {C}}(B) + \log |B| - {\mathrm {C}}(x) > \frac {2}{5} n - \varepsilon - O(\log n)$,where $\log |B| = 2nH(\frac {r}{2n}) + O(\log n)$.This inequality follows from (1) and the inequality below by an easycalculation.

$$H\left( \frac{r}{n}\right) - H\left( \frac{r}{2n}\right) \le \frac{3}{10}. $$

To verify the last inequality one can show that the maximum of the function$H(t) - H(\frac {t}{2})$is equal to$\frac {\ln (\frac {3}{4} +\frac {1}{\sqrt {2}})}{\ln 4} < \frac {3}{10}$.□

1.1 Symmetry of Information

Define C^m(A,B) as the minimal length of a program that on input a pair of strings (a,b) uses at most space m, and outputs the bits (a ∈ A,b ∈ B).

Lemma 14 (Symmetry of information)

AssumeA,B ⊆{0, 1}ⁿ.Then

$$\textup{(a) } \forall m \text{ } {\mathrm{C}}^{p}(A, B) \le {\mathrm{C}}^{m}(A) + {\mathrm{C}}^{m}(B | A) + O(\log ({\mathrm{C}}^{m}(A,B)+m + n)) $$

forp = m + poly(n +C^m(A,B)).

$$\textup{(b) } \forall m \text{ } {\mathrm{C}}^{p}(A) + {\mathrm{C}}^{p}(B | A) \le {\mathrm{C}}^{m}(A, B) + O(\log ({\mathrm{C}}^{m}(A,B)+m + n) ) $$

forp = 2m + poly(n +C^m(A,B)).

Proof Proof of Lemma 14 (a)

The proof is similar to the proof of Theorem 6 (a).□

Proof Proof of Lemma 14 (b)

Let k :=C^m(A,B).Denote by $\mathcal {D}$the family of sets (U,V )such that C^m(U,V ) ≤ k and U,V ⊆{0, 1}ⁿ.It is clear that $|\mathcal {D}| < 2^{k + 1}$.Denote by $\mathcal {D}_{A}$the pairs of $\mathcal {D}$for which the first element is equal to A. Let t satisfy the inequalities$2^{t} \le |\mathcal {D}_{A}| < 2^{t + 1}$.

We prove that

• C^p(B|A)does not exceed t significantly;

• C^p(A)does not exceed k − t significantly.

Here p = 2m + O(n).

We start with the first statement. There exists a program that enumerates all sets from$\mathcal {D}_{A}$using A as an oracleand that works in space 2m + O(n).Indeed, such enumeration can be done in the following way: enumerate all programs of length k andverify the following condition for every pair of n-bit strings. First, a program uses at most m space onthis input and does not loop. To verify it we need to check that the program does not compute longerthan 2^O(m)steps. Second, if a second n-bit string belongs to A then the program outputs 1, and 0otherwise. Append to this program the ordinal number of a program that distinguishes(A,B). This number isnot greater than t + 1.Therefore we have C^p(B|A) ≤ t + O(log(C^m(A,B) + m + n)).

Now we prove the second statement. Note that there exist at most2^k−t+ 1sets U such that $|\mathcal {D}_{U}| \ge 2^{t}$(including A). Hence, if we construct a program that enumerates all sets with such property (anddoes not use much space) then we are finished, because the set A can be described bythe ordinal number of this enumeration. Let us construct such a program. It works asfollows:

Enumerate all sets U that are the first elements from$\mathcal {D}$,i.e. we enumerate programs that distinguish the corresponding sets (say,lexicographically). We go to the next step if the following properties hold. First,$|\mathcal {D}_{U}| \ge 2^{t}$, andsecond: we did not consider set U earlier (i.e. every program whose lexicographicalnumber is smaller does not distinguish U or is not the first element from a set from$\mathcal {D}$).

This program uses 2m + poly(n +C^m(A,B))spaceand has length O(log(C^m(A) + n + m)), and hencesatisfies all requirements. □

Proof Proof of Lemma 11

It suffices to show that $\mathcal {B}$satisfies property (1)^∗with probability at most 2⁻ⁿ,because $\mathcal {B}$satisfies property (2)with probability at most $\frac {1}{4}$.

For this let us show that every part is ‘bad’ (i.e. has at least(n + k)² + 1sets from$\mathcal {B}$) with probabilityat most 2^− 2n.The probability of such event is equal to the probability that a binomial random variable withparameters (2^k, 2^−k(n + 2) ln 2)exceeds (n + k)².To bound this, we use an easy but lengthy sequence of estimations. Forw := 2^k,p := 2^−k(n + 2) ln 2andv := (n + k)²wehave

$$\sum\limits_{i=v}^{w} {{w}\choose{i}} p^{i}(1-p)^{w-i} < w \cdot {{w}\choose{v}} p^{v}(1-p)^{w-v} < w \cdot {{w}\choose{v}} p^{v} < w \frac{(wp)^{v}}{v!}. $$

The leftmostinequality follows from wp = (n + 2) ln 2 ≤ (n + k)² = v.Because wp = (n + 2) ln 2 < 10n,we obtain

$$w \frac{(wp)^{v}}{v!} < \frac{2^{k} (10n)^{(n+k)^{2}}}{((n+k)^{2})!} \ll 2^{-2n}. $$

□