Sampling Strategies for Uncertainty Reduction in Categorical Random Fields: Formulation, Mathematical Analysis and Application to Multiple-Point Simulations

Abstract

The task of optimal sampling for the statistical simulation of a discrete random field is addressed from the perspective of minimizing the posterior uncertainty of non-sensed positions given the information of the sensed positions. In particular, information theoretic measures are adopted to formalize the problem of optimal sampling design for field characterization, where concepts such as information of the measurements, average posterior uncertainty, and the resolvability of the field are introduced. The use of the entropy and related information measures are justified by connecting the task of simulation with a source coding problem, where it is well known that entropy offers a fundamental performance limit. On the application, a one-dimensional Markov chain model is explored where the statistics of the random object are known, and then the more relevant case of multiple-point simulations of channelized facies fields is studied, adopting in this case a training image to infer the statistics of a non-parametric model. In both contexts, the superiority of information-driven sampling strategies is proved in different settings and conditions, with respect to random or regular sampling.

Appendices

Appendix A: Entropy as an Indicator of Simulation Complexity

Here, a formal interpretation of the Shannon entropy as an indicator of complexity is elaborated when the task is to simulate or create a collection of independent and identically distributed (i.i.d.) realizations of a finite alphabet random variable. For that, a connection between simulation and the task of almost lossless source coding (Cover and Thomas 2006) will be presented, from which the entropy is a fundamental performance indicator (Shannon 1948; Cover and Thomas 2006).

1.1 A.1 The Operational Complexity Indicator for Simulation

Considering a finite alphabet random variable X taking values in ${\mathcal {A}}$ with probability $\mu _X\in {\mathcal {P}}({\mathcal {A}})$, the simulation problem of length $n\ge 1$ corresponds to the generation of n-samples in ${\mathcal {A}}^n$ from the product (or n-fold) distribution $\mu _X^n \equiv \mu _X\times \mu _X\times \cdots \mu _X\in {\mathcal {P}}({\mathcal {A}}^n)$.

A covering argument is considered to stipulate the complexity of simulating i.i.d. realizations of $\mu _X^n$ in the product space ${\mathcal {A}}^n$. More precisely, the cardinality of the smallest subset of sequences in ${\mathcal {A}}^n$ that captures almost all the probability with respect to $\mu _X^n$, is proposed as a natural indicator of the complexity of simulating $\mu _X^n$ in ${\mathcal {A}}^n$. This notion can be captured with the following set of definitions:

Definition 1

For $\epsilon >0$, $B\subset {\mathcal {A}}^n$ is $\epsilon$-typical for $\mu ^n_X$, if $\mu ^n_X(B) \ge 1 - \epsilon$.

Definition 2

For $\epsilon >0$, the size k is said to be $\epsilon$-feasible for $\mu ^n_X$, if there is $B\subset {\mathcal {A}}^n$ that is $\epsilon$-typical for $\mu ^n_X$ and $\left| B\right| \le k$.

Definition 3

Finally, considering n i.i.d. samples, an indicator of the complexity of $\mu _x$ is given by

$$\begin{aligned} k(\epsilon , \mu ^n_x) \equiv \min \left\{ k: k\,\hbox {is}\,\epsilon \hbox {-}\hbox {feasible for}\,\mu ^n_X \right\} . \end{aligned}$$

(51)

Adopting these concepts, in particular $k(\epsilon , \mu ^n_x)$, a scenario where a higher number of sequences is needed to capture almost all the probability is more complex (from the point of view of creating i.i.d. simulations), than a case where fewer sequences are needed for the same covering objective. This idea matches the notion of typical set proposed by Shannon to prove source coding theorems (Shannon 1948). In this context, an interesting aspect to pay attention is the exponential growth of $k(\epsilon , \mu ^n_x)$ in the process of making n arbitrarily large. It is simple to note that $k(\epsilon , \mu _x)$ grows with n and it is upper bounded by $\left| {\mathcal {A}}\right| ^n$, therefore $\log _2 k(\epsilon , \mu ^n_x) \le n \log _2 \left| {\mathcal {A}}\right|$ and, consequently, taking the limits on the number of simulations the basic upper bound is

$$\begin{aligned} \lim \sup _{n \longrightarrow \infty } \frac{1}{n} \log _2 k(\epsilon , \mu ^n_x) \le \log _2 \left| {\mathcal {A}}\right| . \end{aligned}$$

(52)

This means that $k(\epsilon , \mu ^n_x)$ grows (with n) at most exponentially with an exponential rate given by $log_2 \left| {\mathcal {A}}\right|$. The next result in (53) stipulates that the precise exponential rate of $k(\epsilon , \mu ^n_x)$ (as n goes to infinity) approaches ${\mathcal {H}}(\mu _x)$, when making $\epsilon$ arbitrary small, in the sense that

$$\begin{aligned} \lim _{\epsilon \longrightarrow 0}\lim _{n \longrightarrow \infty } \frac{1}{n} \log _2 k(\epsilon , \mu ^n_x) = {\mathcal {H}}(\mu _X). \end{aligned}$$

(53)

1.2 A.2 Derivation of the Representation Result in (53)

To show the identity in (53), it is useful to introduce a stronger notion of achievable rate for the entire i.i.d. process $\left\{ \mu _X^n: n\ge 1\right\}$ that is commonly used in the context of almost lossless fixed-rate source coding (Cover and Thomas 2006) and (Yeung 2002).

Definition 4

For the i.i.d. simulation of $\mu _X$, the rate $r>0$ is said to be achievable for $\left\{ \mu _X^n: n\ge 1\right\}$, if there is a sequence of sets $\left\{ B_n\subset {\mathcal {A}}^n: n\ge 1 \right\}$ that captures all the probability in the stronger sense that

$$\begin{aligned} \lim _{n \rightarrow \infty } \mu _X^n(B_n)=1, \end{aligned}$$

and

$$\begin{aligned} \lim \sup _{n \rightarrow \infty } \frac{1}{n}\log _2 \left| B_n \right| \le r. \end{aligned}$$

Definition 5

In this context, the minimum achievable rate for $\left\{ \mu _X^n: n\ge 1\right\}$ is

$$\begin{aligned} R^*(\mu _X) \equiv \min \left\{ r: \hbox {} r>0\,\hbox {is achievable for} \left\{ \mu _X^n: n\ge 1\right\} \right\} . \end{aligned}$$

From the Definitions, it is direct to show that $\lim \sup _{n \longrightarrow \infty } \frac{1}{n} \log _2 k(\epsilon , \mu ^n_x) \le R^*(\mu _X)$ for all $\epsilon >0$ and consequently,

$$\begin{aligned} \lim \sup _{\epsilon \longrightarrow 0}\lim \sup _{n \longrightarrow \infty } \frac{1}{n} \log _2 k(\epsilon , \mu ^n_x) \le R^*(\mu _X). \end{aligned}$$

(54)

Remarkably for our i.i.d. context, it is well-known that the limit in the left-hand side of (54) is well-defined and matches $R^*(\mu _X)$. Furthermore, this expression reduces to the entropy of $\mu _x$. This result was proved in the original paper of Shannon (Shannon 1948) and it is highlighted in the following result:

Theorem 1

(Shannon 1948) $\lim _{\epsilon \longrightarrow 0}\lim _{n \longrightarrow \infty } \frac{1}{n} \log _2 k(\epsilon , \mu ^n_x) =R^*(\mu )= {\mathcal {H}}(\mu _X).$

Therefore, the main conclusion of this analysis is that the Shannon entropy determines the complexity of simulating a finite alphabet probability in the precise operational sense defined in Definitions 3 and 5.

The proof of Theorem 1 is a direct consequence of the celebrated (weak) asymptotic equipartition property (AEP), first stated by Shannon and proved for the i.i.d. case in Shannon (1948). A systematic and clear exposition of this property and the proof of Theorem 1 can be found in Cover and Thomas (2006) and Yeung (2002) in the context of what is known in information theory as the Shannon source coding theorem. To conclude this part, it is important to elaborate some observations and implications about Theorem 1:

• 1. This result shows that there is a collection of sequences $\left\{ B_n\subset {\mathcal {A}}^n: n\ge 1\right\}$ that captures asymptotically all the probability, that is, the collection is typical in the sense that $\lim _{n \rightarrow \infty } \mu _X^n(B_n)=1$, and its cardinality grows exponentially at a rate that is precisely the entropy of $\mu _X$, that is, as n goes to infinity $\left| B_n \right| \approx 2^{n\cdot {\mathcal {H}}(\mu _x)}$.

• 2. If ${\mathcal {H}}(\mu _X) < \log _2 \left| {\mathcal {A}}\right|$, then the collection of typical sequences, mentioned in the previous point, is an arbitrary small fraction of all the possible sequences, in the sense that

  $$\begin{aligned} \frac{\left| B_n \right| }{\left| {\mathcal {A}}^n \right| } \approx \frac{2^{n\cdot {\mathcal {H}}(\mu _x)}}{2^{n\cdot log_2 \left| {\mathcal {A}}\right| }}= 2^{-n\cdot (\log _2 \left| {\mathcal {A}}\right| -{\mathcal {H}}(\mu _x) )} \longrightarrow 0. \end{aligned}$$

  (55)

  Note that the cardinality ratio in (55) goes to zero exponentially with n at a rate given by $\log _2 \left| {\mathcal {A}}\right| -{\mathcal {H}}(\mu _x)>0$. Then for a i.i.d process $\left\{ \mu _X^n: n\ge 1\right\}$ with entropy strictly smaller than $\log _2 \left| {\mathcal {A}}\right|$ there is a tiny fraction of sequences that characterize the i.i.d. process induced by $\mu _X$. Here, $\log _2 \left| {\mathcal {A}}\right|$ is the maximum entropy only achieved by the uniform distribution on ${\mathcal {A}}$ (Cover and Thomas 2006).

• 3. For the achievability part of this result, Shannon proposes a specific collection of typical sequences given by

  $$\begin{aligned} B_n(\epsilon )=\left\{ (x_1,..,x_n)\in {\mathcal {A}}^n: \left| -\frac{1}{n} \log _2 \mu ^n_X(x_1,..,x_n) - {\mathcal {H}}(\mu _X) \right| \le \epsilon \right\} , \end{aligned}$$

  (56)

  that as n goes to infinity has the following properties (Cover and Thomas 2006):

  • (a) it is a typical set: $\lim _{n \rightarrow \infty } \mu _X^n(B_n(\epsilon ))=1$.

  • (b) if $(x_1,..,x_n) \in B_n(\epsilon )$ then

    $$\begin{aligned} 2^{-n\cdot ({\mathcal {H}}(\mu _X)+\epsilon )} \le \mu ^n_X(x_1,..,x_n) \le 2^{-n\cdot ({\mathcal {H}}(\mu _X)-\epsilon )}. \end{aligned}$$

  • (c) $(1-\epsilon ) 2^{n\cdot ({\mathcal {H}}(\mu _X) -\epsilon )} \le \left| B_n(\epsilon ) \right| \le 2^{n\cdot ({\mathcal {H}}(\mu _X) +\epsilon )}$.

  Here, achievability refers to the construction of a sequence that is typical and achieves a rate smaller or equal to the entropy (Cover and Thomas 2006). Thus, the relevant aspect of this construction is the fact that as n progresses to infinity the elements of this typical set become uniformly distributed. Considering $\epsilon$ sufficiently small, for all $(x_1,..,x_n)\in B_n(\epsilon )$, then

  $$\begin{aligned} \mu ^n_X(x_1,..,x_n)\approx 2^{-n\cdot {\mathcal {H}}(\mu _X)} \approx \frac{1}{ \left| B_n(\epsilon ) \right| }. \end{aligned}$$

  Then, within this set $B_n(\epsilon )$, which is typical, all its elements have the same probability. This means that when making i.i.d. samples of the model $\mu ^n_X$ and n is sufficiently large, a single sample of this typical set (that happens with very high probability), has the same probability than any other element of the set. Consequently, from this fact, it is clear that the size of this typical set plays a major role from the point of view of specifying the complexity of the simulation task, which is formalized on the concept of minimum achievable rate in Definition 5, and that is precisely given by ${\mathcal {H}}(\mu _X)$.

Appendix B: Proof of Proposition 1

Proof

After obtaining the sequential rule ${\tilde{f}}^*_k \in {\mathbf {F}}_k$ to solve the adaptive sensing rule of size k, the remaining posterior uncertainty is given by

$$\begin{aligned} H({\hat{X}}_{f^*_k}| {X}_{f^*_k}), \end{aligned}$$

(57)

and the reduction of the uncertainty concerning the information of ${\tilde{f}}^*_k$ to resolve ${\bar{X}}$ is given by

$$\begin{aligned} I({\tilde{f}}^*_k) = H({\overline{X}}) - H({\hat{X}}_{{\tilde{f}}^*_k}| {X}_{{\tilde{f}}^*_k}) = H({X}_{{\tilde{f}}^*_k}) \ge 0. \end{aligned}$$

(58)

As the sets ${\hat{X}}_{{\tilde{f}}^*_k}$ and ${X}_{{\tilde{f}}^*_k}$ conform a partition of ${\overline{X}}$,

$$\begin{aligned} H({\overline{X}}) = H({\hat{X}}_{{\tilde{f}}^*_k} , {X}_{{\tilde{f}}^*_k}). \end{aligned}$$

(59)

In addition, by the chain rule of entropy

$$\begin{aligned} H({\overline{X}}) = H( {X}_{{\tilde{f}}^*_k}) + H({\hat{X}}_{{\tilde{f}}^*_k} , {X}_{{\tilde{f}}^*_k}). \end{aligned}$$

(60)

Applying (59) and (60), the information gain of the rule ${\tilde{f}}^*_k$ can be rewritten by

$$\begin{aligned} I({\tilde{f}}^*_k)&= H( {X}_{{\tilde{f}}^*_k}) + H({\hat{X}}_{{\tilde{f}}^*_k} , {X}_{{\tilde{f}}^*_k}) - H({\hat{X}}_{{\tilde{f}}^*_k}| {X}_{{\tilde{f}}^*_k}) \nonumber \\&= H( {X}_{{\tilde{f}}^*_k}) \nonumber \\&= H( {X}_{{\tilde{f}}^*_k(1)}, {X}_{{\tilde{f}}^*_k(2)}, .. , {X}_{{\tilde{f}}^*_k(k)}) \nonumber \\&= H( {X}_{(i_{1}^{*},j_{1}^{*})} , {X}_{(i_{2}^{*},j_{2}^{*})} , .. , {X}_{(i_{k}^{*},j_{k}^{*})}). \end{aligned}$$

(61)

The expression (61) can be rearranged by using the chain rule as

$$\begin{aligned} I({\tilde{f}}^*_k)&= H( {X}_{(i_{1}^{*},j_{1}^{*})} ) + \sum _{h = 2}^{k} H( {X}_{(i_{h}^{*},j_{h}^{*})} | {X}_{(i_{1}^{*},j_{1}^{*})} , .. , {X}_{(i_{h -1}^{*},j_{h -1}^{*})} ). \end{aligned}$$

(62)

Finally, using (62), the information gain difference between the sampling steps k and $k-1$ is given by

$$\begin{aligned} I({\tilde{f}}^*_k) - I({\tilde{f}}^*_{k-1})&= H( {X}_{(i_{1}^{*},j_{1}^{*})} ) + \sum _{h = 2}^{k} H( {X}_{(i_{h}^{*},j_{h}^{*})} | {X}_{(i_{1}^{*},j_{1}^{*})} , .. , {X}_{(i_{h -1}^{*},j_{h -1}^{*})} ) \nonumber \\&\qquad - H( {X}_{(i_{1}^{*},j_{1}^{*})} ) - \sum _{h = 2}^{k-1} H( {X}_{(i_{h}^{*},j_{h}^{*})} | {X}_{(i_{1}^{*},j_{1}^{*})} , .. , {X}_{(i_{h -1}^{*},j_{h -1}^{*})} ) \nonumber \\&= H( {X}_{(i_{k}^{*},j_{k}^{*})} | {X}_{(i_{1}^{*},j_{1}^{*})}, .., {X}_{(i_{k-1}^{*},j_{k-1}^{*})} ). \end{aligned}$$

(63)

Appendix C: Optimality of the Sequential Rule for Fields with no Inter-Pixel Dependency

Proof

For a field ${\bar{X}}$ with no inter-pixel dependency, every random variable $X_{i,j}$ is statistically independent of others variables in the sense that $H(X_{i_{1},j_{1}} | X_{i_{2},j_{2}}) = H(X_{i_{1},j_{1}})$, for all $(i_{2},j_{2})\in [M]\times [M] \setminus \left\{ (i_{1},j_{1})\right\}$. Then, by the independence bound on entropy (Cover and Thomas 2006), the posterior field entropy associated with the application of the optimal rule $f^{*}_{k}$ posted (18) is given by,

$$\begin{aligned} H({X}_f)&= \sum _{(i,j) \in {f^{*}_{k}} } H(X_{i,j}). \end{aligned}$$

(64)

Thus, for a field with no inter-pixel dependency (nipd), the optimal sampling rule of size k can be state as

$$\begin{aligned} f^{*,nipd}_k = \arg \max _{f\in {\mathbf {F}}_k} \sum _{(i,j) \in {f} } H(X_{i,j}), \end{aligned}$$

(65)

which, by the non negativity of the entropy, is the problem of choosing the k positions with the highest a priori entropy.

In the case of the iterative sequential rule, the k-measurement is now given by,

$$\begin{aligned} (i^{*,nipd}_k,j^{*,nipd}_k)&= \arg \max _{(i,j)\in [M]\times [M] \setminus \left\{ (i^{*,nipd}_l,j^{*,nipd}_l): l=1,..,k-1\right\} } H(X_{i,j}), \end{aligned}$$

(66)

which correspond to choose the location with the k-th highest a priori entropy. Therefore, the sequential rule ${\tilde{f}}^{*,nipd}_k \in {\mathbf {F}}_k$ can be obtained by

$$\begin{aligned} {\tilde{f}}^{*,nipd}_k(1)&=(i^{*,nipd}_1,j^{*,nipd}_1), \nonumber \\ {\tilde{f}}^{*,nipd}_k(2)&=(i^{*,nipd}_2,j^{*,nipd}_2), \nonumber \\ ..., \nonumber \\ {\tilde{f}}^{*,nipd}_k(k)&=(i^{*,nipd}_k,j^{*,nipd}_k). \end{aligned}$$

(67)

Finally, by construction, the optimal combinatorial rule is equal to the sequential approach under the assumption of statistical independence,

$$\begin{aligned} {\tilde{f}}^{*,nipd}_k = f^{*,nipd}_{k}, \end{aligned}$$

(68)

meaning that in this scenario the optimal sampling can be achieve by the sequential approach. This simple case summarizes the nature of the optimal sampling approach that tries to resolve the locations with highest uncertainty in order to increase the average knowledge of the global field. As the inter-pixel dependency increases and the multiple-point statistics of the field becomes more complex, to take into account the high order conditional entropies becomes essential to avoid locations with redundant information.

Appendix D: Analysis of the Adaptive Sampling Scheme for the Markov Chain Case

Here, the AMIS $\left\{ \right. {\tilde{f}}^a_k(\cdot | \cdot ): k \in [N] \left. \right\}$ in (36) is compared with the SMIS approach $\left\{ \right. {\tilde{f}}^*_k(\cdot ):k \in [N] \left. \right\}$ in (28). This comparison is performed in the context of a finite length one-dimensional Markov source presented in Sect. 6 in terms of field resolvability (resolution of uncertainty) and the estimation of non-sensed positions. To evaluate the quality on resolving non-sensed position from the information of sensed position, the conditional entropy conditioning on the data is used, given a sensed rule $f_k$ and its sensed data $x_f=(x_1,..,x_{k-1})\in {\mathcal {A}}^{k-1}$,

$$\begin{aligned} H( {\hat{X}}_{f_k} | X_{f} = x_{f}). \end{aligned}$$

(69)

Fig. 19

Remaining conditional entropy by considering the previous sampled locations and its measurements. Symmetric transition matrix ($\beta = 0.2$)

By extending the result in Proposition 4, there is a simple algorithm to compute (69) by the Markov assumption not reported here for the space constraint. In addition, the problem of estimating non-sensed position from the sensed data is considered for which the minimum mean square error estimator (MMSE) is applied given by the conditional mean (Gray and Davisson 2004). More precisely, given $f_k$ and its sensed data $x_f=(x_1,..,x_{k-1})\in {\mathcal {A}}^{k-1}$, the MMSE estimator of ${\hat{X}}_{f_k}$ from $X_f=x_f$ is given in closed form by $\mathbb {E} ({{\hat{X}}_f| X_f=x_f})\in {\mathcal {A}}^{N-k}$, which is a function of $x_f$. This problem reduces to compute point-wise the expectation for every non-sensed position given $X_f=x_f$, which is a simple task under the Markov assumption.

Fig. 20

Estimation error considering the previous sampled locations and its measurements. Symmetric transition matrix with $\beta = 0.1$. Top: random sampling versus AMIS method, bottom: SMIS versus AMIS

Appendix E: Analysis of the Partial Update of Conditional Entropies in (45)

It can been argued that the use of the partial update of the conditional probabilities proposed in Sect. 7.2 can be justified in the context of SMIS in (27). Considering the stage $k-1$ and k of this algorithm, for an arbitrary unmeasured location, the focus is to evaluate the conditional entropy $H(X_{i,j}|X_{i^*_1,j^*_1},..,X_{i^*_{k-2},j^*_{k-2}})$ for the stage $k-1$, and $H(X_{i,j}|X_{i^*_1,j^*_1},..,$$X_{i^*_{k-2},j^*_{k-2}},$$X_{i^*_{k-1}, j^*_{k-1}})$ for the stage k. Thus, the point-wise reduction of conditional entropy for the location (i, j) from the stage $k-1$ to the state k is given by

$$\begin{aligned} \Delta H^{(k-1) \rightharpoonup k}(X_{i,j})&= H(X_{i,j}|X_{i^*_1,j^*_1},..,X_{i^*_{k-2},j^*_{k-2}}) \nonumber \\&\quad - H(X_{i,j}|X_{i^*_1,j^*_1},..,X_{i^*_{k-2},j^*_{k-2}},X_{i^*_{k-1},j^*_{k-1}}) \nonumber \\&= I(X_{i,j} | X_{i^*_1,j^*_1},..,X_{i^*_{k-2},j^*_{k-2}};X_{i^*_{k-1},j^*_{k-1}}), \end{aligned}$$

(70)

where, for $k = 2$ it is clear that the reduction in entropy corresponds exactly to mutual information

$$\begin{aligned} \Delta H^{(1) \rightharpoonup 2}(X_{i,j})&= H(X_{i,j}) - H(X_{i,j}|X_{i^*_1,j^*_1}) \nonumber \\&= I(X_{i,j} ; X_{i^*_1,j^*_1}). \end{aligned}$$

(71)

As the algorithm works in the search for new measurements, the mutual information is conditioned on previous measurements, and consequently, the approach used in the entropy map update deteriorates. However, recalculating the entropy maps every 20 samples has provided satisfactory results (see Figs. 19, 20).