Maximum Entropy Design in High Dimensions by Composite Likelihood Modelling

Abstract

In maximum entropy sampling (MES), a design is chosen by maximizing the joint Shannon entropy of parameters and observations. However, when the conditional parametric model of the response contains a large number of covariates, the posterior calculations in MES can be challenging or infeasible. In this work, we consider the use of composite likelihood modelling to break down the complexity of the full likelihood and code the original optimization problem into a set of simple partial likelihood problems. We study the optimality behaviour of the composite likelihood sampling approach as the number of design variables grows using both asymptotic analysis and numerical simulations.

Appendix

Proof of Proposition 1.

Let Z=(Z ₁,…,Z _p ) be a p-dimensional random vector with distribution p(z). Singer (2004) shows that if Z _j is independent of Z _k for any j≠k, then $p(\mathbf{z}) = \tilde{p}(\mathbf{z})^{(p)} = \prod_{|E|<p} p_{E}(\mathbf{z}_{E})^{q_{E}}$, where E is a set in the power set of indexes , |E| is the cardinality of the set E, q ^E=(−1)^p+1−|E|, and p _E denotes the distribution of Z _E ⊂Z. Without loss of generality, we start from θ ₁, θ ₂ and θ ₃ and write

$$\log p(\theta_1, \theta_2, \theta_3|y, \xi) = \log\tilde{p}(\varTheta|y,\xi)^{(2)} + \log p(\theta_1| \theta_2, \theta_3, y, \xi) - \log p( \theta_1| \theta_2, y, \xi). $$

Recursively applying the formula by Singer (2004) for 3≤k≤p, gives

(8)

By summing over all such decompositions and taking the expectation with respect to Θ|Y,ξ, we obtain

which implies $L(\xi) = \operatorname {E}_{Y} H(\varTheta|Y, \xi) = L^{(p)}(\xi) + S_{p}(\varTheta|Y,\xi)$. Finally, by our sparsity assumption, the last summand converges to zero as p→∞.