Bayesian maximum entropy and data fusion for processing qualitative data: theory and application for crowdsourced cropland occurrences in Ethiopia

Abstract

Categorical data play an important role in a wide variety of spatial applications, while modeling and predicting this type of statistical variable has proved to be complex in many cases. Among other possible approaches, the Bayesian maximum entropy methodology has been developed and advocated for this goal and has been successfully applied in various spatial prediction problems. This approach aims at building a multivariate probability table from bivariate probability functions used as constraints that need to be fulfilled, in order to compute a posterior conditional distribution that accounts for hard or soft information sources. In this paper, our goal is to generalize further the theoretical results in order to account for a much wider type of information source, such as probability inequalities. We first show how the maximum entropy principle can be implemented efficiently using a linear iterative approximation based on a minimum norm criterion, where the minimum norm solution is obtained at each step from simple matrix operations that converges to the requested maximum entropy solution. Based on this result, we show then how the maximum entropy problem can be related to the more general minimum divergence problem, which might involve equality and inequality constraints and which can be solved based on iterated minimum norm solutions. This allows us to account for a much larger panel of information types, where more qualitative information, such as probability inequalities can be used. When combined with a Bayesian data fusion approach, this approach deals with the case of potentially conflicting information that is available. Although the theoretical results presented in this paper can be applied to any study (spatial or non-spatial) involving categorical data in general, the results are illustrated in a spatial context where the goal is to predict at best the occurrence of cultivated land in Ethiopia based on crowdsourced information. The results emphasize the benefit of the methodology, which integrates conflicting information and provides a spatially exhaustive map of these occurrence classes over the whole country.

Appendices

Appendix A1: Proof for the concavity of the entropy

Let us consider the entropy $H({\mathbf {p}})$ defined as in Eq. (2). It is easy to prove that $H({\mathbf {p}})$ is convex everywhere with respect to ${\mathbf {p}}$, and general proofs can be found in the literature (see e.g. Jaynes 2003; Kapur 2009). We will reproduce the proof here for the sake of completeness in order to discuss the results for the corresponding Hessian matrix in our specific context.

Let us rewrite $H({\mathbf {p}})$ as a function of the vector of $n-1$ first probabilities ${\mathbf {p}}_0=(p_1,\ldots ,p_{n-1})'$, where the last one is recovered from the condition $\mathbf {1'p}_0+p_n=1$, so that

$$ H({\mathbf {p}}_0)=-{\mathbf {p}}'_0\ln {\mathbf {p}}_0-(1-\mathbf {1'p}_0)\ln (1-\mathbf {1'p}_0). $$

Taking the derivatives with respect to ${\mathbf {p}}_0$ yields

$$ \frac{\partial H({\mathbf {p}}_0)}{\partial {\mathbf {p}}_0}=-\ln {\mathbf {p}}_0+{\mathbf {1}}\ln (1-\mathbf {1'p}_0), $$

with derivatives equal to $\mathbf {0}$ when ${\mathbf {p}}_0={\mathbf {1}}(1-\mathbf {1'p}_0)$; i.e., when

$$ {\mathbf {p}}_0=(\mathbf {I+11'})^{-1}{\mathbf {1}}. $$

Using the fact that $(\mathbf {I+11'})^{-1}={\mathbf {I}}-(1/n)\mathbf {11'}$, it thus comes that

$$ {\mathbf {p}}_0={\mathbf {1}}-\frac{1}{n}\mathbf {11'1}={\mathbf {1}}-\frac{n-1}{n}{\mathbf {1}}=\frac{1}{n}{\mathbf {1}}, $$

(10)

where ${\mathbf {1}}$ is the unit vector of lengths $(n-1)$ so that $\mathbf {1'1}=n-1$. Taking additionally derivatives with respect to ${\mathbf {p}}'_0$ yields

$$ \frac{\partial ^2H({\mathbf {p}}_0)}{\partial {\mathbf {p}}_0\partial {\mathbf {p}}'_0}=-\left[ (diag({\mathbf {p}}_0))^{-1}+\frac{1}{1-\mathbf {1'p}_0}\mathbf {11'}\right] , $$

where $1-\mathbf {1'p}_0\ge 0$ and where $\mathbf {11'}$ and $diag({\mathbf {p}}_0)$ are positive semidefinite and positive definite matrices, respectively, thus leading to $\det (H({\mathbf {p}}))<0$ and so the entropy is concave with respect to the probabilities as defined over the unit simplex. In particular, at the absolute minimum where ${\mathbf {p}}_0=(1/n){\mathbf {1}}$, the Hessian matrix is equal to

$$ \frac{\partial ^2H({\mathbf {p}}_0)}{\partial {\mathbf {p}}_0\partial {\mathbf {p}}'_0}\big \vert _{{\mathbf {p}}_0=\frac{1}{n}{\mathbf {1}}}=-n\big ({\mathbf {I}}+\mathbf {11'}\big ). $$

(11)

Appendix A2: Proof for the convexity of the squared norm

It can be proven that the squared norm $||{\mathbf {p}}||^2={\mathbf {p'p}}$ subject to the constraint $\mathbf {1'p}=1$ is convex everywhere, and so $||{\mathbf {p}}||^2$ is also convex over the space restricted by the additional set of linear equations $\mathbf {Ap=b}$. Thus, there is always a solution, and the solution is unique. To prove this, let us rewrite $||{\mathbf {p}}||^2$ as a function of the vector of $n-1$ first probabilities ${\mathbf {p}}_0=(p_1,\ldots ,p_{n-1})'$, where the last one is recovered from the condition $\mathbf {1'p}_0+p_n=1$, so that

$$ ||{\mathbf {p}}_0||^2=\mathbf {p'}_0{\mathbf {p}}_0+(1-\mathbf {1'p}_0)(1-\mathbf {1'p}_0). $$

Taking the derivatives with respect to ${\mathbf {p}}_0$ yields

$$ \frac{\partial ||{\mathbf {p}}_0||^2}{\partial {\mathbf {p}}_0}=2\left[ ({\mathbf {I}}+\mathbf {11'}){\mathbf {p}}_0-{\mathbf {1}})\right] , $$

with derivatives equal to $\mathbf {0}$ when

$$ {\mathbf {p}}_0=(\mathbf {I+11'})^{-1}{\mathbf {1}}\,\Longleftrightarrow \,{\mathbf {p}}_0=\frac{1}{n}{\mathbf {1}}, $$

(12)

Taking additionally derivatives with respect to ${\mathbf {p}}'_0$ yields

$$ \frac{\partial ^2||{\mathbf {p}}_0||^2}{\partial {\mathbf {p}}_0\partial {\mathbf {p}}'_0}=2(\mathbf {I+11'}), $$

showing that the Hessian matrix does not depend on ${\mathbf {p}}_0$ and is equal up to a multiplicative constant to the MaxEnt Hessian matrix when ${\mathbf {p}}_0=(1/n){\mathbf {1}}$, as seen from Eq. (11).

Appendix A3: Polytopes and simplices

Let us define a convex polytope $\overline{P}({\mathbf {V}})$ in ${\mathbb {R}}^n$ as

$$\begin{aligned} \overline{P}({\mathbf {V}})=\{{\mathbf {x}}:{\mathbf {x}}={\mathbf {V}}\varvec{\lambda },{\mathbf {1'}}\varvec{\lambda }=1,\varvec{\lambda }\in {\mathbb {R}}_{\ge 0}^m,{\mathbf {V}}\in {\mathbb {R}}^n\times {\mathbb {R}}^m,{\mathbf {x}}\in {\mathbb {R}}^n\} \end{aligned}$$

(with $m<\infty$) where ${\mathbf {V}}$ is the matrix specifying the m vertices of the polytope, and the coordinates in ${\mathbb {R}}^n$ of the ith vertex are given by the ith column of ${\mathbf {V}}$. ${\mathbf {V}}\varvec{\lambda }$ is a convex linear combination of these vertices, and thus, $\overline{P}({\mathbf {V}})$ is the convex hull (i.e., a convex faceted solid – or convex bounded polyhedron – defined over ${\mathbb {R}}^n$) generated by these linear combinations over ${\mathbb {R}}^n$. Alternatively, let us define

$$ P({\mathbf {A}},{\mathbf {b}})=\{{\mathbf {x}}\in {\mathbb {R}}^n:{\mathbf {A}}{\mathbf {x}}\le {\mathbf {b}}\} $$

as the (possibly unbounded) convex polyhedron generated by the intersection of the set of half-spaces ${\mathbf {A}}{\mathbf {x}}\le {\mathbf {b}}$. If this polyhedron is bounded, then $P({\mathbf {A}},{\mathbf {b}})$ is called the half-spaces (or H-) representation of the corresponding polytope $\overline{P}({\mathbf {V}})$. Identifying the vertices ${\mathbf {V}}$ generated by this intersection of half-spaces is part of the so-called enumeration problem, for which an efficient algorithm is available. From above, it is clear that any polytope can be univoquely defined either from its half-spaces definition or from its vertices. From topological properties, the intersection of two polytopes $\overline{P}({\mathbf {V}}_1)$ and $\overline{P}({\mathbf {V}}_2)$ is a new polytope, where each polytope can possibly be specified by its H-representation if needed.

A specific polytope of interest here is the simplex $S({\mathbf {W}})\equiv \overline{P}({\mathbf {W}})$ when $m=n$, so that it corresponds to a polytope that has n vertices in ${\mathbb {R}}^n$; i.e., these vertices lie on the same hyperplane in ${\mathbb {R}}^n$ so the polytope is, in fact, $(n-1)$ dimensional. In particular, the unit simplex is defined as $S({\mathbf {I}})$, where ${\mathbf {I}}$ is the orthonormal basis in ${\mathbb {R}}^n$. By the light of the general results given above, it thus comes that the intersection between the unit simplex $S({\mathbf {I}})$ (i.e., a polytope) with another polytope $P({\mathbf {A}},{\mathbf {b}})$ yields another simplex $S({\mathbf {W}})$ that is a subset of $S({\mathbf {I}})$. From the topological properties again, it comes that

• 1. any simplex $S({\mathbf {W}})$ can be represented as a simplicial complex, i.e. as the union of a finite set of simplices lying on the same hyperplane in ${\mathbb {R}}^n$, with

  $$ S({\mathbf {W}})=\bigcup _i S({\mathbf {W}}_i) \quad {\text{where }}\dim S({\mathbf {W}}_i)\cap S({\mathbf {W}}_j)<n \quad \forall i\ne j $$

  the condition on the intersection meaning that two distinct simplices $S({\mathbf {W}}_i)$ and $S({\mathbf {W}}_j)$ can, at most, share a common face (noting that the empty set is a face of every simplex, so that for simplices built on a disjoint set of vertices, their intersection, which is the empty set, obeys the previous definition);

• 2. for any arbitrary simplex $S({\mathbf {W}}_i)$, there is always an affine transformation that allows us to map the vertices of the unit simplex $S({\mathbf {I}})$ of the same dimension n to the vertices of $S({\mathbf {W}}_i)$. That is, obviously there exists an infinite set of possible linear transformations such that

  $$ {\mathbf {W}}_i=\mathbf {CI}+{\mathbf {D}}. $$

  In particular, using, e.g., arbitrarily the first column $\mathbf {w}_{i1}$ of ${\mathbf {W}}_i$,

  $$ \mathbf {C=W}_i-{\mathbf {D}}\quad \mathbf {D=w}_{i1}{\mathbf {1'}}, $$

  where ${\mathbf {W}}_i-{\mathbf {D}}$ is a translation of the simplex so that the first vertex is now at the origin of the orthonormal basis.

Considering the particular case of $P({\mathbf {A}},{\mathbf {b}})$ and $S({\mathbf {I}})$, their intersection $\varOmega$ is thus a simplicial complex, with

$$ \varOmega =P({\mathbf {A}},{\mathbf {b}})\cap S({\mathbf {I}})=\bigcup _i S({\mathbf {W}}_i), $$

where (i) all vertices of all simplices $S({\mathbf {W}}_i)$s lie on the same hyperplane, which is the hyperplane where the vertices of $S({\mathbf {I}})$ lie, and (ii) each $S({\mathbf {W}}_i)$ is an affine transform from $S({\mathbf {I}})$ itself. Because all vertices lie on the same hyperplane, their projection ${\mathbf {W}}_{i,p}$ on the same $n-1$ dimensional subspace as obtained by dropping one line in ${\mathbf {W}}_i$ (the same line for all ${\mathbf {W}}_i$’s, of course) yields a set of n points in as $(n-1)$ dimensional space, with the corresponding enclosed volumes $v_i$ given by

$$ v_i=\frac{1}{2}\left| \det \left( \begin{array}{c} {\mathbf {W}}_{i,p} \\ {\mathbf {1'}} \end{array} \right) \right| . $$

These volumes are in the same ratios as the corresponding surfaces of the simplices over the hyperplane. In other words,

$$ w_i=\frac{v_i}{\sum _i v_i} $$

is the percentage of the surface of the simplicial complex over the hyperplane which is covered by the ith simplex.