Computational Algebraic Methods in Efficient Estimation

Abstract

A strong link between information geometry and algebraic statistics is made by investigating statistical manifolds which are algebraic varieties. In particular it is shown how first and second order efficient estimators can be constructed, such as bias corrected Maximum Likelihood and more general estimators, and for which the estimating equations are purely algebraic. In addition it is shown how Gröbner basis technology, which is at the heart of algebraic statistics, can be used to reduce the degrees of the terms in the estimating equations. This points the way to the feasible use, to find the estimators, of special methods for solving polynomial equations, such as homotopy continuation methods. Simple examples are given showing both equations and computations.

Appendices

A Normal Forms

A basic text for the materials in this section is [9]. The rapid growth of modern computational algebra can be credited to the celebrated Buchberger’s algorithm [8].

A monomial ideal $I$ in a polynomial ring $K[x_1,\ldots ,x_n]$ over a field $K$ is an ideal for which there is a collection of monomials $f_1,\ldots , f_m$ such that any $g \in I$ can be expressed as a sum

$$ g = \sum _{i=1}^m g_i(x)f_i(x) $$

with some polynomials $g_i\in K[x_1,\ldots ,x_n]$. We can appeal to the representation of a monomial $x^{\alpha }=x_1^{\alpha _1}\ldots x_n^{\alpha _n}$ by its exponent $\alpha =(\alpha _1,\ldots ,\alpha _n)$. If $\beta \ge 0$ is another exponent then

$$x^{\alpha } x^{\beta } = x^{\alpha + \beta },$$

and $\alpha + \beta$ is in the positive (shorthand for non-negative) “orthant” with corner at $\alpha$. The set of all monomials in a monomial ideal is the union of all positive orthants whose “corners” are given by the exponent vectors of the generating monomial $f_1, \ldots , f_m$. A monomial ordering written $x^{\alpha } \prec x^{\beta }$ is a total (linear) ordering on monomials such that for $\gamma \ge 0$, $x^{\alpha } \prec x^{\beta } \Rightarrow x^{\alpha +\gamma } \prec x^{\beta + \gamma }$. Any polynomial $f(x)$ has a leading terms with respect to $\prec$, written $LT(f)$.

There are, in general, many ways to express a given ideal $I$ as being generated from a basis $I = \langle f_1,\ldots , f_m\rangle$. That is to say, there are many choices of basis. Given an ideal $I$ a set $\{g_1, \ldots g_m\}$ is called a Gröbner basis (G-basis) if:

$$\langle LT(g_1), \ldots , LT(g_m)\rangle \; = \; \langle LT(I)\rangle ,$$

where $\langle LT(I)\rangle$ is the ideal generated by all the monomials in $I$. We sometimes refer to $\langle LT(I) \rangle$ as the leading term ideal. Any ideal $I$ has a Gröbner basis and any Gröbner basis in the ideal is a basis of the ideal.

Given a monomial ordering and an ideal expressed in terms of the G-basis, $I\;=\; \langle g_1,\ldots , g_m\rangle$, any polynomial $f$ has a unique remainder with respect the quotient operation $K[x_1, \ldots , x_k]/I$. That is

$$f = \sum _{i=1}^m s_i(x)g_i(x) + r(x).$$

We call the remainder $r(x)$ the normal form of $f$ with respect to $I$ and write $NF(f)$. Or, to stress the fact that it may depend on $\prec$, we write $NF(f, \prec )$. Given a monomial ordering $\prec$, a polynomial $f=\sum _{\alpha \in L} \theta _{\alpha } x^{\alpha }$ for some $L$ is a normal form with respect to $\prec$ if $x^{\alpha } \notin \langle LT(f) \rangle$ for all $\alpha \in L$. An equivalent way of saying this is: given an ideal $I$ and a monomial ordering $\prec$, for every $f \in K[x_1,\ldots ,x_k]$ there is a unique normal form $NF(f)$ such that $f-NF(f) \in I$.

B Homotopy Continuation Method

Homotopy continuation method is an algorithm to find the solutions of simultaneous polynomial equations numerically. See, for example, [19, 24] for more details of the algorithm and theory.

We will explain the method briefly by a simple example of 2 equations with 2 unknowns

$$ \text {Input:} f, g \in \mathbb {R}[x,y] $$

$$ \text {Output: The solutions of} \,f(x,y)=g(x,y)=0. $$

 

Step 1 :

Select arbitrary polynomials of the form:

$$\begin{aligned} f_0(x,y):=f_0(x):=a_1x^{d_1}-b_1&=0,\nonumber \\ g_0(x,y):=g_0(y):=a_2y^{d_2}-b_2&=0 \end{aligned}$$

(6.8)

where $d_1= \deg (f)$ and $d_2=\deg (g)$. Polynomial equations in this form are easy to solve.

Step 2 :

Take the convex combinations:

$$\begin{aligned} f_t(x,y):=\,&t f(x,y)+ (1-t) f_0(x,y),\\ g_t(x,y):=\,&t g(x,y)+ (1-t) g_0(x,y) \end{aligned}$$

then our target becomes the solution for $t=1$.

Step 3 :

Compute the solution for $t=\delta$ for small $\delta$ by the solution for $t=0$ numerically.

Step 4 :

Repeat this until we obtain the solution for $t=1$.

 

Figure 6.4 shows a sketch of the algorithm. This algorithm is called the (linear) homotopy continuation method and justified if the path connects $t=0$ and $t=1$ continuously without an intersection. That can be proved for almost all $a$ and $b$. See [19].

Fig. 6.4

Paths for the homotopy continuation method

For each computation for the homotopy continuation method, the number of the paths is the number of the solutions of (6.8). In this case, the number of paths is $d_1 d_2$. In general case with $m$ unknowns, it becomes $\prod _{i=1}^m d_i$ and this causes a serious problem for computational cost. Therefore decreasing the degree of second-order efficient estimators plays an important role for the homotopy continuation method.

Note that in order to solve this computational problem, the authors of [16] proposed the nonlinear homotopy continuation methods (or the polyhedral continuation methods). But as we can see in Sect. 6.5.2, the degree of the polynomials still affects the computational costs.