Is EM really necessary here? Examples where it seems simpler not to use EM

Abstract

If one is to judge by counts of citations of the fundamental paper (Dempster in JRSSB 39: 1–38, 1977), EM algorithms are a runaway success. But it is surprisingly easy to find published applications of EM that are unnecessary, in the sense that there are simpler methods available that will solve the relevant estimation problems. In particular, such problems can often be solved by the simple expedient of submitting the observed-data likelihood (or log-likelihood) to a general-purpose routine for unconstrained optimization. This can dispense with the need to derive and code (or modify) the E and M steps, a process which can sometimes be laborious or error-prone. Here, I discuss six such applications of EM in some detail, and in an appendix describe briefly some others that have already appeared in the literature. Whether these are atypical of applications of EM seems an open question, although one that may be difficult to answer; this question is of relevance to current practice, but may also be of historical interest. But it is clear that there are problems traditionally solved by EM (e.g. the fitting of finite mixtures of distributions) that can also be solved by other means. It is suggested that, before going to the effort of devising an EM algorithm to use on a new problem, the researcher should consider whether other methods (e.g. direct numerical maximization or an MM algorithm of some other kind) may be either simpler to implement or more efficient.

Appendix

1.1 A Other published examples

Here, I discuss briefly some previously published examples in which it is apparently simpler not to use EM, plus the applications chapters of Zucchini et al. (2016), all of which use DNM but not EM.

1.1.1 A.1 EM modified to allow for constraints

It is sometimes stated that EM allows automatically for constraints on parameters. For instance, Lange (2010, p. 223) writes that “[...] the EM algorithm handles parameter constraints gracefully. Constraint satisfaction is by definition built into the solution of the M step.” This could, however, be misunderstood to mean all constraints on parameters. EM does indeed handle many constraints automatically, e.g. the nonnegativity and unit-sum constraints on the transition probabilities in a hidden Markov model (Zucchini et al. 2016, p. 72). But see (e.g.) Kim and Taylor (1995) and Shi et al. (2005), the very purpose of which is to modify EM in order to incorporate (respectively) certain linear equality or inequality constraints on parameters.

There has been one very determined but apparently unnecessary attempt to modify EM in order to allow for the linear inequality constraints arising very naturally in one particular problem in genetics, even splitting the M step into seven cases in order to do so (Zhou et al. 2008). There the problem is just one of maximizing a (nonlinear) function of three variables subject to four linear inequality constraints and is easily solved by using the constrained optimizer constrOptim provided by R. EM makes this problem harder than it need be. For further details, see MacDonald and Nkalashe (2015).

1.1.2 A.2 The Altham–Poisson distribution

Leask and Haines (2015) and Leask (2009) considered very fully the possible use of EM for parameter estimation in the case of an Altham–Poisson distribution, that is, a “multiplicative binomial” distribution (Altham 1978) for which n, the number of trials, is taken to have a Poisson distribution. A multiplicative binomial has probability mass function of the form

$$\Pr (Y=y) \propto {n \atopwithdelims ()y} p^y (1-p)^{n-y} \theta ^{y(n-y)}, \quad y=0, 1, \ldots , n ,$$

with $p \in [0,1]$ and $\theta >0$. They concluded that the formulation of EM for such a distribution is “subtle and somewhat complicated” and, because EM was slow to converge, chose instead to maximize the log-likelihood directly.

1.1.3 A.3 Fitting truncated normal distributions

It is straightforward to evaluate the (density-approximated) likelihood of a sample from a truncated normal distribution and maximize it numerically. Nevertheless, Tian et al. (2018) go to considerable lengths to develop an algorithm of EM type to fit truncated normal distributions. Their two examples of model fitting can easily be carried out instead by DNM, although in one case it is clear that their truncated normal is much inferior to a log-normal model. This problem has been more fully discussed by MacDonald (2018).

1.1.4 A.4 Maximization of penalized likelihood

Lee and Pawitan (2014) describe how to estimate the variances of estimators based on the maximization of penalized likelihood; they assume that these estimators have been found by EM. MacDonald and Lapham (2016) describe how to accomplish the same and more without using EM, by instead maximizing the penalized likelihood directly. For the two examples of Lee and Pawitan they find the MLEs, their standard errors, and confidence (or credibility) intervals of two types: those of Wald type and those based directly on penalized likelihood, which can differ considerably from those of Wald type.

1.1.5 A.5 Fitting a zero-truncated Poisson

Meng (1997) describes how to use (inter alia) EM to fit a Poisson distribution to count data from which the number of zeros observed is missing. Among the methods he describes is Newton–Raphson. MacDonald (2014a) claims that the apparent failure of Newton–Raphson for certain starting-values is due only to the fact that the obvious positivity constraint on the Poisson mean has been ignored. Meng (2014) has replied, but does not agree entirely.

1.1.6 A.6 The examples of MacDonald (2014b)

MacDonald (2014b) describes a range of examples in which EM has apparently been used unnecessarily and compares EM with DNM. Included are the analysis of ABO blood-group data, and the fitting of Dirichlet distributions. There has been little or no response published which disagrees materially with the conclusions and recommendations of that paper.

1.1.7 A.7 The applications of Zucchini et al. (2016)

The applications chapters of Zucchini et al. (2016, Chapters 15–24) present a wide variety of models of hidden Markov type or similar, some simple, some complex, but all fitted by direct numerical maximization of likelihood. There is little if any mention of EM in those chapters, as it usually seemed redundant or over-complicated—in spite of the strong historical connection between hidden Markov models and EM, and the traditional use of the Baum–Welch algorithm (i.e. EM) in such models. This seems to support the argument that, in some contexts where EM is often used, it is inessential.

1.2 B A simple optimization problem with implicit constraint

Boyd and Vandenberghe (2004, Exercise 9.10b) present a very simple minimization problem for which they correctly state that the “pure” (i.e. undamped) Newton method can diverge. The problem is to minimize $f(x) = x-\log x$. It is straightforward to establish (analytically) that f has a unique minimum at $x=1$, but if pure Newton is started from $x_0=3$ (for instance), the algorithm does indeed diverge.

But the very nature of f is such that there is the implicit constraint $x>0$. One can (and should) therefore replace the problem by the unconstrained problem of minimizing $g(y) = \exp (y)-y$—or else impose the positivity constraint in some other way. If pure Newton, without any embellishment whatsoever, is then started from $y_0=\log 3$, it converges very fast to $y=0$, as it should. Not surprisingly, the unconstrained minimizer nlm, applied to g and starting from $y_0=\log 3$, also converges very fast to $y=0$.

If, however, I ignore my own advice, put f (unconstrained) into nlm, and start from $x_0=3$, with a few warnings nlm converges in ten iterations to $x=0.9999995$. nlm appears to be sufficiently robust to withstand some rough treatment. Nevertheless, if there are constraints it is unwise to ignore them. And, whatever method one chooses to attempt an optimization problem, there is always the possibility that sufficiently extreme starting values will cause under- or overflow in the objective, and thereby cause the method to fail.