Empirical identifiability in finite mixture models

Abstract

Although the parameters in a finite mixture model are unidentifiable, there is a form of local identifiability guaranteeing the existence of the identifiable parameter regions. To verify its existence, practitioners use the Fisher information on the estimated parameters. However, there exist model/data situations where local identifiability based on Fisher information does not correspond to that based on the likelihood. In this paper, we propose a method to empirically measure degree of local identifiability on the estimated parameters, empirical identifiability, based on one’s ability to construct an identifiable likelihood set. From a detailed topological study of the likelihood region, we show that for any given data set and mixture model, there typically exists limited range of confidence levels where the likelihood region has a natural partition into identifiable subsets. At confidence levels that are too high, there is no natural way to use the likelihood to resolve the identifiability problem.

Appendix: Proofs

Proof of Theorem 1

Suppose that one starts at the highest elevation in the mixture likelihood, $\psi _\mathrm{MLE}$. Then the likelihood region at $\psi _\mathrm{MLE}$ is equal to the set of $K!$ modes. If the elevation $c$ is just below $\psi _\mathrm{MLE}$, the likelihood region consists of $K!$ disjoint modal regions $C_{c}(\hat{{\theta }}^{\sigma })$.

However, if the elevation $c$ goes too low, some properties in Definition 3 could fail so that a unimodal partition cannot exist. Two different problems could arise. The first case is where there exists a secondary mode in the likelihood, with the elevation $\psi _\mathrm{2nd}$. In this case, for the elevation $c$ at or just below $\psi _\mathrm{2nd}$, a secondary set of modal regions is formed containing the $K!$ secondary modes. Since these points are not path-connected to the MLE modes, the property (P3) would be violated. If the primary and secondary modal regions become reconnected at a lower level of $c$, then, even if identifiable, each element contains two modes, thereby violating property (P4).

A second case where a unimodal partition could fail is when $c \le \psi _\mathrm{mm}$, the minimal elevation of the maximin path connecting $C_{c}(\hat{{\theta }})$ to a permuted region $C_{c}(\hat{{\theta }}^{\sigma })$. This causes a violation of property (P1).

As long as the elevation $c$ is larger than $\max \{\psi _\mathrm{mm}, \psi _\mathrm{2nd} \}$, however, then there are no secondary modal regions possible, and there cannot exist connections between the modal regions of the MLE group, as there exist no saddle points. Therefore the $K!$ modal regions $C_{c}(\hat{{\theta }}^{\sigma })$ are disjoint and their union is the elevation $c$ likelihood region $C_{c}^\mathrm{LR}$, so that we have verified properties (P1) and (P3).

We claim that the second property (P2) holds because the $K!$ modal regions $C_{c}(\hat{{\theta }}^{\sigma })$ are disjoint by (P1) and each is connected (by definition). The argument for identifiability goes as follows: first, given any ${\theta }$ in $C_{c}(\hat{{\theta }})$, its permutation ${\theta }^{\sigma }$ cannot also be in the same modal region $C_{c}(\hat{{\theta }})$, as we know it lies in a disjoint $C_{c}(\hat{{\theta }}^{\sigma })$. Secondly, we suppose, for purposes of contradiction, that $C_{c}(\hat{{\theta }})$ contains a degenerate point ${\theta }_{0}$, but is not path-connected to any $C_{c}(\hat{{\theta }}^{\sigma })$, for any permutation $\sigma$. Then the permuted ${\theta }_{0}$, ${\theta }_{0}^{\sigma }$, is contained in only $C_{c}(\hat{{\theta }}^{\sigma })$. Since the region of ${\theta }$ values that generate the same degenerate mixing distribution and hence the same likelihood as ${\theta }_{0}$ is a connected set, ${\theta }_{0}^{\sigma }$ should be in the same connected set. Thus, at the elevation $c \le \psi _\mathrm{deg}$, $C_{c}(\hat{{\theta }})$ and $C_{c}(\hat{{\theta }}^{\sigma })$ are connected to each other by a path. But this is contradicted with the property (P1) that the $K!$ modal regions $C_{c}(\hat{{\theta }}^{\sigma })$ are disjoint. $\square$

Proof of Theorem 2

Suppose the two modal regions, $C_{c}(\hat{{\theta }})$ and $C_{c}(\hat{{\theta }}^{\sigma })$, are connected to each other at the elevation $c$. Then a maximin path connecting them has $\psi _\mathrm{mm} \ge c$. Any path connecting the modes necessarily intersects the hyperplane of Eq (6) in one or more points. Let $c^{\star }$ be the minimal value of the likelihood on the intersection set. Then $c^{\star } \ge \psi _\mathrm{mm}$ by the definition of $\psi _\mathrm{mm}$. However, we also have $c^{\star } \le \psi _\mathrm{hyp}$ by the definition of the latter. We therefore have $\psi _\mathrm{mm} \le \psi _\mathrm{hyp}$. $\square$

Proof of Theorem 3

Suppose there exists a continuous path of $\gamma (t)$ parameter values that connects the profile modes $\gamma ({\theta }_{A})$ and $\gamma ({\theta }_{B})$ such that the profile likelihood along the path stays above the elevation $c$. Then the path ($\gamma (t)$, $\hat{\phi }(\gamma (t))$) is a continuous path in the full parameter space that connects the two modes and whose likelihood stays above $c$. $\square$

Proof of Theorem 4

Suppose $K=2$. Then the default hyperplane generated by Eq.(8) consists of all mixtures with $\xi _{1}=\xi _{2}$, and so $\psi _\mathrm{hyp}=\psi _\mathrm{deg}=\psi _\mathrm{mm}$. Thus, if $c > \psi _\mathrm{deg}$, $C_{c}(\hat{{\theta }})$ and $C_{c}(\hat{{\theta }}^{\sigma })$ cannot have a connecting path. If this is the case, the labels in a selected modal region must satisfy $\xi _{1}> \xi _{2}$ or $\xi _{1} < \xi _{2}$, as the modal set cannot contain points satisfying $\xi _{1}=\xi _{2}$. That is, the points in each modal region must have order restriction labels. The argument for a two-component case extends to $K(>2)$ components. Given a set of ordered $\xi$’s, there is no way to move continuously to a permuted set without one pair becoming equal along the way. $\square$