Efficient computation of nonparametric survival functions via a hierarchical mixture formulation

Abstract

We propose a new algorithm for computing the maximum likelihood estimate of a nonparametric survival function for interval-censored data, by extending the recently-proposed constrained Newton method in a hierarchical fashion. The new algorithm makes use of the fact that a mixture distribution can be recursively written as a mixture of mixtures, and takes a divide-and-conquer approach to break down a large-scale constrained optimization problem into many small-scale ones, which can be solved rapidly. During the course of optimization, the new algorithm, which we call the hierarchical constrained Newton method, can efficiently reallocate the probability mass, both locally and globally, among potential support intervals. Its convergence is theoretically established based on an equilibrium analysis. Numerical study results suggest that the new algorithm is the best choice for data sets of any size and for solutions with any number of support intervals.

Appendix: Linear regression over a simplex

Consider the constrained least squares problem:

for δ>0. It can be solved by the NNLS algorithm of Lawson and Hanson (1974), after a transformation suggested by Dax (1990). Letting y=x/δ and c=b/δ, it is apparent that the problem is equivalent to

which is further equivalent to

(22)

where P=A−(c,…,c). The solution to problem (22) can be found by solving the following least squares problem with only non-negativity constraints:

(23)

By relating the Karush-Kuhn-Tucker conditions for both problems, Dax established that if ${\tilde {\mathbf {y}}}$ solves problem (23), then ${\tilde {\mathbf {y}}}/{\tilde {\mathbf {y}}}^{{\top }}\mathbf {1}$ solves problem (22).

Problem (23) can be solved by the NNLS algorithm of Lawson and Hanson (1974).