Skip to main content
Log in

Nonparametric maximum likelihood computation of a U-shaped hazard function

  • Published:
Statistics and Computing Aims and scope Submit manuscript

Abstract

A new algorithm is presented and studied in this paper for fast computation of the nonparametric maximum likelihood estimate of a U-shaped hazard function. It successfully overcomes a difficulty when computing a U-shaped hazard function, which is only properly defined by knowing its anti-mode, and the anti-mode itself has to be found during the computation. Specifically, the new algorithm maintains the constant hazard segment, regardless of its length being zero or positive. The length varies naturally, according to what mass values are allocated to their associated knots after each updating. Being an appropriate extension of the constrained Newton method, the new algorithm also inherits its advantage of fast convergence, as demonstrated by some real-world data examples. The algorithm works not only for exact observations, but also for purely interval-censored data, and for data mixed with exact and interval-censored observations.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  • Ayer, M., Brunk, H.D., Ewing, G.M., Reid, W.T., Silverman, E.: An empirical distribution function for sampling with incomplete information. Ann. Math. Stat. 26, 641–647 (1955)

    Article  MathSciNet  MATH  Google Scholar 

  • Banerjee, M.: Estimating monotone, unimodal and U-shaped failure rates using asymptotic pivots. Stat. Sin. 18, 467–492 (2008)

    MathSciNet  MATH  Google Scholar 

  • Bray, T.A., Crawford, G.B., Proschan, F.: Maximum Likelihood Estimation of a U-shaped Failure Rate Function. Defense Technical Information Center, Mathematical Note 534, Boeing Research Laboratories, Seattle (1967)

  • Dümbgen, L., Freitag-Wolf, S., Jongbloed, G.: Estimating a unimodal distribution from interval-censored data. J. Am. Stat. Assoc. 101, 1094–1106 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  • Grenander, U.: On the theory of mortality measurement. II. Skand. Aktuarietidskr. 39, 125–153 (1956)

    MathSciNet  MATH  Google Scholar 

  • Groeneboom, P., Jongbloed, G.: Nonparametric Estimation under Shape Constraints: Estimators, Algorithms and Asymptotics. Cambridge University Press, Cambridge (2014)

    Book  MATH  Google Scholar 

  • Groeneboom, P., Jongbloed, G., Wellner, J.A.: The support reduction algorithm for computing non-parametric function estimates in mixture models. Scand. J. Stat. 35, 385–399 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  • Hall, P., Huang, L.S., Gifford, J.A., Gijbels, I.: Nonparametric estimation of hazard rate under the constraint of monotonicity. J. Comput. Graph. Stat. 10, 592–614 (2001)

    Article  MathSciNet  Google Scholar 

  • Huang, J., Wellner, J.A.: Estimation of a monotone density or monotone hazard under random censoring. Scand. J. Stat. 22, 3–33 (1995)

  • Jankowski, H., Wang, I., McCague, H., Wellner, J.A.: R Package ConvexHaz: Nonparametric MLE/LSE of Convex Hazard (Version 0.2). http://cran.r-project.org/web/packages/convexHaz/index.html (2009)

  • Jankowski, H.K., Wellner, J.A.: Computation of nonparametric convex hazard estimators via profile methods. J. Nonparametric Stat. 21, 505–518 (2009a)

    Article  MathSciNet  MATH  Google Scholar 

  • Jankowski, H.K., Wellner, J.A.: Nonparametric estimation of a convex bathtub-shaped hazard function. Bernoulli 15, 1010–1035 (2009b)

    Article  MathSciNet  MATH  Google Scholar 

  • Kaplan, E.L., Meier, P.: Nonparametric estimation from incomplete observations. J. Am. Stat. Assoc. 53, 457–481 (1958)

    Article  MathSciNet  MATH  Google Scholar 

  • Klein, J .P., Moeschberger, M.L.: Survival Analysis: Techniques for Censored and Truncated Data, 2nd edn. Springer, Berlin (2003)

    MATH  Google Scholar 

  • Lawson, C .L., Hanson, R .J.: Solving Least Squares Problems. Prentice-Hall, Inc, Englewood Cliffs (1974)

    MATH  Google Scholar 

  • Lee, E .T., Wang, J .W.: Statistical Methods for Survival Data Analysis, 3rd edn. Wiley, London (2003)

    Book  MATH  Google Scholar 

  • Meyer, M.C., Habtzghi, D.: Nonparametric estimation of density and hazard rate functions with shape restrictions. J. Nonparametric Stat. 23, 455–470 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  • Mykytyn, S.W., Santner, T.J.: Maximum likelihood estimation of the survival function based on censored data under hazard rate assumptions. Commun. Stat. Theory Methods 10, 1369–1387 (1981)

    Article  MathSciNet  MATH  Google Scholar 

  • Peto, R.: Experimental survival curves for interval-censored data. J. R. Stat. Soc. Ser. C 22, 86–91 (1973)

    Google Scholar 

  • Proschan, F.: Theoretical explanation of observed decreasing failure rate. Technometrics 5, 375–383 (1963)

    Article  Google Scholar 

  • R Core Team: R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna (2015)

  • Reboul, L.: Estimation of a function under shape restrictions: applications to reliability. Ann. Stat. 33, 1330–1356 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  • Schick, A., Yu, Q.: Consistency of the GMLE with mixed case interval-censored data. Scand. J. Stat. 27, 45–55 (2000)

    Article  MathSciNet  MATH  Google Scholar 

  • Tsai, W.-Y.: Estimation of the survival function with increasing failure rate based on left truncated and right censored data. Biometrika 75, 319–324 (1988)

    Article  MathSciNet  MATH  Google Scholar 

  • Turnbull, B.W.: Nonparametric estimation of a survivorship function with doubly censored data. J. Am. Stat. Assoc. 69, 169–173 (1974)

    Article  MathSciNet  MATH  Google Scholar 

  • Wang, Y.: On fast computation of the non-parametric maximum likelihood estimate of a mixing distribution. J. R. Stat. Soc. Ser. B 69, 185–198 (2007)

    Article  MathSciNet  MATH  Google Scholar 

  • Wang, Y.: Dimension-reduced nonparametric maximum likelihood computation for interval-censored data. Comput. Stat. Data Anal. 52, 2388–2402 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  • Wang, Y.: npsurv: Non-parametric Survival Analysis (R Package Version 0.3-4). http://cran.r-project.org/package=npsurv (2015)

  • Wellner, J.A.: Interval censoring case 2: alternative hypotheses. In: Koul, H., Deshpande, J.V. (eds.) Analysis of Censored Data, Proceedings of the Workshop on Analysis of Censored Data, vol. 27, pp. 271–291. University of Pune, Pune (1995)

    Chapter  Google Scholar 

Download references

Acknowledgements

The authors thank the editor, associated editor and two referees for their constructive suggestions, which led to many improvements in the manuscript.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Yong Wang.

Appendices

Appendix 1: Derivatives and the Hessian matrix

Consider the partial derivatives of the modified log-likelihood function (3) with respect to masses, in the general case when there exist both exact and interval-censored data. The first partial derivatives are simply the gradient functions evaluated at the corresponding support points, i.e.,

$$\begin{aligned}&\frac{\partial {\tilde{\ell }}}{\partial \alpha } = d_0(h),\\&\frac{\partial {\tilde{\ell }}}{\partial \nu _j} = d_1(\tau _j; h),\\&\frac{\partial {\tilde{\ell }}}{\partial \mu _j} = d_2(\eta _j; h). \end{aligned}$$

The Hessian matrix \(\mathbf {H}\) can be computed by \( \mathbf {H}= -{\mathbf {D}}^\top {\mathbf {D}}\), where \({\mathbf {D}}\) is \(n \times (|\mathcal{I}| + m)\), with its (ij)-th element given by, for \(i \in 1, \ldots , |\mathcal{I}|\),

$$\begin{aligned} D_{ij} = \left\{ \begin{array}{lll} \frac{1}{h(T_i)}, &{}\quad j = 1; \\ \frac{(\tau _{j-1} - T_i)_+^p}{h(T_i)}, &{} \quad j = 2, \ldots , k+1; \\ \frac{(T_i - \eta _{j-k-1})_+^p}{h(T_i)}, &{} \quad j = k+2, \ldots , m+k+1; \\ \end{array} \right. \end{aligned}$$

and, for \(i = |\mathcal{I}| + 1, \ldots , n\),

$$\begin{aligned} D_{ij} = \left\{ \begin{array}{l} (L_i - R_i) \delta _i(H), \quad j = 1; \\ \{(\tau _{j-1} - R_i)^{p+1}_+ - (\tau _{j-1} - L_i)^{p+1}_+\} \frac{\delta _i(H)}{p+1}, \\ \quad j = 2, \ldots , k+1; \\ \{(L_i - \eta _{j-k-1})^{p+1}_+ - (R_i - \eta _{j-k-1})^{p+1}_+\} \frac{\delta _i(H)}{p+1}, \\ \quad j = k+2, \ldots , m+k+1, \\ \end{array} \right. \end{aligned}$$

where

$$\begin{aligned} \delta _i(H) = \frac{\sqrt{\exp \{H(L_i) - H(R_i)\}}}{1- \exp \{H(L_i) - H(R_i)\}}. \end{aligned}$$

Appendix 2: Proofs

Proof of Lemma 2

Since \({\varvec{\pi }}' = {{\mathrm{arg\,min}}}_{{\varvec{\pi }}\ge 0} ||{\mathbf {S}}{\varvec{\pi }}- \mathbf {b}||\), it holds that \(||{\mathbf {S}}{\varvec{\pi }}' - \mathbf {b}|| \le ||{\mathbf {S}}\mathbf {0}- \mathbf {b}|| = ||\mathbf {b}||\) and hence \(||{\mathbf {S}}{\varvec{\pi }}'|| \le ||{\mathbf {S}}{\varvec{\pi }}' - \mathbf {b}|| + ||\mathbf {b}|| \le 2 ||\mathbf {b}||\). Therefore, \(||{\mathbf {S}}{\varvec{\delta }}|| \le 2 ||\mathbf {b}|| + \sqrt{n}\). Because \(||\mathbf {b}||\) only depends on \(h(T_i)\), \(i \in \mathcal{I}\), which is bounded away from zero for all \(h \in \mathcal{K}_0\). \(\square \)

Proof of Lemma 3

Since \({\varvec{\delta }}\equiv {\varvec{\pi }}' - {\varvec{\pi }}\) maximizes

$$\begin{aligned} ({\mathbf {S}}^{+\top } \mathbf {1}+ {\varvec{\beta }})^\top {\varvec{\delta }}- \frac{1}{2} {\varvec{\delta }}^\top {\mathbf {S}}^{+\top } {\mathbf {S}}{\varvec{\delta }}\end{aligned}$$

under restriction \({\varvec{\pi }}' \ge 0\), we have

$$\begin{aligned} {\varvec{\delta }}^\top {\mathbf {S}}^{+\top } {\mathbf {S}}^+ {\varvec{\delta }}\le ({\mathbf {S}}^{+\top } \mathbf {1}+ {\varvec{\beta }})^\top {\varvec{\delta }}. \end{aligned}$$

Noting the Taylor series expansion

$$\begin{aligned}&{\tilde{\ell }}({\varvec{\pi }}+ {\varvec{\delta }}, {\varvec{\theta }}) - {\tilde{\ell }}({\varvec{\pi }}, {\varvec{\theta }}) \\&\quad = ({\mathbf {S}}^{+\top } \mathbf {1}+ {\varvec{\beta }})^\top {\varvec{\delta }}- \frac{1}{2} {\varvec{\delta }}^\top {\mathbf {S}}^{+\top } {\mathbf {S}}{\varvec{\delta }}+ o(||{\mathbf {S}}{\varvec{\delta }}||^2), \end{aligned}$$

for any \(0< \alpha < \frac{1}{2}\), there is a \(\lambda > 0\) such that if \(||{\mathbf {S}}{\varvec{\delta }}|| \le \lambda \), then

$$\begin{aligned} {\tilde{\ell }}({\varvec{\pi }}+ {\varvec{\delta }}, {\varvec{\theta }}) - {\tilde{\ell }}({\varvec{\pi }}, {\varvec{\theta }}) \ge \alpha ({\mathbf {S}}^{+\top } \mathbf {1}+ {\varvec{\beta }})^\top {\varvec{\delta }}, \end{aligned}$$

thus satisfying the Armijo rule.

If \(||{\mathbf {S}}{\varvec{\delta }}|| > \lambda \), then \(||\sigma ^k {\mathbf {S}}{\varvec{\delta }}|| \le \lambda \) holds for some \(k > 0\). Because \(||{\mathbf {S}}{\varvec{\delta }}|| \le u\) from Lemma 2, we need at most

$$\begin{aligned} \bar{k} \equiv \max \left\{ \left\lceil \log _\sigma \left( \frac{\lambda }{u}\right) \right\rceil , 0\right\} \end{aligned}$$

steps for Armijo’s rule to be satisfied in all cases. \(\square \)

Proof of Theorem 5

Owing to its monotone increase, \({\tilde{\ell }}(h_s)\) will converge to a finite value no greater than \({\tilde{\ell }}({\hat{h}})\), where \({\hat{h}}\) maximizes \({\tilde{\ell }}(h)\). Further,

$$\begin{aligned}&{\tilde{\ell }}(h_{s+1}) - {\tilde{\ell }}(h_s) \\&\quad \ge \alpha \sigma ^{{\bar{k}}} ({\mathbf {S}}_s^{+\top } \mathbf {1}+ {\varvec{\beta }}_s)^\top {\varvec{\delta }}_s \\&\quad \ge \alpha \sigma ^{{\bar{k}}} \left\{ ({\mathbf {S}}_s^{+\top } \mathbf {1}+ {\varvec{\beta }}_s)^\top {\varvec{\delta }}_s - \frac{1}{2} {\varvec{\delta }}_s^\top {\mathbf {S}}_s^{+\top } {\mathbf {S}}_s^+ {\varvec{\delta }}_s\right\} , \end{aligned}$$

because of Armijo’s rule and the nonnegative definiteness of \({\mathbf {S}}_s^{+\top } {\mathbf {S}}_s^+\).

Consider all point-mass directions \(e \in \{\pm e_0, \pm e_{1,\tau }, \pm e_{2,\eta }\}\) from \(h_s\), that are valid in the sense that there exists an \(\epsilon > 0\) such that \(h_s + \epsilon e \in \mathcal{K}\). Denote the steepest ascent direction by

$$\begin{aligned} e^*_{s} = {{\mathrm{arg\,max}}}_{e} d(h_s + e; h_s) \end{aligned}$$

and \({\varvec{\delta }}_s^*\) the direction resulting from \(h_s\) to \(h_s + e_s^*\). Hence, from any \(\epsilon \in {\mathbb {R}}\) such that \(h_s + \epsilon e_s^* \in \mathcal{K}\), we have

$$\begin{aligned}&{\tilde{\ell }}(h_{s+1}) - {\tilde{\ell }}(h_s) \\&\quad \ge \alpha \sigma ^{{\bar{k}}} \left\{ \epsilon ({\mathbf {S}}_s^{+\top } \mathbf {1}+ {\varvec{\beta }}_s)^\top {\varvec{\delta }}_s^* - \frac{\epsilon ^2}{2} {\varvec{\delta }}_s^{*\top } {\mathbf {S}}_s^{+\top } {\mathbf {S}}_s^+ {\varvec{\delta }}_s^*\right\} , \end{aligned}$$

because of the optimality of \({\varvec{\delta }}_s\).

Now, let us assume that \(d(h_s + e_s^*; h_s)\) does not approach 0 as \(s \rightarrow \infty \). There are, hence, infinitely many s such that \(d(h_s + e_s^*; h_s) \ge \tau \), for some \(\tau > 0\). For such an s and noting that

$$\begin{aligned} d(h_s + e_s^*; h_s) = ({\mathbf {S}}_s^{+\top } \mathbf {1}+ {\varvec{\beta }}_s)^\top {\varvec{\delta }}_s^*, \end{aligned}$$

we have, with Lemma 2,

$$\begin{aligned} {\tilde{\ell }}(h_{s+1}) - {\tilde{\ell }}(h_s)&\ge \alpha \sigma ^{{\bar{k}}} \left\{ \epsilon \tau - \frac{\epsilon ^2 u^2}{2}\right\} . \end{aligned}$$

Without loss of generality, assume \(\tau \le u^2\) and let \(\epsilon = \tau / u^2\). As a result,

$$\begin{aligned} {\tilde{\ell }}(h_{s+1}) - {\tilde{\ell }}(h_s)&\ge \frac{\alpha \sigma ^{{\bar{k}}} \tau ^2}{2 u^2}, \end{aligned}$$

a positive value that is independent of s. Since this violates the Cauchy property of a convergent sequence, we must have \(d(h_s + e_s^*; h_s) \rightarrow 0\) as \(s \rightarrow \infty \). Therefore, \(d({\hat{h}}; h_s) \le d(h_s + e_s^*; h_s) (|h_s| + |{\hat{h}}|) \rightarrow 0\) from Corollary 2, and \({\tilde{\ell }}(h_s) \rightarrow {\tilde{\ell }}({\hat{h}})\) from Lemma 1. \(\square \)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, Y., Fani, S. Nonparametric maximum likelihood computation of a U-shaped hazard function. Stat Comput 28, 187–200 (2018). https://doi.org/10.1007/s11222-017-9724-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11222-017-9724-z

Keywords

Navigation