Estimation of the volume under the ROC surface in presence of nonignorable verification bias

  • Khanh To Duc
  • Monica ChiognaEmail author
  • Gianfranco Adimari
  • for the Alzheimer’s Disease Neuroimaging Initiative
Original Paper


The volume under the receiver operating characteristic surface (VUS) is useful for measuring the overall accuracy of a diagnostic test when the possible disease status belongs to one of three ordered categories. In medical studies, the VUS of a new test is typically estimated through a sample of measurements obtained by some suitable sample of patients. However, in many cases, only a subset of such patients has the true disease status assessed by a gold standard test. In this paper, for a continuous-scale diagnostic test, we propose four estimators of the VUS which accommodate for nonignorable missingness of the disease status. The estimators are based on a parametric model which jointly describes both the disease and the verification process. Identifiability of the model is discussed. Consistency and asymptotic normality of the proposed estimators are shown, and variance estimation is discussed. The finite-sample behavior is investigated by means of simulation experiments. An illustration is provided.


Diagnostic test Nonignorable missing data mechanism ROC analysis 



The authors thank the Alzheimers Disease Neuroimaging Initiative research group for kindly permitting access to the data analyzed in this paper. Data collection and sharing for this project was funded by the Alzheimer’s Disease Neuroimaging Initiative (ADNI) (National Institutes of Health Grant U01 AG024904) and DOD ADNI (Department of Defense Award Number W81XWH-12-2-0012). ADNI is funded by the National Institute on Aging, the National Institute of Biomedical Imaging and Bioengineering, and through generous contributions from the following: AbbVie, Alzheimers Association; Alzheimers Drug Discovery Foundation; Araclon Biotech; BioClinica, Inc.; Biogen; Bristol-Myers Squibb Company; CereSpir, Inc.; Eisai Inc.; Elan Pharmaceuticals, Inc.; Eli Lilly and Company; EuroImmun; F. Hoffmann-La Roche Ltd and its affiliated company Genentech, Inc.; Fujirebio; GE Healthcare; IXICO Ltd.; Janssen Alzheimer Immunotherapy Research & Development, LLC.; Johnson & Johnson Pharmaceutical Research & Development LLC.; Lumosity; Lundbeck; Merck & Co., Inc.; Meso Scale Diagnostics, LLC.; NeuroRx Research; Neurotrack Technologies; Novartis Pharmaceuticals Corporation; Pfizer Inc.; Piramal Imaging; Servier; Takeda Pharmaceutical Company; and Transition Therapeutics. The Canadian Institutes of Health Research is providing funds to support ADNI clinical sites in Canada. Private sector contributions are facilitated by the Foundation for the National Institutes of Health ( The grantee organization is the Northern California Institute for Research and Education, and the study is coordinated by the Alzheimer’s Disease Cooperative Study at the University of California, San Diego. ADNI data are disseminated by the Laboratory for Neuro Imaging at the University of Southern California.


  1. Baker SG (1995) Evaluating multiple diagnostic tests with partial verification. Biometrics 51(1):330–337CrossRefzbMATHGoogle Scholar
  2. Chi YY, Zhou XH (2008) Receiver operating characteristic surfaces in the presence of verification bias. J R Stat Soc Ser C (Appl Stat) 57(1):1–23MathSciNetCrossRefGoogle Scholar
  3. Fluss R, Reiser B, Faraggi D, Rotnitzky A (2009) Estimation of the ROC curve under verification bias. Biom J 51(3):475–490MathSciNetCrossRefGoogle Scholar
  4. Fluss R, Reiser B, Faraggi D (2012) Adjusting ROC curve for covariates in the presence of verification bias. J Stat Plan Inference 142(1):1–11MathSciNetCrossRefzbMATHGoogle Scholar
  5. Kang L, Tian L (2013) Estimation of the volume under the ROC surface with three ordinal diagnostic categories. Comput Stat Data Anal 62:39–51MathSciNetCrossRefzbMATHGoogle Scholar
  6. Li J, Zhou XH (2009) Nonparametric and semiparametric estimation of the three way receiver operating characteristic surface. J Stat Plan Inference 139(12):4133–4142MathSciNetCrossRefzbMATHGoogle Scholar
  7. Little RJ, Rubin DB (2002) Statistical analysis with missing data. Wiley, New YorkCrossRefzbMATHGoogle Scholar
  8. Liu D, Zhou XH (2010) A model for adjusting for nonignorable verification bias in estimation of the ROC curve and its area with likelihood-based approach. Biometrics 66(4):1119–1128MathSciNetCrossRefzbMATHGoogle Scholar
  9. Nakas CT, Yiannoutsos CT (2004) Ordered multiple-class ROC analysis with continuous measurements. Stat Med 23(22):3437–3449CrossRefGoogle Scholar
  10. Rotnitzky A, Faraggi D, Schisterman E (2006) Doubly robust estimation of the area under the receiver-operating characteristic curve in the presence of verification bias. J Am Stat Assoc 101(475):1276–1288MathSciNetCrossRefzbMATHGoogle Scholar
  11. Scurfield BK (1996) Multiple-event forced-choice tasks in the theory of signal detectability. J Math Psychol 40(3):253–269CrossRefzbMATHGoogle Scholar
  12. To Duc K (2017) bcROCsurface: an R package for correcting verification bias in estimation of the ROC surface and its volume for continuous diagnostic tests. BMC Bioinform 18(1):503CrossRefGoogle Scholar
  13. To Duc K, Chiogna M, Adimari G (2016) Bias-corrected methods for estimating the receiver operating characteristic surface of continuous diagnostic tests. Electron J Stat 10(2):3063–3113MathSciNetCrossRefzbMATHGoogle Scholar
  14. van der Vaart AW (2000) Asymptotic statistics. Cambridge University Press, CambridgeGoogle Scholar
  15. Xiong C, van Belle G, Miller JP, Morris JC (2006) Measuring and estimating diagnostic accuracy when there are three ordinal diagnostic groups. Stat Med 25(7):1251–1273MathSciNetCrossRefGoogle Scholar
  16. Zhang Y, Alonzo TA (2018) Estimation of the volume under the receiver-operating characteristic surface adjusting for non-ignorable verification bias. Stat Methods Med Res 27(3):715–739MathSciNetCrossRefGoogle Scholar
  17. Zhou XH, Castelluccio P (2003) Nonparametric analysis for the ROC areas of two diagnostic tests in the presence of nonignorable verification bias. J Stat Plan Inference 115(1):193–213MathSciNetCrossRefzbMATHGoogle Scholar
  18. Zhou XH, Castelluccio P (2004) Adjusting for non-ignorable verification bias in clinical studies for Alzheimer’s disease. Stat Med 23(2):221–230CrossRefGoogle Scholar
  19. Zhou XH, Rodenberg CA (1998) Estimating an ROC curve in the presence of nonignorable verification bias. Commun Stat 27(3):273–285Google Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Department of Statistical SciencesUniversity of PadovaPadovaItaly
  2. 2.Department of Statistical Sciences “Paolo Fortunati”University of BolognaBolognaItaly

Personalised recommendations