Though many clinical trials rely on medical image evaluations for primary or key secondary endpoints, the methods to monitor reader performance are all too often mired in the legacy use of adjudication rates. If misused, this simple metric can be misleading and sometimes entirely contradictory. Furthermore, attempts to overcome the limitations of adjudication rates using de novo or ad hoc methods often ignore well-established research conducted over the last half-century and can lead to inaccurate conclusions or variable interpretations. Underperforming readers can be missed, expert readers retrained, or worse, replaced. This paper aims to standardize reader performance evaluations using proven statistical methods. Additionally, these methods will describe how to discriminate between scenarios of concern and normal medical interpretation variability. Statistical methods are provided for inter-reader and intra-reader variability and bias, including the adjudicator's bias. Finally, we have compiled guidelines for calculating correct sample sizes, considerations for intra-reader memory recall, and applying alternative designs for independent readers.
This is a preview of subscription content, access via your institution.
Buy single article
Instant access to the full article PDF.
Price excludes VAT (USA)
Tax calculation will be finalised during checkout.
Clinicaltrials.gov search terms: Oncology, 2011–2020.
Clinicaltrials.gov search terms: Recruiting, Active, not recruiting, Completed, Suspended, Terminated, Withdrawn Studies | Interventional Studies | Oncology | imaging OR recist OR Lugano OR cheson OR MRI OR ct OR ultrasound OR pet OR rano OR Choi OR PFS OR ORR OR BOR | Phase Early Phase I, PhaseII, Phase III, Phase II/III.
See ( See Fig. 1 and Online Resource 1 “A Description of Reader Symmetry).
Note that the confidence limits provided in Ford 2016 need to be adjusted by the number of studies to establish an estimate of coverage.
Adjudication rate (the rate of disagreement between two readers evaluating the same patient)
Adjudicator selection rate (the rate that an adjudicator selects a reader from a paired team of two readers when those two readers disagree)
Blinded independent central review (the readers used by an Imaging Core Lab to assess the images scanned at an investigator site)
Confidence interval (to show the range of confidence in a parameter estimate)
Complete response (classification of response)
Date of progression (used for event-related endpoints like progression-free survival)
Fluid-attenuated inversion recovery (an MRI sequence used to suppress fluids)
Glioblastoma multiforme (a to date uncurable brain cancer)
Imaging Core Lab (a central facility that manages the BICR)
Intra-reader variability (the disagreement of readers with a previous assessment made on the same set of images)
Progressive disease (classification of response)
Partial response (classification of response)
Response evaluation criteria in solid tumors (an objective tumor size assessment criteria to categorize a patient’s response to treatment)
Stable disease (classification of response)
Clinton B, Gore A. Reinventing the regulation of cancer drugs: accelerating approval and expanding access. Natl Perform Rev 1996.
Miller C, Noever K. Taking care of your subject’s image: the role of medical imaging core laboratories. Good Clin Pract J. 2003;10(9):21–4.
Conklin J. Interview of James Conklin on the emergence of imaging CROs. In: Raunig D, editor. 2019.
Schmid A, Raunig D, Ford R, Miller C. Radiologists and clinical trials: part 1. The truth about reader disagreements therapeutic innovation & regulatory science. 2021
Sharma M, O'connor JM, Singareddy A, editors. Reader disagreement index: a better measure of overall review quality monitoring in an oncology trial compared to adjudication rate. Medical imaging 2019: image perception, observer performance, and technology assessment; 2019: International Society for Optics and Photonics.
Pintad. Pharmaceutical imaging network for therapeutics and diagnostics 2020. https://Www.Pintad.Net/.
Eldevik OP, Dugstad G, Orrison WW, Haughton VM. The effect of clinical bias on the interpretation of myelography and spinal computed tomography. Radiology. 1982;145(1):85–9.
Sica GT. Bias in research studies. Radiology. 2006;238(3):780–9.
Ford R, Schwartz L, Dancey J, Dodd LE, Eisenhauer EA, Gwyther S, et al. Lessons learned from independent central review. Eur J Cancer. 2009;45(2):268–74.
Amit O, Mannino F, Stone AM, Bushnell W, Denne J, Helterbrand J, Burger HU. Blinded independent central review of progression in cancer clinical trials: results from a meta-analysis. Eur J C. 2011;47(12):1772–8.
Floquet A, Vergote I, Colombo N, Fiane B, Monk BJ, Reinthaller A, et al. Progression-free survival by local investigator versus independent central review: comparative analysis of the AGO-OVAR16 trial. Gynecol Oncol. 2015;136(1):37–42.
Wu YL, Saijo N, Thongprasert S, Yang JH, Han B, Margono B, et al. Efficacy according to blind independent central review: post-hoc analyses from the phase III, randomized, multicenter, IPASS study of first-line GEFITINIB versus carboplatin/paclitaxel in Asian patients with EGFR mutation-positive Advanced NSCLC. Lung Cancer. 2017;104:119–25.
Miller CG, Krasnow J, Schwartz LH, editors. Medical imaging in clinical trials. London: Springer; 2014.
Ebel RL. Estimation of the reliability of ratings. J Psychometrika. 1951;16(4):407–24.
Korner IN, Westwood D. Inter-rater agreement in judging student adjustment from projective tests. J Clin Psychol. 1955;11(2):167–70.
Buckner DN. The predictability of ratings as a function of interrater agreement. J Appl Psychol. 1959;43(1):60.
Landis JR, Koch G. The measurement of observer agreement for categorical data. Biometrics 1977:159–74.
Kraemer HC, Periyakoil VS, Noda A. Kappa coefficients in medical research. Stat Med. 2002;21(14):2109–29.
Shrout PE, Fleiss JL. Intraclass correlations: uses in assessing rater reliability. Psychol Bull. 1979;86(2):420.
Cicchetti DV. Assessing inter-rater reliability for rating scales: resolving some basic issues. Br J Psychiatry. 1976;129(5):452–6.
Dobson KS, Shaw BF, Vallis TM. Reliability of a measure of the quality of cognitive therapy. Br J Clin Psychol. 1985;24(4):295–300.
Bendig A. Rater reliability and judgmental fatigue. J Appl Psychol. 1955;39(6):451.
Henkelman RM, Kay I, Bronskill MJ. Receiver operator characteristic (ROC) analysis without truth. Med Decis Mak. 1990;10(1):24–9.
Weller SC, Mann NC. Assessing rater performance without a" gold standard" using consensus theory. Med Decis Mak. 1997;17(1):71–9.
Armato SG, Roberts RY, McNitt-Gray MF, Meyer CR, Reeves AP, McLennan G, et al. The lung image database consortium (LIDC): ensuring the integrity of expert-defined “truth.” Acad Radiol. 2007;14(12):1455–63.
Eefting D, Schrage YM, Geirnaerdt MJ, Le Cessie S, Taminiau AH, Bovée JV, Hogendoorn PC. Assessment of interobserver variability and histologic parameters to improve reliability in classification and grading of central cartilaginous tumors. Am J Surg Pathol. 2009;33(1):50–7.
Smith AK, Stephenson AJ, Lane BR, Larson BT, Thomas AA, Gong MC, et al. Inadequacy of biopsy for diagnosis of upper tract urothelial carcinoma: implications for conservative management. Urology. 2011;78(1):82–6.
Patel SP, Kurzrock R. Pd-L1 expression as a predictive biomarker in cancer immunotherapy. Mol Cancer Ther. 2015;14(4):847–56.
Gniadek TJ, Li QK, Tully E, Chatterjee S, Nimmagadda S, Gabrielson E. Heterogeneous expression of Pd-L1 in pulmonary squamous cell carcinoma and adenocarcinoma: implications for assessment by small biopsy. Modern Pathol. 2017;30(4):530–8.
US Food and Drug Administration US FDA W, DC. Guidance for industry: developing medical imaging drug and biological products, part 3: design, analysis, and interpretation of clinical studies. 2004.
Cheson BD, Fisher RI, Barrington SF, Cavalli F, Schwartz LH, Zucca E, Lister TA. Recommendations for initial evaluation, staging, and response assessment of Hodgkin and non-Hodgkin lymphoma: the Lugano classification. J Clin Oncol. 2014;32(27):3059.
Obuchowski NA. How many observers are needed in clinical studies of medical imaging? Am J Roentgenol. 2004;182(4):867–69.
FDA. United States Food And Drug Administration guidance for industry: standards for clinical trials imaging endpoints. In: Services Udohah, editor. Rockville, MD 2018.
Prasad SR, Jhaveri KS, Saini S, Hahn PF, Halpern EF, Sumner JE. CT tumor measurement for therapeutic response assessment: comparison of unidimensional, bidimensional, and volumetric techniques—initial observations. Radiology. 2002;225(2):416–9.
Hayward RM, Patronas N, Baker EH, Vézina G, Albert PS, Warren KE. Inter-observer variability in the measurement of diffuse intrinsic pontine gliomas. J Neuro-Oncol. 2008;90(1):57–61.
McErlean A, Panicek DM, Zabor EC, Moskowitz CS, Bitar R, Motzer RJ, et al. Intra-and interobserver variability in CT measurements in oncology. Radiology. 2013;269(2):451–9.
Zhao B, Tan Y, Bell DJ, Marley SE, Guo P, Mann H, Scott ML, Schwartz LH, Ghiorghiu DC. Exploring intra-and inter-reader variability in uni-dimensional, bi-dimensional, and volumetric measurements of solid tumors on CT scans reconstructed at different slice intervals. Eur J Radiol. 2013;82(6):959–68.
Weiß Ch. EWMA monitoring of correlated processes of poisson counts. Qual Technol Quant Manag. 2009;6(2):137–53.
Barrett HH, Abbey CK, Gallas BD, Eckstein MP, editors. Stabilized estimates of hotelling-observer detection performance in patient-structured noise. Medical imaging 1998: Image Perception; 1998: International Society for Optics and Photonics.
Myers KJ, Barrett HH. Addition of a channel mechanism to the ideal-observer model. JOSA A. 1987;4(12):2447–57.
Agresti A. Categorical data analysis. New York: Wiley; 2003.
Uebersax JS, Grove WM. A latent trait finite mixture model for the analysis of rating agreement. Biometrics 1993:823–35.
Lorentzen HF, Gottrup F. Clinical assessment of infection in nonhealing ulcers analyzed by latent class analysis. Wound Repair Regener. 2006;14(3):350–3.
Petrick N, Sahiner B, Armato SG III, Bert A, Correale L, Delsanto S, Chan HP. Evaluation of computer-aided detection and diagnosis systems a. Med Phys. 2013;40(8):087001.
Patterson BF, Wind SA, Engelhard G Jr. Incorporating criterion ratings into model-based rater monitoring procedures using latent-class signal detection theory. Appl Psychol Measur. 2017;41(6):472–91.
Obuchowski N. How many observers are needed in clinical studies of medical imaging? Am J Roentgenol. 2004;182(4):867–9.
Jasani B, Bänfer G, Fish R, Waelput W, Sucaet Y, Barker C, et al. Evaluation of an online training tool for scoring programmed cell death ligand-1 (Pd-L1) diagnostic tests for lung cancer. Diagn Pathol. 2020;15:1–6.
Presant CA, Russell W, Alexander R, Fu Y. Soft-tissue and bone sarcoma histopathology peer review: the frequency of disagreement in diagnosis and the need for second pathology opinions. The Southeastern Cancer Study Group experience. J Clin Oncol. 1986;4(11):1658–61.
Pierro J, Kleiman R. The benefits of advanced imaging management systems. Appl Clin Trials. 2020;29(1/2):14–6.
Magnotta VA, Heckel D, Andreasen NC, Cizadlo T, Corson PW, Ehrhardt JC, et al. Measurement of brain structures with artificial neural networks: two-and three-dimensional applications. Radiology. 1999;211(3):781–90.
Durkee BY, Mudd SR, Roen CN, Clipson L, Newton MA, Weichert JP, et al. Reproducibility of tumor volume measurement at microct colonography in living mice. Acad Radiol. 2008;15(3):334–41.
Birkelo CC, Chamberlain WE, Phelps PS, Schools PE, Zacks D, Yerushalmy J. Tuberculosis case finding: a comparison of the effectiveness of various roentgenographic and photofluorographic methods. J Am Med Assoc. 1947;133(6):359–66.
Ford R, Oneal M, Moskowitz S, Fraunberger J. Adjudication rates between readers in blinded independent central review of oncology studies. J Clin Trials. 2016;6:289.
Fay MP, Shaw PA. Exact and asymptotic weighted logrank tests for interval censored data: the interval R package. J Stat Softw 2010;36(2)
Newcombe RG. Two-sided confidence intervals for the single proportion: comparison of seven methods. Stat Med. 1998;17(8):857–72.
Vollset SE. Confidence intervals for a binomial proportion. Stat Med. 1993;12(9):809–24.
Simpson Eh. The interpretation of interaction in contingency tables. J R Stat Soc. 1951;13(2):238–41.
Western E. Statistical quality control handbook. New York: Western Electric Co.; 1956.
Montgomery DC. Introduction to statistical quality control. New York: Wiley; 2020.
Nelson LS. Standardization of Shewhart control charts. J Qual Technol. 1989;21(4):287–9.
Zeng L, Zhao W, Wang C, Wang Z. Statistical properties of WECO rule combinations through simulations.
Cohen K, Gönen M, Ford R. Monitoring reader metrics in blinded independent central review of oncology studies. J Clin Trials. 2015;2915(5):4.
Bhapkar V. Notes on analysis of categorical data. North Carolina: Dept. Of Statistics, North Carolina State University; 1966.
Agresti A, Lang JB. Quasi-symmetric latent class models, with application to rater agreement. Biometrics 1993:131–139.
Reichmann WM, Maillefert JF, Hunter DJ, Katz JN, Conaghan PG, Losina E. Responsiveness to change and reliability of measurement of radiographic joint space width in osteoarthritis of the knee: a systematic review. Osteoarthritis Cartilage. 2011;19(5):550–6.
de Oliveira PG, da Câmara CP, Coelho PV. Intra-and interreader variability of orbital volume quantification using 3D computed tomography for reconstructed orbital fractures. J Cranio-Maxillofac Surg. 2019;47(7):1060–4.
Boone D, Halligan S, Mallett S, Taylor SA, Altman DG. Systematic review: bias in imaging studies-the effect of manipulating clinical context, recall bias and reporting intensity. Eur Radiol. 2012;22(3):495–550.
Metz CE. Some practical issues of experimental design and data analysis in radiological ROC studies. Invest Radiol. 1989;24(3):234–45.
Hardesty LA, Ganott MA, Hakim CM, Cohen CS, Clearfield RJ, Gur D. “Memory effect” in observer performance studies of mammograms1. Acad Radiol. 2005;12(3):286–90.
Ryan JT, Haygood TM, Yamal JM, Evanoff M, O’Sullivan P, McEntee M, Brennan PC. The “memory effect” for repeated radiologic observations. Am J Roentgenol. 2011;197(6):W985–91.
Montgomery DC. Design and analysis of experiments. New York: Wiley; 2017.
Abramson RG, McGhee CR, Lakomkin N, Arteaga CL. Pitfalls in RECIST data extraction for clinical trials: beyond the basics. Acad Radiol. 2015;22(6):779–86.
US Food and Drug Administration. FDA briefing document oncologic drugs advisory committee meeting-Ucm250378. 2011.
Ford R, Mozley PD. Report of task force II: best practices in the use of medical imaging techniques in clinical trials. Drug Inf J. 2008;42(5):515–23.
Keil S, Barabasch A, Dirrichs T, Bruners P, Hansen NL, Bieling HB, Kuhl CK. Target lesion selection: an important factor causing variability of response classification in the response evaluation criteria for solid tumors 1.1. Investig Radiol. 2014;49(8):509–17.
Kuhl CK, Alparslan Y, Sequeira B, Schmoe J, Engelke H, Keulers A, et al. Effect of target lesions selection on between-reader variability of response assessment according to recist 11. Am Soc Clin Oncol; 2017.
Sridhara R, Mandrekar SJ, Dodd LE. Missing data and measurement variability in assessing progression-free survival endpoint in randomized clinical trials. Clin Cancer Res. 2013;19(10):2613–20.
Bogaerts J, Ford R, Sargent D, Schwartz LH, Rubinstein L, Lacombe D, et al. Individual patient data analysis to assess modifications to the recist criteria. Eur J Cancer. 2009;45(2):248–60.
Cornelis FH, Martin M, Saut O, Buy X, Kind M, Palussiere J, et al. Precision of manual two-dimensional segmentations of lung and liver metastases and its impact on tumour response assessment using recist 1.1. Eur Radiol Exper. 2017;1(1):16.
The authors would like to express their sincere appreciation to Drs. Anthony Fotenos and Alex Hofling (FDA/CDER) and Dr. Joseph Pierro (eResearch Technology) for their valuable insight, participation, and contributions to the discussions and development of the methodologies within this manuscript. The authors would further like to acknowledge Liz Kuney's contribution for her valuable insight in developing and editing this manuscript and to the reviewers for their valuable and insightful comments.
No funding sources.
Conflict of interest
The authors declare that there are no conflicts of interest.
Below is the link to the electronic supplementary material.
About this article
Cite this article
Raunig, D.L., Schmid, A.M., Miller, C.G. et al. Radiologists and Clinical Trials: Part 2: Practical Statistical Methods for Understanding and Monitoring Independent Reader Performance. Ther Innov Regul Sci 55, 1122–1138 (2021). https://doi.org/10.1007/s43441-021-00317-5