Skip to main content
SpringerLink
Log in

Radiologists and Clinical Trials: Part 2: Practical Statistical Methods for Understanding and Monitoring Independent Reader Performance

Therapeutic Innovation & Regulatory Science volume 55, pages 1122–1138 (2021)Cite this article

Abstract

Though many clinical trials rely on medical image evaluations for primary or key secondary endpoints, the methods to monitor reader performance are all too often mired in the legacy use of adjudication rates. If misused, this simple metric can be misleading and sometimes entirely contradictory. Furthermore, attempts to overcome the limitations of adjudication rates using de novo or ad hoc methods often ignore well-established research conducted over the last half-century and can lead to inaccurate conclusions or variable interpretations. Underperforming readers can be missed, expert readers retrained, or worse, replaced. This paper aims to standardize reader performance evaluations using proven statistical methods. Additionally, these methods will describe how to discriminate between scenarios of concern and normal medical interpretation variability. Statistical methods are provided for inter-reader and intra-reader variability and bias, including the adjudicator's bias. Finally, we have compiled guidelines for calculating correct sample sizes, considerations for intra-reader memory recall, and applying alternative designs for independent readers.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Figure 1
Figure 2
Figure 3
Figure 4
Figure 5
Figure 6
Figure 7
Figure 8

Notes

  1. Clinicaltrials.gov search terms: Oncology, 2011–2020.

  2. Clinicaltrials.gov search terms: Recruiting, Active, not recruiting, Completed, Suspended, Terminated, Withdrawn Studies | Interventional Studies | Oncology | imaging OR recist OR Lugano OR cheson OR MRI OR ct OR ultrasound OR pet OR rano OR Choi OR PFS OR ORR OR BOR | Phase Early Phase I, PhaseII, Phase III, Phase II/III.

  3. See ( See Fig. 1 and Online Resource 1 “A Description of Reader Symmetry).

  4. Note that the confidence limits provided in Ford 2016 need to be adjusted by the number of studies to establish an estimate of coverage.

Abbreviations

AR:

Adjudication rate (the rate of disagreement between two readers evaluating the same patient)

ASR:

Adjudicator selection rate (the rate that an adjudicator selects a reader from a paired team of two readers when those two readers disagree)

BICR:

Blinded independent central review (the readers used by an Imaging Core Lab to assess the images scanned at an investigator site)

CI:

Confidence interval (to show the range of confidence in a parameter estimate)

CR:

Complete response (classification of response)

DoP:

Date of progression (used for event-related endpoints like progression-free survival)

FLAIR:

Fluid-attenuated inversion recovery (an MRI sequence used to suppress fluids)

GBM:

Glioblastoma multiforme (a to date uncurable brain cancer)

ICL:

Imaging Core Lab (a central facility that manages the BICR)

IRV:

Intra-reader variability (the disagreement of readers with a previous assessment made on the same set of images)

PD:

Progressive disease (classification of response)

PR:

Partial response (classification of response)

RECIST:

Response evaluation criteria in solid tumors (an objective tumor size assessment criteria to categorize a patient’s response to treatment)

SD:

Stable disease (classification of response)

References

  1. Clinton B, Gore A. Reinventing the regulation of cancer drugs: accelerating approval and expanding access. Natl Perform Rev 1996.

  2. Miller C, Noever K. Taking care of your subject’s image: the role of medical imaging core laboratories. Good Clin Pract J. 2003;10(9):21–4.

    Google Scholar 

  3. Conklin J. Interview of James Conklin on the emergence of imaging CROs. In: Raunig D, editor. 2019.

  4. Schmid A, Raunig D, Ford R, Miller C. Radiologists and clinical trials: part 1. The truth about reader disagreements therapeutic innovation & regulatory science. 2021

  5. Sharma M, O'connor JM, Singareddy A, editors. Reader disagreement index: a better measure of overall review quality monitoring in an oncology trial compared to adjudication rate. Medical imaging 2019: image perception, observer performance, and technology assessment; 2019: International Society for Optics and Photonics.

  6. Pintad. Pharmaceutical imaging network for therapeutics and diagnostics 2020. https://Www.Pintad.Net/.

  7. Eldevik OP, Dugstad G, Orrison WW, Haughton VM. The effect of clinical bias on the interpretation of myelography and spinal computed tomography. Radiology. 1982;145(1):85–9.

    Article  CAS  PubMed  Google Scholar 

  8. Sica GT. Bias in research studies. Radiology. 2006;238(3):780–9.

    Article  PubMed  Google Scholar 

  9. Ford R, Schwartz L, Dancey J, Dodd LE, Eisenhauer EA, Gwyther S, et al. Lessons learned from independent central review. Eur J Cancer. 2009;45(2):268–74.

    Article  CAS  PubMed  Google Scholar 

  10. Amit O, Mannino F, Stone AM, Bushnell W, Denne J, Helterbrand J, Burger HU. Blinded independent central review of progression in cancer clinical trials: results from a meta-analysis. Eur J C. 2011;47(12):1772–8.

    Article  CAS  Google Scholar 

  11. Floquet A, Vergote I, Colombo N, Fiane B, Monk BJ, Reinthaller A, et al. Progression-free survival by local investigator versus independent central review: comparative analysis of the AGO-OVAR16 trial. Gynecol Oncol. 2015;136(1):37–42.

    Article  PubMed  Google Scholar 

  12. Wu YL, Saijo N, Thongprasert S, Yang JH, Han B, Margono B, et al. Efficacy according to blind independent central review: post-hoc analyses from the phase III, randomized, multicenter, IPASS study of first-line GEFITINIB versus carboplatin/paclitaxel in Asian patients with EGFR mutation-positive Advanced NSCLC. Lung Cancer. 2017;104:119–25.

    Article  PubMed  Google Scholar 

  13. Miller CG, Krasnow J, Schwartz LH, editors. Medical imaging in clinical trials. London: Springer; 2014.

    Google Scholar 

  14. Ebel RL. Estimation of the reliability of ratings. J Psychometrika. 1951;16(4):407–24.

    Article  Google Scholar 

  15. Korner IN, Westwood D. Inter-rater agreement in judging student adjustment from projective tests. J Clin Psychol. 1955;11(2):167–70.

    Article  CAS  PubMed  Google Scholar 

  16. Buckner DN. The predictability of ratings as a function of interrater agreement. J Appl Psychol. 1959;43(1):60.

    Article  Google Scholar 

  17. Landis JR, Koch G. The measurement of observer agreement for categorical data. Biometrics 1977:159–74.

  18. Kraemer HC, Periyakoil VS, Noda A. Kappa coefficients in medical research. Stat Med. 2002;21(14):2109–29.

    Article  Google Scholar 

  19. Shrout PE, Fleiss JL. Intraclass correlations: uses in assessing rater reliability. Psychol Bull. 1979;86(2):420.

    Article  CAS  PubMed  Google Scholar 

  20. Cicchetti DV. Assessing inter-rater reliability for rating scales: resolving some basic issues. Br J Psychiatry. 1976;129(5):452–6.

    Article  CAS  PubMed  Google Scholar 

  21. Dobson KS, Shaw BF, Vallis TM. Reliability of a measure of the quality of cognitive therapy. Br J Clin Psychol. 1985;24(4):295–300.

    Article  PubMed  Google Scholar 

  22. Bendig A. Rater reliability and judgmental fatigue. J Appl Psychol. 1955;39(6):451.

    Article  Google Scholar 

  23. Henkelman RM, Kay I, Bronskill MJ. Receiver operator characteristic (ROC) analysis without truth. Med Decis Mak. 1990;10(1):24–9.

    Article  CAS  Google Scholar 

  24. Weller SC, Mann NC. Assessing rater performance without a" gold standard" using consensus theory. Med Decis Mak. 1997;17(1):71–9.

    Article  CAS  Google Scholar 

  25. Armato SG, Roberts RY, McNitt-Gray MF, Meyer CR, Reeves AP, McLennan G, et al. The lung image database consortium (LIDC): ensuring the integrity of expert-defined “truth.” Acad Radiol. 2007;14(12):1455–63.

    Article  PubMed  PubMed Central  Google Scholar 

  26. Eefting D, Schrage YM, Geirnaerdt MJ, Le Cessie S, Taminiau AH, Bovée JV, Hogendoorn PC. Assessment of interobserver variability and histologic parameters to improve reliability in classification and grading of central cartilaginous tumors. Am J Surg Pathol. 2009;33(1):50–7.

    Article  PubMed  Google Scholar 

  27. Smith AK, Stephenson AJ, Lane BR, Larson BT, Thomas AA, Gong MC, et al. Inadequacy of biopsy for diagnosis of upper tract urothelial carcinoma: implications for conservative management. Urology. 2011;78(1):82–6.

    Article  PubMed  Google Scholar 

  28. Patel SP, Kurzrock R. Pd-L1 expression as a predictive biomarker in cancer immunotherapy. Mol Cancer Ther. 2015;14(4):847–56.

    Article  CAS  PubMed  Google Scholar 

  29. Gniadek TJ, Li QK, Tully E, Chatterjee S, Nimmagadda S, Gabrielson E. Heterogeneous expression of Pd-L1 in pulmonary squamous cell carcinoma and adenocarcinoma: implications for assessment by small biopsy. Modern Pathol. 2017;30(4):530–8.

    Article  CAS  Google Scholar 

  30. US Food and Drug Administration US FDA W, DC. Guidance for industry: developing medical imaging drug and biological products, part 3: design, analysis, and interpretation of clinical studies. 2004.

  31. Cheson BD, Fisher RI, Barrington SF, Cavalli F, Schwartz LH, Zucca E, Lister TA. Recommendations for initial evaluation, staging, and response assessment of Hodgkin and non-Hodgkin lymphoma: the Lugano classification. J Clin Oncol. 2014;32(27):3059.

    Article  PubMed  PubMed Central  Google Scholar 

  32. Obuchowski NA. How many observers are needed in clinical studies of medical imaging? Am J Roentgenol. 2004;182(4):867–69.

    Article  Google Scholar 

  33. FDA. United States Food And Drug Administration guidance for industry: standards for clinical trials imaging endpoints. In: Services Udohah, editor. Rockville, MD 2018.

  34. Prasad SR, Jhaveri KS, Saini S, Hahn PF, Halpern EF, Sumner JE. CT tumor measurement for therapeutic response assessment: comparison of unidimensional, bidimensional, and volumetric techniques—initial observations. Radiology. 2002;225(2):416–9.

    Article  PubMed  Google Scholar 

  35. Hayward RM, Patronas N, Baker EH, Vézina G, Albert PS, Warren KE. Inter-observer variability in the measurement of diffuse intrinsic pontine gliomas. J Neuro-Oncol. 2008;90(1):57–61.

    Article  Google Scholar 

  36. McErlean A, Panicek DM, Zabor EC, Moskowitz CS, Bitar R, Motzer RJ, et al. Intra-and interobserver variability in CT measurements in oncology. Radiology. 2013;269(2):451–9.

    Article  PubMed  Google Scholar 

  37. Zhao B, Tan Y, Bell DJ, Marley SE, Guo P, Mann H, Scott ML, Schwartz LH, Ghiorghiu DC. Exploring intra-and inter-reader variability in uni-dimensional, bi-dimensional, and volumetric measurements of solid tumors on CT scans reconstructed at different slice intervals. Eur J Radiol. 2013;82(6):959–68.

    Article  PubMed  Google Scholar 

  38. Weiß Ch. EWMA monitoring of correlated processes of poisson counts. Qual Technol Quant Manag. 2009;6(2):137–53.

    Article  Google Scholar 

  39. Barrett HH, Abbey CK, Gallas BD, Eckstein MP, editors. Stabilized estimates of hotelling-observer detection performance in patient-structured noise. Medical imaging 1998: Image Perception; 1998: International Society for Optics and Photonics.

  40. Myers KJ, Barrett HH. Addition of a channel mechanism to the ideal-observer model. JOSA A. 1987;4(12):2447–57.

    Article  CAS  Google Scholar 

  41. Agresti A. Categorical data analysis. New York: Wiley; 2003.

    Google Scholar 

  42. Uebersax JS, Grove WM. A latent trait finite mixture model for the analysis of rating agreement. Biometrics 1993:823–35.

  43. Lorentzen HF, Gottrup F. Clinical assessment of infection in nonhealing ulcers analyzed by latent class analysis. Wound Repair Regener. 2006;14(3):350–3.

    Article  Google Scholar 

  44. Petrick N, Sahiner B, Armato SG III, Bert A, Correale L, Delsanto S, Chan HP. Evaluation of computer-aided detection and diagnosis systems a. Med Phys. 2013;40(8):087001.

    Article  PubMed  PubMed Central  Google Scholar 

  45. Patterson BF, Wind SA, Engelhard G Jr. Incorporating criterion ratings into model-based rater monitoring procedures using latent-class signal detection theory. Appl Psychol Measur. 2017;41(6):472–91.

    Article  Google Scholar 

  46. Obuchowski N. How many observers are needed in clinical studies of medical imaging? Am J Roentgenol. 2004;182(4):867–9.

    Article  Google Scholar 

  47. Jasani B, Bänfer G, Fish R, Waelput W, Sucaet Y, Barker C, et al. Evaluation of an online training tool for scoring programmed cell death ligand-1 (Pd-L1) diagnostic tests for lung cancer. Diagn Pathol. 2020;15:1–6.

    Article  CAS  Google Scholar 

  48. Presant CA, Russell W, Alexander R, Fu Y. Soft-tissue and bone sarcoma histopathology peer review: the frequency of disagreement in diagnosis and the need for second pathology opinions. The Southeastern Cancer Study Group experience. J Clin Oncol. 1986;4(11):1658–61.

    Article  CAS  PubMed  Google Scholar 

  49. Pierro J, Kleiman R. The benefits of advanced imaging management systems. Appl Clin Trials. 2020;29(1/2):14–6.

    Google Scholar 

  50. Magnotta VA, Heckel D, Andreasen NC, Cizadlo T, Corson PW, Ehrhardt JC, et al. Measurement of brain structures with artificial neural networks: two-and three-dimensional applications. Radiology. 1999;211(3):781–90.

    Article  CAS  PubMed  Google Scholar 

  51. Durkee BY, Mudd SR, Roen CN, Clipson L, Newton MA, Weichert JP, et al. Reproducibility of tumor volume measurement at microct colonography in living mice. Acad Radiol. 2008;15(3):334–41.

    Article  PubMed  PubMed Central  Google Scholar 

  52. Birkelo CC, Chamberlain WE, Phelps PS, Schools PE, Zacks D, Yerushalmy J. Tuberculosis case finding: a comparison of the effectiveness of various roentgenographic and photofluorographic methods. J Am Med Assoc. 1947;133(6):359–66.

    Article  CAS  PubMed  Google Scholar 

  53. Ford R, Oneal M, Moskowitz S, Fraunberger J. Adjudication rates between readers in blinded independent central review of oncology studies. J Clin Trials. 2016;6:289.

    Google Scholar 

  54. Fay MP, Shaw PA. Exact and asymptotic weighted logrank tests for interval censored data: the interval R package. J Stat Softw 2010;36(2)

  55. Newcombe RG. Two-sided confidence intervals for the single proportion: comparison of seven methods. Stat Med. 1998;17(8):857–72.

    Article  CAS  PubMed  Google Scholar 

  56. Vollset SE. Confidence intervals for a binomial proportion. Stat Med. 1993;12(9):809–24.

    Article  CAS  PubMed  Google Scholar 

  57. Simpson Eh. The interpretation of interaction in contingency tables. J R Stat Soc. 1951;13(2):238–41.

    Google Scholar 

  58. Western E. Statistical quality control handbook. New York: Western Electric Co.; 1956.

    Google Scholar 

  59. Montgomery DC. Introduction to statistical quality control. New York: Wiley; 2020.

    Google Scholar 

  60. Nelson LS. Standardization of Shewhart control charts. J Qual Technol. 1989;21(4):287–9.

    Article  Google Scholar 

  61. Zeng L, Zhao W, Wang C, Wang Z. Statistical properties of WECO rule combinations through simulations.

  62. Cohen K, Gönen M, Ford R. Monitoring reader metrics in blinded independent central review of oncology studies. J Clin Trials. 2015;2915(5):4.

    Google Scholar 

  63. Bhapkar V. Notes on analysis of categorical data. North Carolina: Dept. Of Statistics, North Carolina State University; 1966.

    Google Scholar 

  64. Agresti A, Lang JB. Quasi-symmetric latent class models, with application to rater agreement. Biometrics 1993:131–139.

  65. Reichmann WM, Maillefert JF, Hunter DJ, Katz JN, Conaghan PG, Losina E. Responsiveness to change and reliability of measurement of radiographic joint space width in osteoarthritis of the knee: a systematic review. Osteoarthritis Cartilage. 2011;19(5):550–6.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  66. de Oliveira PG, da Câmara CP, Coelho PV. Intra-and interreader variability of orbital volume quantification using 3D computed tomography for reconstructed orbital fractures. J Cranio-Maxillofac Surg. 2019;47(7):1060–4.

    Article  Google Scholar 

  67. Boone D, Halligan S, Mallett S, Taylor SA, Altman DG. Systematic review: bias in imaging studies-the effect of manipulating clinical context, recall bias and reporting intensity. Eur Radiol. 2012;22(3):495–550.

    Article  PubMed  Google Scholar 

  68. Metz CE. Some practical issues of experimental design and data analysis in radiological ROC studies. Invest Radiol. 1989;24(3):234–45.

    Article  CAS  PubMed  Google Scholar 

  69. Hardesty LA, Ganott MA, Hakim CM, Cohen CS, Clearfield RJ, Gur D. “Memory effect” in observer performance studies of mammograms1. Acad Radiol. 2005;12(3):286–90.

    Article  PubMed  Google Scholar 

  70. Ryan JT, Haygood TM, Yamal JM, Evanoff M, O’Sullivan P, McEntee M, Brennan PC. The “memory effect” for repeated radiologic observations. Am J Roentgenol. 2011;197(6):W985–91.

    Article  Google Scholar 

  71. Montgomery DC. Design and analysis of experiments. New York: Wiley; 2017.

    Google Scholar 

  72. Abramson RG, McGhee CR, Lakomkin N, Arteaga CL. Pitfalls in RECIST data extraction for clinical trials: beyond the basics. Acad Radiol. 2015;22(6):779–86.

    Article  PubMed  PubMed Central  Google Scholar 

  73. US Food and Drug Administration. FDA briefing document oncologic drugs advisory committee meeting-Ucm250378. 2011.

  74. Ford R, Mozley PD. Report of task force II: best practices in the use of medical imaging techniques in clinical trials. Drug Inf J. 2008;42(5):515–23.

    Article  Google Scholar 

  75. Keil S, Barabasch A, Dirrichs T, Bruners P, Hansen NL, Bieling HB, Kuhl CK. Target lesion selection: an important factor causing variability of response classification in the response evaluation criteria for solid tumors 1.1. Investig Radiol. 2014;49(8):509–17.

    Article  CAS  Google Scholar 

  76. Kuhl CK, Alparslan Y, Sequeira B, Schmoe J, Engelke H, Keulers A, et al. Effect of target lesions selection on between-reader variability of response assessment according to recist 11. Am Soc Clin Oncol; 2017.

  77. Sridhara R, Mandrekar SJ, Dodd LE. Missing data and measurement variability in assessing progression-free survival endpoint in randomized clinical trials. Clin Cancer Res. 2013;19(10):2613–20.

    Article  PubMed  Google Scholar 

  78. Bogaerts J, Ford R, Sargent D, Schwartz LH, Rubinstein L, Lacombe D, et al. Individual patient data analysis to assess modifications to the recist criteria. Eur J Cancer. 2009;45(2):248–60.

    Article  PubMed  Google Scholar 

  79. Cornelis FH, Martin M, Saut O, Buy X, Kind M, Palussiere J, et al. Precision of manual two-dimensional segmentations of lung and liver metastases and its impact on tumour response assessment using recist 1.1. Eur Radiol Exper. 2017;1(1):16.

    Article  CAS  Google Scholar 

Download references

Acknowledgements

The authors would like to express their sincere appreciation to Drs. Anthony Fotenos and Alex Hofling (FDA/CDER) and Dr. Joseph Pierro (eResearch Technology) for their valuable insight, participation, and contributions to the discussions and development of the methodologies within this manuscript. The authors would further like to acknowledge Liz Kuney's contribution for her valuable insight in developing and editing this manuscript and to the reviewers for their valuable and insightful comments.

Funding

No funding sources.

Author information

Author notes

  1. David L. Raunig and Annette M. Schmid contributed equally to the development and writing of the manuscript.

Authors and Affiliations

  1. Takeda, 300 Massachusetts Ave, Cambridge, MA, 02139, USA

    David L. Raunig Ph.D. & Annette M. Schmid Ph.D.

  2. The Bracken Group, Newtown, PA, USA

    Colin G. Miller Ph.D.

  3. WorldCare Clinical, Boston, MA, USA

    Richard C. Walovitch Ph.D.

  4. Parexel, Billerica, MA, USA

    Michael O’Connor Ph.D.

  5. Bioclinica, Basel, Switzerland

    Klaus Noever M.D.

  6. Independent Consultant, Vienna, Austria

    Ivalina Hristova M.S.

  7. Bioclinica, Princeton, NJ, USA

    Michael O’Neal M.D.

  8. Bayer Schering Plough AG, Berlin, Germany

    Guenther Brueggenwerth M.D.

  9. Bioclinica, Belle Mead, NJ, USA

    Robert R. Ford M.D.

Authors
  1. David L. Raunig Ph.D.
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Annette M. Schmid Ph.D.
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Colin G. Miller Ph.D.
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Richard C. Walovitch Ph.D.
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Michael O’Connor Ph.D.
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Klaus Noever M.D.
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Ivalina Hristova M.S.
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Michael O’Neal M.D.
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Guenther Brueggenwerth M.D.
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Robert R. Ford M.D.
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

DLR and AMS both equally contributed to concept, design, drafting, finalizing and approved all content. CGM contributed to the concept, design, drafting, finalizing, and approval of all content. RCW contributed to the concept, drafting and finalizing all content. MO’C contributed to the concept, drafting, and finalizing all content. KN contributed to the concept, drafting, and finalizing all content. IH contributed to the concept, drafting, and finalizing all content. MO’N contributed to the concept, drafting, and finalizing all content. GB contributed to the concept, drafting, and finalizing all content. RRF contributed to the concept, design, drafting, finalizing, and approval of all content. All authors agree to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Corresponding author

Correspondence to David L. Raunig Ph.D..

Ethics declarations

Conflict of interest

The authors declare that there are no conflicts of interest.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (DOCX 159 kb)

Rights and permissions

Reprints and Permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Raunig, D.L., Schmid, A.M., Miller, C.G. et al. Radiologists and Clinical Trials: Part 2: Practical Statistical Methods for Understanding and Monitoring Independent Reader Performance. Ther Innov Regul Sci 55, 1122–1138 (2021). https://doi.org/10.1007/s43441-021-00317-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s43441-021-00317-5

Keywords

search

Navigation