Benchmarking Quantitative Imaging Biomarker Measurement Methods Without a Gold Standard

  • Hennadii MadanEmail author
  • Franjo Pernuš
  • Žiga Špiclin
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10434)


Validation of quantitative imaging biomarker (QIB) measurement methods is generally based on the concept of a reference method, also called a gold standard (GS). Poor quality of the GS, for example due to inter- and intra-rater variabilities in segmentation, may lead to biased error estimates and thus adversely impact the validation. Herein we propose a novel framework for benchmarking multiple measurement methods without a GS. The framework consists of (i) an error model accounting for correlated random error between measurements extracted by the methods, (ii) a novel objective based on a joint posterior probability of the error model parameters (iii) Markov chain Monte Carlo to sample the posterior. Analysis of the posterior enables not only to estimate the error model parameters (systematic and random error) and thereby benchmark the methods, but also to estimate the unknown true values of QIB. Validation of the proposed framework on multiple sclerosis total lesion load measurements by four automated segmentation methods applied to a clinical brain MRI dataset showed a very good agreement of the error model and true value estimates with corresponding least squares estimates based on a known GS.


Bayesian inference Markov Chain Monte Carlo Validation Brain lesion segmentation Clinical dataset 



This work supported by Slovenian Research Agency under grants J2-5473 and P2-0232.


  1. 1.
    Grand Challenges in Biomedical Image Analysis (2017). 24 Feb 2017
  2. 2.
    Barnard, J., McCulloch, R., Meng, X.L.: Modeling covariance matrices in terms of standard deviations and correlations, with application to shrinkage. Statistica Sinica 10, 1281–1311 (2000).
  3. 3.
    Foreman-Mackey, D., Hogg, D.W., Lang, D., Goodman, J.: emcee: the MCMC hammer. Publ. Astron. Soc. Pac. 125(925), 306 (2013)CrossRefGoogle Scholar
  4. 4.
    Galimzianova, A., Lesjak, Z., Likar, B., Pernus, F., Spiclin, Z.: Locally adaptive MR intensity models and MRF-based segmentation of multiple sclerosis lesions. In: Proceedings of SPIE International Society Optics Engineering, vol. 9413, p. 94133G, 20 March 2015Google Scholar
  5. 5.
    Galimzianova, A., Pernus, F., Likar, B., Spiclin, Z.: Stratified mixture modeling for segmentation of white-matter lesions in brain MR images. NeuroImage 124(Pt A), 1031–1043 (2016)CrossRefGoogle Scholar
  6. 6.
    Jain, S., Sima, D.M., Ribbens, A., et al.: Automatic segmentation and volumetry of multiple sclerosis brain lesions from MR images. NeuroImage: Clin. 8, 367–375 (2015)CrossRefGoogle Scholar
  7. 7.
    Jerman, T., Galimzianova, A., Pernuš, F., Likar, B., Špiclin, Ž.: Combining unsupervised and supervised methods for lesion segmentation. In: Crimi, A., Menze, B., Maier, O., Reyes, M., Handels, H. (eds.) BrainLes 2015. LNCS, vol. 9556, pp. 45–56. Springer, Cham (2016). doi: 10.1007/978-3-319-30858-6_5CrossRefGoogle Scholar
  8. 8.
    Kupinski, M.A., Hoppin, J.W., Clarkson, E., Barrett, H.H., Kastis, G.A.: Estimation in medical imaging without a gold standard. Acad. Radiol. 9(3), 290–297 (2002)CrossRefGoogle Scholar
  9. 9.
    Obuchowski, N.A., Reeves, A.P., Huang, E.A.: Quantitative imaging biomarkers: a review of statistical methods for computer algorithm comparisons. Stat. Methods Med. Res. 24(1), 68–106 (2015)MathSciNetCrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Hennadii Madan
    • 1
    Email author
  • Franjo Pernuš
    • 1
  • Žiga Špiclin
    • 1
  1. 1.Faculty of Electrical Engineering, Laboratory of Imaging TechnologiesUniversity of LjubljanaLjubljanaSlovenia

Personalised recommendations