Benchmarking Quantitative Imaging Biomarker Measurement Methods Without a Gold Standard
Validation of quantitative imaging biomarker (QIB) measurement methods is generally based on the concept of a reference method, also called a gold standard (GS). Poor quality of the GS, for example due to inter- and intra-rater variabilities in segmentation, may lead to biased error estimates and thus adversely impact the validation. Herein we propose a novel framework for benchmarking multiple measurement methods without a GS. The framework consists of (i) an error model accounting for correlated random error between measurements extracted by the methods, (ii) a novel objective based on a joint posterior probability of the error model parameters (iii) Markov chain Monte Carlo to sample the posterior. Analysis of the posterior enables not only to estimate the error model parameters (systematic and random error) and thereby benchmark the methods, but also to estimate the unknown true values of QIB. Validation of the proposed framework on multiple sclerosis total lesion load measurements by four automated segmentation methods applied to a clinical brain MRI dataset showed a very good agreement of the error model and true value estimates with corresponding least squares estimates based on a known GS.
KeywordsBayesian inference Markov Chain Monte Carlo Validation Brain lesion segmentation Clinical dataset
This work supported by Slovenian Research Agency under grants J2-5473 and P2-0232.
- 1.Grand Challenges in Biomedical Image Analysis (2017). https://grand-challenge.org/All_Challenges/. 24 Feb 2017
- 2.Barnard, J., McCulloch, R., Meng, X.L.: Modeling covariance matrices in terms of standard deviations and correlations, with application to shrinkage. Statistica Sinica 10, 1281–1311 (2000). http://www.jstor.org/stable/24306780?seq=1#page_scan_tab_contents
- 4.Galimzianova, A., Lesjak, Z., Likar, B., Pernus, F., Spiclin, Z.: Locally adaptive MR intensity models and MRF-based segmentation of multiple sclerosis lesions. In: Proceedings of SPIE International Society Optics Engineering, vol. 9413, p. 94133G, 20 March 2015Google Scholar
- 7.Jerman, T., Galimzianova, A., Pernuš, F., Likar, B., Špiclin, Ž.: Combining unsupervised and supervised methods for lesion segmentation. In: Crimi, A., Menze, B., Maier, O., Reyes, M., Handels, H. (eds.) BrainLes 2015. LNCS, vol. 9556, pp. 45–56. Springer, Cham (2016). doi: 10.1007/978-3-319-30858-6_5CrossRefGoogle Scholar