Abstract
This work proposes a novel approach to assessing confidence measures for software classification systems in demanding applications such as those in the safety critical domain. Our focus is the Bayesian framework for developing a model-averaged probabilistic classifier implemented using Markov chain Monte Carlo (MCMC) and where appropriate its reversible jump variant (RJ-MCMC). Within this context we suggest a new technique, building on the reject region idea, to identify areas in feature space that are associated with “unsure” classification predictions. We term such areas “uncertainty envelopes” and they are defined in terms of the full characteristics of the posterior predictive density in different regions of the feature space. We argue this is more informative than use of a traditional reject region which considers only point estimates of predictive probabilities. Results from the method we propose are illustrated on synthetic data and also usefully applied to real life safety critical systems involving medical trauma data.
Similar content being viewed by others
References
Bishop CM (1998) Neural networks for pattern recognition. Oxford University Press, New York
Bouissou M, Martin F, Ourghanlian A (1999) Assessment of safety critical system including software: a Bayesian belief network for evidence sources. In: Proceedings of the reliability and maintainability symposium, Washington DC, January 1999
Denison DGT, Holmes CC, Mallick BK, Smith AFM (2002) Bayesian methods for nonlinear classification and regression. Wiley, London
Fukunaga K (1990) Introduction to statistical pattern recognition. Morgan Kaufmann
Green PJ (1995) Reversible jump Markov chain Monte Carlo computation and Bayesian model determination. Biometrika 82:711–732
Little B, Stringini L (1993) Validation of ultra-high dependability for software-based systems. Commun ACM 36(11):69–80
MacKay DJC (1992) Bayesian interpolation. Neural Comput 4(3):415–447
MacKay DJC (1998) Introduction to Monte Carlo methods. In: Jordan MI (ed) Learning in graphical models, NATO Science Series. Kluwer, Dordrecht, pp 175–204
Nabney IT (2002) NETLAB: algorithms for pattern recognition. Springer, Berlin Heidelberg New York
Ripley BD (1994) Neural networks and related methods for classification. J R Stat Soc B 56(3):409–456
Acknowledgements
This work was supported by grant GR/R24357/01 of the UK Engineering and Physical Sciences Research Council.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Bailey, T.C., Everson, R.M., Fieldsend, J.E. et al. Representing classifier confidence in the safety critical domain: an illustration from mortality prediction in trauma cases. Neural Comput & Applic 16, 1–10 (2007). https://doi.org/10.1007/s00521-006-0053-y
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00521-006-0053-y