Skip to main content

Effective Confidence Region Prediction Using Probability Forecasters

  • Conference paper
Artificial Intelligence in Medicine (AIME 2005)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 3581))

Included in the following conference series:

  • 1234 Accesses

Abstract

Confidence region prediction is a practically useful extension to the commonly studied pattern recognition problem. Instead of predicting a single label, the constraint is relaxed to allow prediction of a subset of labels given a desired confidence level 1 – δ. Ideally, effective region predictions should be (1) well calibrated – predictive regions at confidence level 1 – δ should err with relative frequency at most δ and (2) be as narrow (or certain) as possible. We present a simple technique to generate confidence region predictions from conditional probability estimates (probability forecasts). We use this ‘conversion’ technique to generate confidence region predictions from probability forecasts output by standard machine learning algorithms when tested on 15 multi-class datasets. Our results show that approximately 44% of experiments demonstrate well-calibrated confidence region predictions, with the K-Nearest Neighbour algorithm tending to perform consistently well across all data. Our results illustrate the practical benefits of effective confidence region prediction with respect to medical diagnostics, where guarantees of capturing the true disease label can be given.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Similar content being viewed by others

References

  1. Vovk, V.: On-line Confidence Machines are well-calibrated. In: Proc. of the Forty Third Annual Symposium on Foundations of Computer Science. IEEE Computer Society, Los Alamitos (2002)

    Google Scholar 

  2. Zadrozny, B., Elkan, C.: Transforming Classifier Scores into Accurate Multiclass Probability Estimates. In: Proc. of the 8th ACM SIGKDD, pp. 694–699. ACM Press, New York (2002)

    Google Scholar 

  3. Lindsay, D., Cox, S.: Improving the Reliability of Decision Tree and Naive Bayes Learners. In: Proc. of the 4th ICDM, pp. 459–462. IEEE, Los Alamitos (2004)

    Google Scholar 

  4. Berner, E.S., Wesbter, G.D., Shugerman, A.A., Jackson, J.R., Algina, J., Baker, A.L., Ball, E.V., Cobbs, C.G., Dennis, V.W., Frenkel, E.P., Hudson, L.D., Mancall, E.L., Rackley, C.E., Taunton, O.D.: Performance of four computer-based diagnostic systems. The New England Journal of Medicine 330, 1792–1796 (1994)

    Article  Google Scholar 

  5. Gammerman, A., Thatcher, A.R.: Bayesian diagnostic probabilities without assuming independence of symptoms. Yearbook of Medical Informatics, 323–330 (1992)

    Google Scholar 

  6. Berger, J.O., Delampady, M.: Testing precise hypotheses. Statistical Science 2, 317–335 (1987); David R. Cox’s comment, pp. 335–336

    Google Scholar 

  7. Dawid, A.P.: Calibration-based empirical probability (with discussion). Annals of Statistics 13, 1251–1285 (1985)

    Article  MATH  MathSciNet  Google Scholar 

  8. Murphy, A.H.: A New Vector Partition of the Probability Score. Journal of Applied Meteorology 12, 595–600 (1973)

    Article  Google Scholar 

  9. Witten, I., Frank, E.: Data Mining - Practical Machine Learning Tools and Techniques with Java Implementations. Morgan Kaufmann, San Francisco (2000)

    Google Scholar 

  10. Devroye, L., Györfi, L., Lugosi, G.: A Probabilistic Theory of Pattern Recognition. Springer, New York (1996)

    MATH  Google Scholar 

  11. Blake, C., Merz, C.: UCI repository of machine learning databases (1998)

    Google Scholar 

  12. Breiman, L.: Bagging predictors. Machine Learning 24, 123–140 (1996)

    MATH  MathSciNet  Google Scholar 

  13. Freund, Y., Schapire, R.E.: Game theory, on-line prediction and boosting. In: Proc. 9th Ann. Conf. on Computational Learning Theory, Assoc. of Comp. Machinery, pp. 325–332 (1996)

    Google Scholar 

  14. Boutell, M.R., Luo, J., Shen, X., Brown, C.M.: Learning multi-label scene classification. Pattern Recognition 37, 1757–1771 (2004)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2005 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Lindsay, D.G., Cox, S. (2005). Effective Confidence Region Prediction Using Probability Forecasters. In: Miksch, S., Hunter, J., Keravnou, E.T. (eds) Artificial Intelligence in Medicine. AIME 2005. Lecture Notes in Computer Science(), vol 3581. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11527770_66

Download citation

  • DOI: https://doi.org/10.1007/11527770_66

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-540-27831-3

  • Online ISBN: 978-3-540-31884-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics