Assessing the Effectiveness of Bayesian Feature Selection

  • Ian T. Nabney
  • David J. Evans
  • Yann Brulé
  • Caroline Gordon
Part of the Advanced Information and Knowledge Processing book series (AI&KP)


A practical Bayesian approach for inference in neural network models has been available for ten years, and yet it is not used frequently in medical applications. In this chapter we show how both regularization and feature selection can bring significant benefits in diagnostic tasks through two case studies: heart arrhythmia classification based on ECG data and the prognosis of lupus. In the first of these, the number of variables was reduced by two thirds without significantly affecting performance, while in the second, only the Bayesian models had an acceptable accuracy. In both tasks, neural networks outperformed other pattern recognition approaches.


Systemic Lupus Erythematosus True Positive Rate Hide Unit Ventricular Ectopic Beat Weight Decay 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. [1]
    J. O. Berger. Statistical Decision Theory and Bayesian Analysis. Springer-Verlag, New York, 2nd edition, 1985.Google Scholar
  2. [2]
    C. M. Bishop. Neural Networks for Pattern Recognition. Oxford University Press, 1995.Google Scholar
  3. [3]
    C. M. Bishop, M. Svensén, and C. K. I. Williams. GTM: The Generative Topographic Mapping. Neural Computation, 10(1):215–235, 1996.CrossRefGoogle Scholar
  4. [4]
    J. Boutkan. A Guide to Electrocardiography. Macmillan, 1972.Google Scholar
  5. [5]
    Y. Brulé. Lupus prognosis: a clinical study. Master’s thesis, Aston University, Birmingham, UK, 2000.Google Scholar
  6. [6]
    C. Chatfield and A. J. Collins. Introduction to Multivariate Analysis. Chapman and Hall, 1980.Google Scholar
  7. [7]
    L. Gamlyn, P. Needham, S. M. Sopher, and T. J. Harris. The development of a neural network-based ambulatory ECG monitor. Neural Computing and Applications, 8:273–278, 1999.CrossRefGoogle Scholar
  8. [8]
    B. Hassibi and D. G. Stork. Second order derivatives for network pruning: optimal brain surgeon. In S. J. Hanson, J. D. Cowan, and C. L. Giles, editors, Advances in Neural Information Processing Systems, volume 5, pages 164–171, San Mateo, CA, 1993. Morgan Kaufmann.Google Scholar
  9. [9]
    E. M. Hay, P. A. Bacon, C. Gordon, D. A. Isenberg, P. Maddison, M. L. Snaith, D. P. M. Symmons, N. Viner, and A. Zoma. The BILAG index: a reliable and valid instrument for measuring clinical disease activity in systemic lupus erythematosus. Quart. J. Medicine, 86:447–458, 1993.Google Scholar
  10. [10]
    D. A. Isenberg and C. Gordon. From BILAG to BLIPS. Disease activity assessment in lupus: past, present and future. Lupus, 9:651–654, 2000.CrossRefGoogle Scholar
  11. [11]
    A. E. Johnson, C. Gordon, R. G. Palmer, and P. A. Bacon. The prevalence and incidence of systemic lupus erythematosus in Birmingham, UK, related to ethnicity and country of birth. Arthritis Rheum., 38:551–558, 1995.Google Scholar
  12. [12]
    T. Kohonen. Self-Organizing Maps. Springer-Verlag, Berlin, 1995.Google Scholar
  13. [13]
    Y. Le Cun, J. S. Denker, and S. A. Solla. Optimal brain damage. In D. S. Touretzky, editor, Advances in Neural Information Processing Systems, volume 2, pages 598–605, San Mateo, CA, 1990. Morgan Kaufmann.Google Scholar
  14. [14]
    D. J. C. MacKay. Bayesian interpolation. Neural Computation, 4(3):415–447, 1992.Google Scholar
  15. [15]
    D. J. C. MacKay. The evidence framework applied to classification networks. Neural Computation, 4(5):720–736, 1992.Google Scholar
  16. [16]
    D. J. C. MacKay. A practical Bayesian framework for back-propagation networks. Neural Computation, 4(3):448–472, 1992.Google Scholar
  17. [17]
    I. T. Nabney. Netlab: Algorithms for Pattern Recognition. Springer-Verlag, London, 2002.Google Scholar
  18. [18]
    R. M. Neal. Bayesian Learning for Neural Networks. Lecture Notes in Statistics: Number 118. Springer-Verlag, New York, 1996.Google Scholar
  19. [19]
    P. Standing, M. Dent, A. Craig, and B. Glenville. Changes in referral patterns to cardiac out-patient clinics with ambulatory ECG monitoring in general practice. Brit. J. Cardiol., 6:394–398, 2001.CrossRefGoogle Scholar
  20. [20]
    T. Stoll, C. Gordon, B. Seifert, K. Richardson, J. Malik, P. A. Bacon, and D. A. Isenberg. Consistency and validity of patient administered assessment of quality of life by the mos sf-36; its association with disease activity and damage in patients with systemic lupus erythematosus. J. Rheum, 24:1608–1614, 1997.Google Scholar
  21. [21]
    H. H. Thodberg. Ace of Bayes: application of neural networks with pruning. Technical Report 1132E, The Danish Meat Research Institute, Maglegaardsvej 2, DK-4000 Roskilde, Denmark, 1993.Google Scholar

Copyright information

© Springer-Verlag London Limited 2005

Authors and Affiliations

  • Ian T. Nabney
    • 1
  • David J. Evans
    • 1
  • Yann Brulé
    • 1
  • Caroline Gordon
    • 2
  1. 1.Neural Computing Research GroupAston UniversityBirminghamUK
  2. 2.Faculty of Medicine and DentistryUniversity of BirminghamBirminghamUK

Personalised recommendations