Advertisement

Springer Nature is making SARS-CoV-2 and COVID-19 research free. View research | View latest news | Sign up for updates

Number Needed to Detect

Nuances in the Use of a Simple and Intuitive Signal Detection Metric

  • 66 Accesses

  • 1 Citations

Abstract

Data mining algorithms are increasingly being used to support the process of signal detection and evaluation in pharmacovigilance. Published data mining exercises formulated within a screening paradigm typically calculate classical performance indicators such as sensitivity, specificity, predictive value and receiver operator characteristic curves. Extrapolating signal detection performance from these isolated data mining exercises to performance in real-world pharmacovigilance scenarios is complicated by numerous factors and some published exercises may promote an inappropriate and exclusive focus on only one aspect of performance. In this article, we discuss a variation on positive predictive value that we call the ‘number needed to detect’ that provides a simple and intuitive screening metric that might usefully supplement the usual presentations of data mining performance. We use a series of figures to demonstrate the nature and application of this metric, and selected adaptive variations. Even with simple and intuitive metrics, precisely quantifying the performance of contemporary data mining algorithms in pharmacovigilance is complicated by the complexity of the phenomena under surveillance and the manner in which the data are recorded in spontaneous reporting systems.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

References

  1. 1.

    Bate A, Lindquist M, Edwards IR, et al. A bayesian neural network method for adverse drug reaction signal generation. Eur J Clin Pharmacol 1998; 54: 315–21

  2. 2.

    DuMouchel W. Bayesian data mining in large frequency tables with an application to the FDA spontaneous reporting system. Am Stat 1999; 53 (3): 177–202

  3. 3.

    Moore N. Thiessard F, Begaud B. The history of disproportionality measures (reporting odds ratio, proportional reporting rates) in spontaneous reporting of adverse drug reactions. Pharmacoepidemiol Drug Saf 2005; 14: 285–86

  4. 4.

    Hauben M, Madigan D, Gerrits C, et al. The role of data mining in pharmacovigilance. Expert Opin Drug Saf 2005; 4 (5): 929–48

  5. 5.

    Hauben M. Signal detection in the pharmaceutical industry: integrating clinical and computational approaches. Drug Saf 2007; 30 (7): 627–30

  6. 6.

    Hauben M, Reich L, Chung S. Post-marketing surveillance of potentially fatal reactions to oncology drugs: potential utility of two signal detection algorithms. Eur J Clin Pharmacol 2004; 60 (10): 747–50

  7. 7.

    Hauben M, Reich L. Communication of findings in pharmacovigilance: use of the term “signal” and the need for precision in its use. Eur J Clin Pharmacol 2005; 61 (5–6): 479–80

  8. 8.

    Hauben M, Reich L. Reply: the evaluation of data mining methods for the simultaneous and systematic detection of safety signals in large databases: lessons to be learned. Response to letter by Levine, et al. Br J Clin Pharmacol 2006; 61: 115–7

  9. 9.

    Hauben M, Aronson JK. Gold standards in pharmacovigilance-the use of definitive anecdotal reports of adverse drug reactions as pure gold and high-grade ore. Drug Saf 2007; 30: 645–55

  10. 10.

    Henegar C, Bousquet C, Lillo-Le Louët A, et al. A knowledge based approach for automated signal generation in pharmacovigilance. Proceedings of the 11th World Congress on Medical Informatics; 2004 Sep 7–11; San Francisco (CA) [online]. Available from URL: http://cmbi.bjmu.edu.cn/news/report/2004/medinfo2004/pdffiles/papers/4696Henegar.pdf [Accessed 2008 Jan 28]

  11. 11.

    Hauben M, Van Puijenbroek EP, Gerrits C, et al. Data mining in pharmacovigilance: lessons from phantom ships. Eur J Clin Pharmacol 2006; 62 (11): 967–70

Download references

Acknowledgements

The views expressed in this paper are those of the authors and not necessarily the official view of the European Medicines Agency. No sources of funding were used to assist in the preparation of this review. The authors have no conflicts of interest that are directly relevant to the content of this review. The authors would like to thank the following individuals who generously shared their time in reviewing early versions of this manuscript and providing thoughtful insights: Alan Hochberg, Jeffrey Aronson, David Goldsmith, David Madigan and Panos Tsintis.

Author information

Correspondence to Dr Manfred Hauben.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Hauben, M., Vogel, U. & Maignen, F. Number Needed to Detect. Pharm Med 22, 13–22 (2008). https://doi.org/10.1007/BF03256678

Download citation

Keywords

  • Data Mining Algorithm
  • Spontaneous Reporting System
  • Credible Signal
  • Classifier Efficiency
  • Disproportionate Reporting