The Validation of Screening Tests: Meet the New Screen same as the Old Screen?
- 345 Downloads
The focus of this report is to examine the process of validation of new screening tests designed to detect the problem gambler in research and practice settings. A hierarchical or phases of evaluation model is presented as a conceptual framework to describe the basic features of the validation process and its implications for application and interpretation of test results. The report describes a number of threats to validity in the form of sources of unintended bias that when unrecognized may lead to incorrect interpretations of study results and the drawing of incorrect conclusions about the usefulness of the new screening tests. Examples drawn from the gambling literature on problem gambling are used to illustrate some of the more important concepts including spectrum bias and clinical variation in test accuracy. The concept of zones of severity and the bias inherent in selecting criterion thresholds are reviewed. A definition of reference or study gold standard is provided. The use of 2-stage designs to establish validity by efficiently using reference standards to determine indices of accuracy and prevalence is recommended.
KeywordsValidation Screening tests Unintended bias Phases of evaluation research Gold standards Reference standards Severity continuum
- Abramson, J. H. (1996). Cross-sectional studies. In W. W. Holland, R. Detals, & G. Knox (Eds.), Methods in public health (pp. 517–535). Oxford: Oxford University Press.Google Scholar
- American Psychiatric Association. (1994). Diagnostic and statistical manual of mental disorders (Fourth ed.). Washington, DC: American Psychiatric Association.Google Scholar
- Bellringer, M., Abbott, M. W., Volberg, R. A., Garrett, N., & Coombes, R. (2007). Problem gambling assessment and screening instruments: Phase one report. Auckland: Gambling Research Centre, Auckland University of Technology.Google Scholar
- Chipman, M., Govoni, R., Jazmaji, V., Wilson, S., & Gao, P. (2008). High vs. low risk gambling: What is the difference? Reported prepared for the Ontario Problem Gambling Research Centre.Google Scholar
- Dickerson, M. (1993). A preliminary exploration of a two-stage methodology in the assessment of the extent and degree of gambling related problems in the Australian population. In W. R. Eadington, J. Cornelius, & J. I. Tabor (Eds.), Gambling behavior and problem gambling (pp. 347–363). Reno, Nevada: Institute for the Study of Gambling and Commercial Gaming, University of Nevada.Google Scholar
- Ferris, J., & Wynne, H. (2001). The Canadian problem gambling index: Final report. Ottawa: Canadian Centre on Substance Abuse.Google Scholar
- Gambino, B. (2005a). Interpreting prevalence estimates of pathological gambling: Implications for policy. Journal of Gambling Issues, 14. Available at http://www.camh.net/egambling/issue14/jgi_14_gambino.html.
- Gambino, B. (2005b). Going for the gold. The brief addiction science information source. Available at http://www.basisonline.org.
- Gambino, B. (2006c). Clarifying the at-risk label: A commentary. Journal of Gambling Issues, 16. Available at http://www.camh.net/egambling/issue16/issue16/jgi_16_gambino.html.
- Gambino, B. (2009). Should gambling be included in public health surveillance systems? Journal of Gambling Issues, Issue 23, June. Available at http://www.camh.net/egambling/issue23pdfs/07b-gambino.pdf.
- Gambino, B., & Lesieur, H. (2006). The South Oaks gambling screen (SOGS): A rebuttal to critics. Journal of Gambling Issues, 17. Available at http://www.camh.net/egambling/issue17/gambino.html.
- Gebauer, L., LaBrie, R., & Shaffer, H. J. (2010). Optimizing DSM-IV-TR classification accuracy: A brief biosocial screen for detecting current gambling disorders among gamblers in the general household population. Canadian Journal of Psychiatry, 55, 82–90.Google Scholar
- Gerstein, D., Hoffman, J., Larison, C., Murphy, S., Palmer, A., Chuchro, L., et al. (1999). Gambling impact and behavior study. Report to the national gambling impact study commission. Chicago: NORC.Google Scholar
- Grove, R., McBride, O., & Slade, T. (2010). Toward DSM-V: Exploring diagnostic thresholds for alcohol dependence and abuse. Alcohol and Addiction, 45, 45–52.Google Scholar
- Hawkins, R. C. (2005). The evidence based medicine approach to diagnostic testing: Practicalities and limitations. Clinical Biochemical Review, 26, 1–7.Google Scholar
- Hodgins, D. C., Stea, J. N., & Grant, J. E. (2011). Gambling disorders. Lancet, online first pp. 1–11.Google Scholar
- Kleinbaum, D. G., Kupper, L. L., & Morgenstern, H. (1982). Epidemiologic research: Principles and quantitative methods. New York: Van Nostrand Reinhold.Google Scholar
- Knottnerus, J. A., & van Weel, C. (2002). General introduction: evaluation of diagnostic procedures. In J. A. Knottnerus (Ed.), The evidence base of clinical diagnosis (pp. 1–17). London: BMJ Publishing Group.Google Scholar
- Kraemer, H. C. (1992). Evaluating medical tests: Objective and quantitative guidelines. Newbury Park, California: Sage.Google Scholar
- Kroenke, K., & Spitzer, R. L. (2002). The PHQ-9: A new depression diagnostic and severity measure. Psychiatric Annals, 32, 1–7.Google Scholar
- LaPlante, D. A., Nelson, S. E., LaBrie, R. A., & Shaffer, H. J. (2008). Stability and progression of disordered gambling: Lessons from longitudinal studies. Canadian Journal of Psychiatry, 53, 52–60.Google Scholar
- Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 13–103). Washington, DC: American Council on Education and National Council on Measurement in Education.Google Scholar
- Neal, P., Delfabbro, P., & O’Neil, M. (2005). Problem gambling and harm: Towards a national definition. Adelaide: South Australian Centre for Economic Studies.Google Scholar
- Pepe, M. S. (2003). The statistical evaluation of medical tests for classification and prediction. Oxford UK: Oxford University Press.Google Scholar
- Petry, N. (2011). Non-substance related addictions: Their place in DSM-V (abstract). Paper to be presented at American Public Health Association Meetings, October 31, 2011.Google Scholar
- Rutjes, A. W., Reitsma, J. B., Coomarasamy, A., Khan, K. S., & Bossuyt, P. M. (2007). Evaluation of diagnostic tests when there is no gold standard. Health, Technological Assessment, 11, 50.Google Scholar
- Sackett, D. L., Haynes, R. B., & Tugwell, P. (1985). Clinical epidemiology: A basic science for clinical medicine. Boston: Little Brown.Google Scholar
- Schatzkin, A., Connor, R. J., Taylor, P. R., & Bunnag, B. (1987). Comparing new and old screening tests when a reference procedure cannot be performed on all screenees: Example of automated cytometry for early detection of cervical cancer. American Journal of Epidemiology, 125, 672–678.PubMedGoogle Scholar
- Schlesselman, J. J. (1982). Case-control studies. New York: Oxford University Press.Google Scholar
- Shaffer, H. J., & Gambino, B. (1990). Epilogue: Integrating treatment choices. In H. B. Milkman & L. I. Sederer (Eds.), Treatment choices for alcoholism and substance abuse (pp. 351–375). Lexington, MA: Lexington Books.Google Scholar
- Shaffer, H. J., Hall, M. N., & Vander Bilt, J. (1997). Estimating the prevalence of disordered gambling behavior in the United States and Canada: A meta-analysis Boston. MA: Harvard Medical School Division on Addictions.Google Scholar
- Sox, H. C., Jr., Blatt, M. A., Higgins, M. C., & Marton, K. I. (1988). Medical decision making. Boston: Butterworths-Heinemann.Google Scholar
- Statistics Canada (2009). Table 3. Stratum-specific likelihood ratios for selected health status characteristics. Validation of disability categories derived from Health utilities index mark 3 scores. Health reports, 20, no. 2. Available at http://www.statcan.gc.ca.
- Stinchfield, R., Govoni, R., & Frisch, G. R. (2007a). A review of screening and assessment instruments for problem and pathological gambling. In G. Smith, D. Hodgins, & R. J. Williams (Eds.), Research and measurement issues in gambling studies (pp. 179–213). New York: Academic Press.Google Scholar
- Thomas, S., Jackson, A., & Blaszczynski, A. (2003). Measuring problem gambling: Evaluation of the Victorian gambling screen. Melbourne: Gambling Research Panel.Google Scholar
- Toce-Gerstein, M., & Gerstein, D. R. (2004). Of time and the chase: Lifetime versus past-year measures of pathological gambling. E Journal of Gambling Issues, 10. Available at: http://www.camh.net/egambling/issue10/ejgi_10_gerstein.html.
- Volberg, R. A. (2002). Gambling and problem gambling in Nevada. Report to the Nevada Department of Human Resources.Google Scholar
- Volberg, R. A. (2003). Gambling and problem gambling in Arizona. Report to the Arizona Lottery.Google Scholar
- Volberg, R. A., & Bernhard, B. (2006). The 2006 study of gambling and problem gambling in New Mexico. Report to the responsible gaming association of New Mexico.Google Scholar
- Volberg, R. A., & Young, M. M. (2008). Using SOGS vs. CPGI in problem gambling screening and assessment. Toronto: Ontario Problem Gambling Research Centre.Google Scholar
- Wenzel, M., McMillen, J., Marshall, D., & Ahmed, E. (2004). Validation of the Victorian gambling screen. Melbourne: Australian National University.Google Scholar
- Whiting, P., Rutjes, A. W., Reitsma, J. B., Glas, A. S., Bossuyt, P. M., & Kleijnen, J. (2005). Sources of variation and bias in studies of diagnostic accuracy. Annals of Internal Medicine, 140, 189–202.Google Scholar
- Wiebe, J., Maitland, S. B., Hodgins, D., Davey, A., & Gottlieb, B. (2009). Transition and stability of problem gambling behaviours. Final report to the addiction foundation of Manitoba.Google Scholar