Abstract
Very few national security screening programs have been subjected to a rigorous scientific examination of instrument validity. Yet, validity is a core element in program evaluation and refers to gathering evidence to support the intended use and interpretation of tests (here, screening methods). The lack of such validation assessments as an element of developing a scientific evidence base for counterterrorism programming is surprising, given our nation’s strong national security agenda. This chapter addresses considerations in designing and implementing validation assessments. After presenting types of validity and reliability and describing the applicability of each to security screening programs, the author presents methodologies for validation assessment and considers challenges in the implementation of such assessments in the context of operationally deployed programs.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
There is a second level of negative results that should also be considered; that is, the individuals who were not selected for the screening at all (i.e., not sent through the AIT). For these cases, it is unknown whether they would have correctly or incorrectly been identified as a positive or negative on the AIT.
References
American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Standards for educational and psychological testing. Washington: American Educational Research Association, American Psychological Association, & National Council on Measurement in Education.
Caldeira, J. D. (1980). Parametric assumptions of some “nonparametric” measures of sensory efficiency. Human Factors, 22(1), 119–120.
Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37–46.
Cook, T. D., & Campbell, D. T. (1979). Quasi experimentation: Design and analytical issues for field settings. Chicago: Rand McNally.
Cornfield, J. (1951). A method of estimating comparable rates from clinical data. Journal of the National Cancer Institute, 11, 1269–1275.
Craig, A. (1979). Nonparametric measures of sensory efficiency for sustained monitoring tasks. Human Factors, 21(1), 69–78.
Crocker, L., & Algina, J. (1986). Introduction to classical and modern test theory. New York: Holt, Rinehart, and Winston.
Davies, H. T. O., Crombie, I. K., & Tavakoli, M. (1998). When can odds ratios mislead? British Medical Journal, 316, 989–991.
DePaulo, B. M., Lindsay, J. J., Malone, B. E., Muhlenbruck, L., Charlton, K., & Cooper, H. (2003). Cues to deception. Psychological Bulletin, 129, 74–118.
Edwards, A. W. F. (1963). The measure of association in a 2  ×  2 table. Journal of the Royal Statistical Society, Series A, 126(1), 109–114.
Farrington, D. P. (2003). Methodological quality standards for evaluation research. The Annals of the American Academy of Political and Social Science, 587(1), 49–68.
Frank, M. G., Maccario, C. J., & Govindaraju, V. (2009). Behavior and security. In P. Seidenstat & F. X. Splane (Eds.), Protecting airline passengers in the age of terrorism. Santa Barbara: Praeger.
Grant, J. A. (1974). Quantitative evaluation of a screening program. American Journal of Public Health, 64(1), 66–71.
Green, D. M., & Swets, J. A. (1966). Signal detection theory and psychophysics. New York: Wiley. (Reprinted by Peninsula Publishing, Los Altos, 1988).
LaFree, G. (2011). Generating terrorism event data bases: Results from the global terrorism database, 1970 to 2008. In C. Lum & L. W. Kennedy (Eds.), Evidence-based counterterrorism policy. New York: Springer.
Lawshe, C. H. (1975). A quantitative approach to content validity. Personnel Psychology, 28, 563–575.
Lum, C., Kennedy, L. W., & Sherley, A. J. (2006). Are counter-terrorism strategies effective?: The results of the Campbell systematic reviews on counter-terrorism evaluation research. Journal of Experimental Criminology, 2(4), 489–516.
National Research Council. (2003). The polygraph and lie detection. Washington: National Academies Press.
Norman, D. A. (1964). A comparison of data obtained under different false-alarm rates. Psychological Science, 1, 125–126.
Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory (3rd ed.). New York: McGraw Hill.
Pedhazur, E. J., & Schmelkin, L. P. (1991). Measurement, design, and analysis: An integrated approach. Hillsdale: Lawrence Erlbaum Associates.
Sackett, P. R., & Decker, P. J. (1979). Detection of deception in the employment context: A review and critical analysis. Personnel Psychology, 32, 487–506.
Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston: Houghton Mifflin.
Sheehan, I. S. (2011). Assessing and comparing data sources for terrorism research. In C. Lum & L. W. Kennedy (Eds.), Evidence-based counterterrorism policy. New York: Springer.
Sistrom, C. L., & Garvan, C. W. (2004). Proportions, odds, and risk. Radiology, 230(1), 12–19.
Swets, J. A. (1996). Signal detection theory and ROC analysis in psychology and diagnostics: Collected papers. Mahwah: Lawrence Erlbaum Associates.
U.S. Department of Homeland Security. (2011). Privacy impact assessment update for TSA advanced imaging technology. Washington: U.S. Government Printing Office.
U.S. Transportation Security Administration. (2011). Retrieved April 13, 2011, from http://www.tsa.gov/approach/tech/ait/how_it_works.shtm.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2012 Springer Science+Business Media, LLC
About this chapter
Cite this chapter
Costigan, T.E. (2012). The Importance of Instrument Validity in Evaluating Security Screening Programs. In: Lum, C., Kennedy, L. (eds) Evidence-Based Counterterrorism Policy. Springer Series on Evidence-Based Crime Policy, vol 3. Springer, New York, NY. https://doi.org/10.1007/978-1-4614-0953-3_9
Download citation
DOI: https://doi.org/10.1007/978-1-4614-0953-3_9
Published:
Publisher Name: Springer, New York, NY
Print ISBN: 978-1-4614-0952-6
Online ISBN: 978-1-4614-0953-3
eBook Packages: Humanities, Social Sciences and LawSocial Sciences (R0)