Abstract
Validity is a fundamental issue in the development and evaluation of psychometric instruments in health care. With an aim towards informing validation practice, we investigated the reporting of validity evidence in the field of health care, focusing on a journal that, by design and scope, is meant to influence health outcomes researchers, technology developers, regulators, policy makers, and health economists: Value in Health, the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) official journal. A systematic search of papers since the journal’s inception (January 1998) to December 2010 was conducted using the official journal website. Only empirical psychometric validation papers were included in the analysis. The coding was conducted by two of the authors independently. A total of 68 articles met our inclusion criteria. The number and percentage of reports of the broad categories of the sources of validity evidence included internal consistency reliability (69.1 %), construct (50.0 %), discriminant (33.8 %), convergent (33.8 %), content (25.0 %), criterion (20.6 %), response processes (4.4 %), and consequences (2.9 %). Researchers conducting validation studies are not relying on only one source of validity evidence at the exclusion of all others and hence representing a broad perspective on psychometric validity. However, some sources of validity evidence such as response processes and consequences are rarely reported. Readers and practitioners should not uncritically accept the often-stated conclusion that “the instrument is valid”.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Acquadro, C., Berzon, R., Dubois, D., Leidy, N. K., Marquis, P., Revicki, D., et al. (2003). Incorporating patient’s perspective into drug development and communication: An ad hoc task force report of the patient-reported outcomes (PRO) harmonization group meeting at the Food and Drug Administration, February 16, 2001. Value in Health, 6, 522–531.
American Educational Research Association (AERA), American Psychological Association (APA), & National Council on Measurement in Education (NCME). (1999). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.
Auewarakul, C., Downing, S. M., Jaturatamrong, U., & Praditsuwan, R. (2005). Sources of validity evidence for an internal medicine student evaluation system: An evaluative study of assessment methods. Medical Education, 39, 276–283.
Chan, A. W., & Altman, D. G. (2005). Epidemiology and reporting of randomised trials published in PubMed journals. Lancet, 365, 1159–1162.
Cizek, G. J., Rosenberg, S. L., & Koons, H. H. (2008). Sources of validity evidence for educational and psychological tests. Educational and Psychological Measurement, 68, 397–412.
Cizek, G. J., Bowen, D., & Church, K. (2010). Sources of validity evidence for educational and psychological tests: A follow-up study. Educational and Psychological Measurement, 70, 732–743.
Food and Drug Administration. (2009). Guidance for industry: Patient-reported outcome measures: Use in medical product development to support labeling claims. Rockville: Department of Health and Human Services, Food and Drug Administration, Center for Drug Evaluation and Research.
Gelin, M. N., & Zumbo, B. D. (2003). Differential item functioning results may change depending on how an item is scored: An illustration with the Center for Epidemiologic Studies Depression Scale. Educational and Psychological Measurement, 63, 65–74.
Hojat, M. (2007). Empathy in patient care: Antecedents, development, measurement, and outcomes. New York: Springer.
Hojat, M., Mangione, S., Nasca, T. J., Cohen, M. J. M., Gonnella, J. S., Erdmann, J. B., et al. (2001). The Jefferson scale of empathy: Development and preliminary psychometric data. Educational and Psychological Measurement, 61, 349–365.
Hubley, A. M., & Zumbo, B. D. (2011). Validity and the consequences of test interpretation and use. Social Indicators Research, 103, 219–230.
Hubley, A. M., & Zumbo, B. D. (2013). Psychometric characteristics of assessment procedures: An overview. In K. F. Geisinger (Ed.), APA handbook of testing and assessment in psychology (Vol. 1, pp. 3–19). Washington, DC: American Psychological Association Press.
Kane, M. T. (2006). Validation. In R. L. Brennan (Ed.), Educational measurement (4th ed., pp. 17–64). Westport: American Council on Education/Praeger.
Kopec, J. A., & Willison, K. D. (2003). A comparative review of four preference-weighted measures of health-related quality of life. Journal of Clinical Epidemiology, 56, 317–325.
Maurer, D. M. (2012). Screening for depression. American Family Physician, 85, 139–144.
Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 13–103). New York: American Council on Education and Macmillan.
Messick, S. (1995). Validity of psychological assessment: Validation of inferences from person’s responses and performances as scientific inquiry into score meaning. American Psychologist, 50, 741–749.
Moher, D., Tetzlaff, J., Tricco, A. C., Sampson, M., & Altman, D. G. (2007). Epidemiology and reporting characteristics of systematic reviews. PLoS Medicine, 4, e78.
Patrick, D. L., Burke, L., Gwaltney, C. J., Leidy, N. K., Martin, M. L., Molsen, E., et al. (2011a). Content validity – establishing and reporting the evidence in newly-developed patient-reported outcomes (PRO) instruments for medical product evaluation: ISPOR PRO good research practices task force report: Part I – Eliciting concepts for a new PRO instrument. Value in Health, 14, 967–977.
Patrick, D. L., Burke, L., Gwaltney, C. J., Leidy, N. K., Martin, M. L., Molsen, E., et al. (2011b). Content validity – establishing and reporting the evidence in newly developed patient-reported outcomes (PRO) instruments for medical product evaluation: ISPOR PRO good research practices task force report: Part 2 – assessing respondent understanding. Value in Health, 14, 978–988.
Pignone, M. P., Gaynes, B. N., & Rushton, J. L. (2002). Screening for depression in adults: A summary of the evidence for the U.S. Preventive Services Task Force. Annals of Internal Medicine, 136, 765–776.
Radloff, L. S. (1977). The CES-D scale: A self-report depression scale for research in the general population. Applied Psychological Measurement, 3, 385–401.
Richardson, C. G., & Zumbo, B. D. (2000). A statistical examination of the Health Utility Index-Mark III as a summary measure of health. Social Indicators Research, 51, 171–191.
Santanello, N. C., Baker, D., & Cappelleri, J. C. (2002). Regulatory issues for health-related quality of life – PhRMA Health Outcomes Committee Workshop, 1999. Value in Health, 5, 14–25.
Scientific Advisory Committee of the Medical Outcomes Trust. (2002). Assessing health status and quality-of-life instruments: Attributes and review criteria. Quality of Life Research, 11, 193–205.
Sharp, L. K., & Lipsky, M. S. (2002). Screening for depression across the lifespan: A review of measures for use in primary care settings. American Family Physician, 66, 1001–1008.
Valderas, J. M., Ferrer, J., Mendívil, M., et al. (2008). Development of EMPRO: A tool for the standardized assessment of patient-reported outcome measures. Value in Health, 11, 700–708.
Zillich, A. J., Doucette, W. R., & Carter, B. L. (2005). Development and initial validation of an instrument to measure physician–pharmacist collaboration from the physician perspective. Value in Health, 8, 59–66.
Zumbo, B. D. (1999). A handbook on the theory and methods of differential item functioning (DIF): Logistic regression modeling as a unitary framework for binary and Likert-type (ordinal) item scores. Ottawa: Directorate of Human Resources Research and Evaluation, Department of National Defense.
Zumbo, B. D. (2007). Validity: Foundational issues and statistical methodology. In C. R. Rao & S. Sinharay (Eds.), Psychometrics (Handbook of statistics, Vol. 26, pp. 45–79). Amsterdam/Boston: Elsevier Science B.V.
Zumbo, B. D. (2009). Validity as contextualized and pragmatic explanation, and its implications for validation practice. In R. W. Lissitz (Ed.), The concept of validity: Revisions, new directions and applications (pp. 65–82). Charlotte: Information Age Publishing.
Acknowledgement
To obtain a list of the articles included in this study, please contact the corresponding authors.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2014 Springer International Publishing Switzerland
About this chapter
Cite this chapter
Chan, E.K.H., Zumbo, B.D., Darmawanti, I., Mulyana, O.P. (2014). Reporting of Validity Evidence in the Field of Health Care: A Focus on Papers Published in Value in Health . In: Zumbo, B., Chan, E. (eds) Validity and Validation in Social, Behavioral, and Health Sciences. Social Indicators Research Series, vol 54. Springer, Cham. https://doi.org/10.1007/978-3-319-07794-9_15
Download citation
DOI: https://doi.org/10.1007/978-3-319-07794-9_15
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-07793-2
Online ISBN: 978-3-319-07794-9
eBook Packages: Humanities, Social Sciences and LawSocial Sciences (R0)