Recommendations for Practice: Justifying Claims of Generalizability
- Larry V. Hedges
- … show all 1 hide
Rent the article at a discountRent now
* Final gross prices may vary according to local VAT.Get Access
Recommendations for practice are routinely included in articles that report educational research. Robinson et al. suggest that reports of primary research should not routinely do so. They argue that single primary research studies seldom have sufficient external validity to support claims about practice policy. In this article, I draw on recent statistical research that has formalized subjective notions about generalizability from experiments. I show that even rather large experiments often do not support generalizations to policy-relevant inference populations. This suggests that single primary studies are unlikely to be sufficiently generalizable to support recommendations for practice.
- Cochran, W. G. (1968). The effectiveness of adjustment by subclassification in removing bias in observational studies. Biometrics, 24, 295–313. CrossRef
- Hedges, L. V., & O'Muircheartaigh, C. (2010). Improving inference for population level treatment effects in social experiments. Working paper, Northwestern University Institute for Policy Research.
- Kalton, G. (1968). Standardization: a technique to control for extraneous variables. Applied Statistics, 17, 118–136. CrossRef
- Kitagawa, E. M. (1964). Standardized comparisons in population research. Demography, 1, 296–315.
- Kruskal, W., & Mosteller, F. (1979). Representative sampling III: the current statistical literature. International Statistical Review, 47, 245–265. CrossRef
- Little, R. J. A., & Rubin, D. B. (2002). Statistical analysis with missing data. New York: Wiley.
- Mosteller, F., Light, R. J., & Sachs, J. A. (1996). Sustained inquiry in education: lessons learned from skill grouping and class size. Harvard Educational Review, 66, 797–842.
- Nye, B., Hedges, L. V., & Konstantopoulos, S. (2000). The effects of small classes on achievement: the results of the Tennessee class size experiment. American Educational Research Journal, 37, 123–151. CrossRef
- O'Muircheartaigh, C., & Hedges, L. V. (2013). Generalizing from unrepresentative experiments: a stratified propensity score approach. Journal of the Royal Statistical Society, Series C (in press)
- Oaxaca, R. (1973). Male–female wage differentials in urban labor markets. International Economic Review, 14, 693–709. CrossRef
- Robinson, D. H., Levin, J. R., Schraw, G., Patal, E. A., & Hunt, E. B. (2013). On going (way) beyond one's data: a proposal to restrict recommendations for practice in primary educational research journals. Educational Psychology Review (in press).
- Roschelle, J., Shechtman, N., Tatar, D., & Hegedus, S. (2010). Integration of technology, curriculum, and professional development for advancing middle school mathematics. American Educational Research Journal, 47, 833–878. CrossRef
- Rosenbaum, P. R., & Rubin, D. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70, 41–55. CrossRef
- Rosenberg, P. (1962). Test factor standardization as a method of interpretation. Social Forces, 41, 53–61. CrossRef
- Stuart, E. A., Cole, S. R., Bradshaw, C. P., & Leaf, P. J. (2011). The use of propensity scores to assess the generalizability of results from randomized trials. Journal of the Royal Statistical Society, Series A, Part, 2, 369–386.
- Tipton, E. (2013). Improving generalizations from experiments using propensity score subclassification: assumptions, properties, and contexts. Journal of Educational and Behavioral Statistics. (in press)
- Tipton, E., & Hedges, L. V. (2013). Sample selection in randomized experiments: A new method using propensity score stratified sampling. Journal of Research on Educational Effectiveness (in press)
- Recommendations for Practice: Justifying Claims of Generalizability
Educational Psychology Review
Volume 25, Issue 3 , pp 331-337
- Cover Date
- Print ISSN
- Online ISSN
- Springer US
- Additional Links
- Practice policy
- Larry V. Hedges (1)
- Author Affiliations
- 1. Institute for Policy Research, Northwestern University, 2040 N. Sheridan Road, Evanston, IL, 60208, USA