Educational Psychology Review

, Volume 25, Issue 3, pp 331–337 | Cite as

Recommendations for Practice: Justifying Claims of Generalizability

Commentary

Abstract

Recommendations for practice are routinely included in articles that report educational research. Robinson et al. suggest that reports of primary research should not routinely do so. They argue that single primary research studies seldom have sufficient external validity to support claims about practice policy. In this article, I draw on recent statistical research that has formalized subjective notions about generalizability from experiments. I show that even rather large experiments often do not support generalizations to policy-relevant inference populations. This suggests that single primary studies are unlikely to be sufficiently generalizable to support recommendations for practice.

Keywords

Replication Practice policy Generalization 

References

  1. Cochran, W. G. (1968). The effectiveness of adjustment by subclassification in removing bias in observational studies. Biometrics, 24, 295–313.CrossRefGoogle Scholar
  2. Hedges, L. V., & O'Muircheartaigh, C. (2010). Improving inference for population level treatment effects in social experiments. Working paper, Northwestern University Institute for Policy Research.Google Scholar
  3. Kalton, G. (1968). Standardization: a technique to control for extraneous variables. Applied Statistics, 17, 118–136.CrossRefGoogle Scholar
  4. Kitagawa, E. M. (1964). Standardized comparisons in population research. Demography, 1, 296–315.Google Scholar
  5. Kruskal, W., & Mosteller, F. (1979). Representative sampling III: the current statistical literature. International Statistical Review, 47, 245–265.CrossRefGoogle Scholar
  6. Little, R. J. A., & Rubin, D. B. (2002). Statistical analysis with missing data. New York: Wiley.Google Scholar
  7. Mosteller, F., Light, R. J., & Sachs, J. A. (1996). Sustained inquiry in education: lessons learned from skill grouping and class size. Harvard Educational Review, 66, 797–842.Google Scholar
  8. Nye, B., Hedges, L. V., & Konstantopoulos, S. (2000). The effects of small classes on achievement: the results of the Tennessee class size experiment. American Educational Research Journal, 37, 123–151.CrossRefGoogle Scholar
  9. O'Muircheartaigh, C., & Hedges, L. V. (2013). Generalizing from unrepresentative experiments: a stratified propensity score approach. Journal of the Royal Statistical Society, Series C (in press)Google Scholar
  10. Oaxaca, R. (1973). Male–female wage differentials in urban labor markets. International Economic Review, 14, 693–709.CrossRefGoogle Scholar
  11. Robinson, D. H., Levin, J. R., Schraw, G., Patal, E. A., & Hunt, E. B. (2013). On going (way) beyond one's data: a proposal to restrict recommendations for practice in primary educational research journals. Educational Psychology Review (in press).Google Scholar
  12. Roschelle, J., Shechtman, N., Tatar, D., & Hegedus, S. (2010). Integration of technology, curriculum, and professional development for advancing middle school mathematics. American Educational Research Journal, 47, 833–878.CrossRefGoogle Scholar
  13. Rosenbaum, P. R., & Rubin, D. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70, 41–55.CrossRefGoogle Scholar
  14. Rosenberg, P. (1962). Test factor standardization as a method of interpretation. Social Forces, 41, 53–61.CrossRefGoogle Scholar
  15. Stuart, E. A., Cole, S. R., Bradshaw, C. P., & Leaf, P. J. (2011). The use of propensity scores to assess the generalizability of results from randomized trials. Journal of the Royal Statistical Society, Series A, Part, 2, 369–386.Google Scholar
  16. Tipton, E. (2013). Improving generalizations from experiments using propensity score subclassification: assumptions, properties, and contexts. Journal of Educational and Behavioral Statistics. (in press)Google Scholar
  17. Tipton, E., & Hedges, L. V. (2013). Sample selection in randomized experiments: A new method using propensity score stratified sampling. Journal of Research on Educational Effectiveness (in press)Google Scholar

Copyright information

© Springer Science+Business Media New York 2013

Authors and Affiliations

  1. 1.Institute for Policy ResearchNorthwestern UniversityEvanstonUSA

Personalised recommendations