Advertisement

Social Indicators Research

, 103:231 | Cite as

Validation of Multilevel Constructs: Validation Methods and Empirical Findings for the EDI

  • Barry Forer
  • Bruno D. Zumbo
Article

Abstract

The purposes of this paper are to highlight the foundations of multilevel construct validation, describe two methodological approaches and associated analytic techniques, and then apply these approaches and techniques to the multilevel construct validation of a widely-used school readiness measure called the Early Development Instrument (EDI; Janus and Offord 2007). Validation evidence is presented regarding the multilevel covariance structure of the EDI, the appropriateness of aggregation to classroom and neighbourhood levels, and the effects of teacher and classroom characteristics on these structural patterns. The results are then discussed in the context of the theoretical framework of the EDI, with suggestions for future validation work.

Keywords

Multilevel Construct validation School readiness 

References

  1. Andrich, D., & Styles, I. (2004). Report on the psychometric properties of the Early Development Instrument (EDI) using the Rasch model. A technical paper commissioned for the development of the Australian Early Development Instrument (AEDI), Murdoch University.Google Scholar
  2. Beswick, J. F., Sloat, E. A., & Willms, J. D. (2004). A comparative study of bias in teacher ratings of language and emergent literacy skills. Unpublished manuscript.Google Scholar
  3. Blatchford, P., Bassett, P., Goldstein, H., & Martin, C. (2003). Are class size differences related to pupils’ educational progress and classroom processes? Findings from the institute of education class size study of children aged 5 to 7 years old. British Educational Research Journal, 29, 709–730.CrossRefGoogle Scholar
  4. Blatchford, P., Russell, A., Bassett, P., Brown, P., & Martin, C. (2007). The effect of class size on the teaching of pupils aged 7–11 years old. School Effectiveness and School Improvement, 18, 147–172.CrossRefGoogle Scholar
  5. Bliese, P. D. (2000). Within-group agreement, non-independence, and reliability: Implications for data aggregation and analysis. In K. J. Klein & S. J. Kozlowski (Eds.), Multilevel theory, research, and methods in organizations (pp. 349–381). San Francisco: Jossey-Bass.Google Scholar
  6. Bliese, P. D., & Halverson, R. R. (1998). Group size and measures of group-level properties: An examination of eta-squared and ICC values. Journal of Management, 24, 157–172.Google Scholar
  7. Bollen, K., & Lennox, R. (1991). Conventional wisdom on measurement: A structural equation perspective. Psychological Bulletin, 110, 305–314.CrossRefGoogle Scholar
  8. Brinkman, S., Silburn, S., Lawrence, D., Goldfeld, S., Sayers, M., & Oberklaid, F. (2007). Investigating the validity of the Australian Early Development Index. Early Education and Development, 18, 427–451.CrossRefGoogle Scholar
  9. Bronfenbrenner, U. (1977). Toward an experimental ecology of human development. American Psychologist, 32, 513–531.CrossRefGoogle Scholar
  10. Bryk, A. S., & Raudenbush, S. W. (1992). Hierarchical linear models. Thousand Oaks, CA: Sage.Google Scholar
  11. Byrne, B. M. (1998). Structural equation modeling with LISREL, PRELIS, and SIMPLIS: Basic concepts, applications, and programming. Mahwah, NJ: Erlbaum.Google Scholar
  12. Carpiano, R. M., Lloyd, J. E. V., & Hertzman, C. (2009). Concentrated affluence, concentrated disadvantage, and children’s readiness for school: A population-based, multi-level investigation. Social Science and Medicine, 69, 420–432.CrossRefGoogle Scholar
  13. Chan, D. (1998). Functional relations among constructs in the same content domain at different levels of analysis: A typology of composition models. Journal of Applied Psychology, 83, 234–246.CrossRefGoogle Scholar
  14. Chen, G., Mathieu, J. E., & Bliese, P. D. (2004a). A framework for conducting multi-level construct validation. In F. J. Yammarino & F. Dansereau (Eds.), Multi-level issues in organizational behaviour and processes. (pp. 273–303). The Netherlands: Elsevier.Google Scholar
  15. Chen, G., Mathieu, J. E., & Bliese, P. D. (2004b). Validating frogs and ponds in multi-level contexts: Some afterthoughts. In F. J. Yammarino & F. Dansereau (Eds.), Multi-level issues in organizational behaviour and processes. (pp. 335–343). The Netherlands: Elsevier.Google Scholar
  16. Cizek, G. J., & Bunch, M. B. (2007). Standard setting: A guide to establishing and evaluating performance standards on tests. Thousand Oaks, CA: Sage Publications, Inc.Google Scholar
  17. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Mahwah, NJ: Erlbaum.Google Scholar
  18. Cronbach, L., & Meehl, P. (1955). Construct validity in psychological tests. Psychological Bulletin, 52, 281–302.CrossRefGoogle Scholar
  19. Dansereau, F., Alutto, J. A., & Yammarino, F. J. (1984). Theory testing in organizational behavior: The “varient” approach. Englewood Cliffs, NJ: Prentice-Hall.Google Scholar
  20. Dansereau, F., Cho, J., & Yammarino, F. J. (2006). Avoiding the fallacy of the wrong level. Group and Organization Management, 31, 536–577.CrossRefGoogle Scholar
  21. Dansereau, F., & Yammarino, F. J. (2000). Within and between analysis: The varient paradigm as an underlying approach to theory building and testing. In K. J. Klein & S. J. Kozlowski (Eds.), Multilevel theory, research, and methods in organizations (pp. 425–466). San Francisco: Jossey-Bass.Google Scholar
  22. Dansereau, F., & Yammarino, F. J. (2004). Overview: Multi-level issues in organizational behavior and processes. In F. J. Yammarino & F. Dansereau (Eds.), Multi-level issues in organizational behaviour and processes (pp. Xiii–xxxiii). The Netherlands: Elsevier.CrossRefGoogle Scholar
  23. Dansereau, F., & Yammarino, F. J. (2006). Is more discussion about levels of analysis really necessary? When is such discussion sufficient? The Leadership Quarterly, 17, 537–552.CrossRefGoogle Scholar
  24. Diez-Roux, A. V. (1998). Bringing context back into epidemiology: Variables and fallacies in multilevel analysis. American Journal of Public Health, 88, 216–222.CrossRefGoogle Scholar
  25. Duku, E., & Janus, M. (2004). Stability and reliability of the Early Development Instrument: A population-based measure for communities (EDI). Department of Psychiatry and Biobehavioural Sciences, McMaster University, Canada. Annual Research Day.Google Scholar
  26. Duncan, G. J., Claessens, A., Huston, A. C., Pagani, L. S., Engel, M., Sexton, H., et al. (2007). School readiness and later achievement. Developmental Psychology, 43, 1428–1446.CrossRefGoogle Scholar
  27. Ellwein, M. C., Walsh, D. J., Eads, G. M., & Miller, A. (1991). Using readiness tests to route kindergarten students: The snarled intersection of psychometrics, policy, and practice. Educational Evaluation and Policy Analysis, 13, 159–175.Google Scholar
  28. Finn, J. D., Pannozzo, G. M., & Achilles, C. M. (2003). The “whys” of class size: Student behavior in small classes. Review of Educational Research, 73, 321–368.CrossRefGoogle Scholar
  29. Forer, B. (2009). Validation of multilevel constructs: Methods and empirical findings for the Early Development Instrument. Unpublished doctoral dissertation, University of British Columbia. Google Scholar
  30. Forget-Dubois, N., Lemelin, J.-P., Boivin, M., Ginette, D., Séguin, J. R., Vitaro, F., et al. (2007). Predicting early school achievement with the EDI: A longitudinal population-based study. Early Education and Development, 18, 405–426.CrossRefGoogle Scholar
  31. Gorsuch, R. L. (1983). Factor analysis (2nd ed.). Hillsdale, NJ: Erlbaum.Google Scholar
  32. Griffith, J. (2002). Is quality/effectiveness an empirically demonstrable school attribute? Statistical aids for determining appropriate levels of analysis. School Effectiveness and School Improvement, 13, 91–122.CrossRefGoogle Scholar
  33. Grilli, L., & Rampichini, C. (2007). Multilevel factor models for ordinal variables. Structural Equation Modeling, 14, 1–25.Google Scholar
  34. Guhn, M. & Goelman, H. (2011). Bioecological theory, early child development, and the validation of the population-level Early Development Instrument. Social Indicators Research. doi: 10.1007/s11205-011-9842-5.
  35. Guhn, M., Janus, M., & Hertzman, C. (2007). The Early Development Instrument: Translating school readiness assessment into community actions and policy planning. Early Education and Development, 18, 369–374.CrossRefGoogle Scholar
  36. Hofmann, D. A., & Jones, L. M. (2004). Some foundational and guiding questions for multi-level construct validation. In F. J. Yammarino & F. Dansereau (Eds.), Multi-level issues in organizational behavior and processes (pp. 305–315). The Netherlands: Elsevier.CrossRefGoogle Scholar
  37. Hox, J. (2002). Multilevel analysis: Techniques and applications. Mahwah, NJ: Erlbaum.Google Scholar
  38. Hu, L. T., & Bentler, P. M. (1995). Evaluating model fit. In R. H. Hoyle (Ed.), Structural equation modeling: Concepts, issues, and applications (pp. 76–99). Thousand Oaks, CA: Sage.Google Scholar
  39. Hymel, S., LeMare, L., & McKee, W. (2011). The Early Development Instrument (EDI): An examination of convergent and discriminant validity. Social Indicators Research. doi: 10.1007/s11205-011-9845-2.
  40. Janus, M. (2001). Validation of a teacher measure of school readiness with parent and child-care provider reports. Department of Psychiatry and Biobehavioural Sciences, McMaster University, Canada. Annual Research Day.Google Scholar
  41. Janus, M. (2002). Validation of the Early Development Instrument in a sample of First Nations children. Department of Psychiatry and Biobehavioural Sciences, McMaster University, Canada. Annual Research Day.Google Scholar
  42. Janus, M., Brinkman, S., Duku, E., Hertzman, C., Santos, R., Sayers, M., et al. (2007). The Early Development Instrument: A population-based measure for communities. A handbook on development, properties, and use. Hamilton, ON: Offord Centre for Child Studies.Google Scholar
  43. Janus, M., Harren, T., & Duku, E. (2004). Neighbourhood perspective on school readiness in kindergarten, academic testing in Grade 3, and affluence levels. Paper presented at the McMaster University Psychiatry Research Day, Hamilton, ON.Google Scholar
  44. Janus, M., & Offord, D. (2000). Readiness to learn at school. ISUMA Canadian Journal of Policy Research, 1, 71–75.Google Scholar
  45. Janus, M., & Offord, D. (2007). Development and psychometric properties of the Early Development Instrument (EDI): A measure of children’s school readiness. Canadian Journal of Behavioural Science, 39, 1–22.Google Scholar
  46. Janus, M., Offord, D., & Walsh, C. (2001). Population-level assessment of readiness to learn at school for 5-year-olds in Canada: Relation to child and parent measures. Presented at the Society for Research on Child Development conference, Minneapolis.Google Scholar
  47. Janus, M., Walsh, C., & Duku, E. (2005). Early development instrument: Factor structure, sub-domains and Multiple Challenge Index. Department of Psychiatry and Biobehavioural Sciences, McMaster University, Annual Research Day.Google Scholar
  48. Janus, M., Walsh, C., Viveiros, H., Duku, E., & Offord, D. (2003). School readiness to learn and neighbourhood characteristics. Poster presented at the Biennial meeting of the Society for Research on Child Development, Tampa, FL.Google Scholar
  49. Janus, M., Willms, J. D., & Offord, D. R. (2000). Psychometric properties of the Early Development Instrument (EDI): A teacher-completed measure of children’s readiness to learn at school entry. Unpublished manuscript.Google Scholar
  50. Kane, M. T. (2006). Validation. In R. L. Brennan (Ed.), Educational measurement (4th ed., pp. 17–64). Westport, CT: American Council on Education/Praeger.Google Scholar
  51. Keating, D. P. (2007). Formative evaluation of the Early Development Instrument: Progress and prospects. Early Education and Development, 18(3), 561–570.CrossRefGoogle Scholar
  52. Keating, D. P., & Hertzman, C. (Eds.). (1999). Developmental health and the wealth of nations. New York: Guildford.Google Scholar
  53. Kershaw, P., & Forer, B. (2006). What are the social and economic indicators of nurturing neighborhoods? Poster presented at the American Educational Research Association conference, San Francisco, California.Google Scholar
  54. Kershaw, P., Irwin, L., Trafford, K., & Hertzman, C. (2005). The British Columbia atlas of child development. Vancouver, BC: Human Early Learning Partnership.Google Scholar
  55. Kim, K. (2004). An additional view of conducting multi-level construct validation. In F. J. Yammarino & F. Dansereau (Eds.), Multi-level issues in organizational behaviour and processes (pp. 317–333). The Netherlands: Elsevier.CrossRefGoogle Scholar
  56. Kim, J., & Suen, H. K. (2003). Predicting children’s academic achievement from early assessment scores: A validity generalization study. Early Childhood Research Quarterly, 18, 547–566.CrossRefGoogle Scholar
  57. Klein, K. J., Bliese, P. D., Kozlowski, S. W. J., Dansereau, F., Gavin, M. B., Griffin, M. A., et al. (2000). Multilevel analytic techniques: Commonalities, differences, and continuing questions. In K. J. Klein & S. W. J. Kozlowski (Eds.), Multilevel theory, research, and methods in organizations (pp. 512–553). San Francisco: Jossey-Bass.Google Scholar
  58. Klein, K. J., Dansereau, F., & Hall, R. J. (1994). Levels issues in theory development, data collection, and analysis. Academy of Management Review, 19, 195–229.Google Scholar
  59. Kreiger, J. (2003). Class size reduction: Implementation and solutions. Paper presented at the SERVE Research and Policy Class Size Symposium, Raleigh, SC.Google Scholar
  60. LaPointe, V. R. (2006). Conceptualizing and examining the impact of neighbourhoods on the school readiness of kindergarten children in British Columbia. Unpublished Doctoral Dissertation, Vancouver, The University of British Columbia.Google Scholar
  61. LaPointe, V. R., Ford, L., & Zumbo, B. D. (2007). Examining the relationship between neighbourhood environment and school readiness for kindergarten children. Early Education and Development, 18, 473–496.CrossRefGoogle Scholar
  62. Linn, R. L. (2009). The concept of validity in the context of NCLB. In R. W. Lissitz (Ed.), The concept of validity: Revisions, new directions and applications (pp. 195–212). Charlotte, NC: IAP-Information Age Publishing, Inc.Google Scholar
  63. Lissitz, R. W. (2009). The concept of validity: Revisions, new directions and applications. Charlotte, NC.: IAP-Information Age Publishing, Inc.Google Scholar
  64. Lloyd, J. E. V., & Hertzman, C. (2009). From Kindergarten readiness to fourth-grade assessment: Longitudinal analysis with linked population data. Social Science and Medicine, 68, 111–123.CrossRefGoogle Scholar
  65. Mashburn, A. J., Hamre, B. K., Downer, J. T., & Pianta, R. C. (2006). Teacher and classroom characteristics associated with teachers’ ratings of prekindergartners’ relationships and behaviors. Journal of Psychoeducational Assessment, 24, 367–380.CrossRefGoogle Scholar
  66. Meisels, S. J. (1997). Using Work Sampling in authentic performance assessments. Educational Leadership, 54, 60–65. Google Scholar
  67. Meisels, S. J. (1999). Assessing readiness. In R. C. Pianta & M. J. Cox (Eds.), The transition to kindergarten (pp. 39–66). Baltimore, MD: Paul H. Brookes.Google Scholar
  68. Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 13–103). New York: Macmillan.Google Scholar
  69. Muthen, B. O. (1994). Multilevel covariance structure analysis. Sociological Methods and Research, 22, 276–398.CrossRefGoogle Scholar
  70. Muthen, L. K., & Muthen, B. O. (2006). Mplus User’s Guide, version 4. Los Angeles: Muthen and Muthen.Google Scholar
  71. National Research Council Institute of Medicine. (2000). From neurons to neighbourhoods: The science of early childhood development. In J. P. Shonkoff & D. A. Phillips (Eds.), Committee on integrating the science of early childhood development (pp. 328–336). Washington, D.C.: National Academy Press.Google Scholar
  72. O’Brien, R. M. (1990). Estimating the reliability of aggregate-level variables based on individual-level characteristics. Sociological Methods and Research, 18, 473–504.CrossRefGoogle Scholar
  73. Ostroff, C. (1992). The relationship between satisfaction, attitudes and performance: An organizational-level analysis. Journal of Applied Psychology, 77, 963–974.CrossRefGoogle Scholar
  74. Pedder, D. (2006). Are small classes better? Understanding relationships between class size, classroom processes, and pupils’ learning. Oxford Review of Education, 32, 213–234.CrossRefGoogle Scholar
  75. Pellegrini, A., & Blatchford, P. (2000). The child at school: Interactions with peers and teachers. London: Edward Arnold.Google Scholar
  76. Province of British Columbia (2008). Strategic plan 2008/092010/11. Retrieved from http://www.bcbudget.gov.bc.ca/2008/stplan/2008_Strategic_Plan.pdf on September 18, 2008.
  77. Rimm-Kaufman, S. E., & Pianta, R. C. (2000). An ecological perspective on the transition to kindergarten: A theoretical framework to guide empirical research. Journal of Applied Developmental Psychology, 21, 491–511.CrossRefGoogle Scholar
  78. Robinson, W. S. (1950). Ecological correlations and the behavior of individuals. American Sociological Review, 15, 351–357.CrossRefGoogle Scholar
  79. Rowan, B., Raudenbush, S. W., & Kang, S. K. (1991). School climate in secondary schools. In S. W. Raudenbush & J. D. Willms (Eds.), Schools, classrooms, and pupils: International studies of schooling from a multilevel perspective (pp. 203–223). San Diego: Academic.Google Scholar
  80. Satorra, A., & Bentler, P. M. (1999). A scaled difference Chi-square test statistic for moment structure analysis. Technical Report, University of California, Los Angeles. http://preprints.stat.ucla.edu/260/chisquare.pdf.
  81. Stevens, J. P. (2009). Applied multivariate statistics for the social sciences (5th ed.). New York: Routledge.Google Scholar
  82. Vygotsky, L. S. (1978). Mind in society: The development of higher psychological processes. M. Cole, V. John-Steiner, S. Scribner, & E. Souberman (Eds.), Cambridge, MA: Harvard University Press.Google Scholar
  83. Zumbo, B. D. (2007). Validity: Foundational issues and statistical methodology. In C. R. Rao & S. Sinharay (Eds.), Handbook of Statistics, Vol. 26: Psychometrics (pp. 45–79). Elsevier Science B.V.: The Netherlands.Google Scholar
  84. Zumbo, B. D. (2009). Validity as contextualized and pragmatic explanation, and its implications for validation practice. In R. W. Lissitz (Ed.), The concept of validity: Revisions, new directions and applications (pp. 65–82). Charlotte, NC: IAP-Information Age Publishing, Inc.Google Scholar
  85. Zumbo, B. D. & Forer, B. (2011). Testing and measurement from a multilevel view: Psychometrics and validation. In J. Bovaird, K. Geisinger, & C. Buckendahl (Eds.). High stakes testing in educationscience and practice in K-12 settings (Festschrift to Barbara Plake). American Psychological Press: Washington, D.C.Google Scholar

Copyright information

© Springer Science+Business Media B.V. 2011

Authors and Affiliations

  1. 1.Human Early Learning Partnership (HELP)University of British ColumbiaVancouverCanada
  2. 2.Department of Educational Psychology and Special EducationUniversity of British ColumbiaVancouverCanada

Personalised recommendations