Skip to main content
SpringerLink
Log in

Validation and invariance of the Conceptions of Assessment-III Abridged (COA-IIIA) among pre-service and in-service teachers in the United States

  • Published:
Educational Assessment, Evaluation and Accountability Aims and scope Submit manuscript

Abstract

As beliefs are well-known antecedents of teachers’ practices, including assessment practices, sound measurement of teacher beliefs is critical for scholarly research as well as practical purposes. The present study examined the validity of inferences derived from the Conceptions of Assessment III—Abridged (COA-IIIA) instrument with US PK-12 pre-service (N = 554) and in-service (N = 341) teachers. Prior research on the COA-IIIA in diverse contexts and with different teacher populations has yielded highly divergent findings concerning this instrument’s internal score structure. The study relies on confirmatory factor analyses to investigate the COA-IIIA’s score structure among US pre-service and in-service teachers and to test the measurement invariance of select COA-IIIA measurement models across these groups. Findings indicate poor fit of most prior COA-IIIA measurement models with US pre-service and in-service teachers, though we were able to specify several simplified models that were at least metrically invariant across these populations.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

Data availability

The data that support the findings of this study are available from the authors upon reasonable request.

Notes

  1. Our review of the literature focuses only on peer-reviewed studies in which the COA-IIIA was unmodified in terms of item content or response scale (to the extent we were able to verify this). However, some studies reviewed did translate the language of the items.

  2. This included post-baccalaureate programs and combined undergraduate and graduate teacher education programs.

  3. Of the PST sample, 0.92% selected “other” for their intended school level.

  4. School level taught was indeterminable for 1.44% of IST participants with a non-missing response for this variable.

  5. Missing data for PST demographic and professional variables constituted between 0.18 and 72.02% of PST cases and between 0.29 and 18.48% of IST cases. Some of this is due to inconsistencies across studies and datasets in available non-COA-IIIA data. Some of these estimates should be considered very unreliable.

  6. As noted by a reviewer, future studies of the COA-IIIA may consider bifactor modeling approaches, which have enjoyed some use in prior research on this instrument (Brown & Remesal, 2017).

References

  • Barnes, N., Fives, H., & Dacey, C. M. (2015). Teachers’ beliefs about assessment. In H. Fives & M. G. Gill (Eds.), International handbook of research on teachers’ beliefs (pp. 284–300). Routledge.

    Google Scholar 

  • Barnes, N., Fives, H., & Dacey, C. M. (2017). US teachers’ conceptions of the purposes of assessment. Teaching and Teacher Education, 65, 107–116.

    Article  Google Scholar 

  • Bentler, P. M. (1990). Comparative fit indexes in structural models. Psychological Bulletin, 107, 238–246.

    Article  Google Scholar 

  • Bentler, P. M., & Bonett, D. G. (1980). Significance tests and goodness of fit in the analysis of covariance structures. Psychological Bulletin, 88, 588–606.

    Article  Google Scholar 

  • Brosseau-Liard, P. E., & Savalei, V. (2014). Adjusting incremental fit indices for nonnormality. Multivariate Behavioral Research, 49(5), 460–470.

    Article  Google Scholar 

  • Brosseau-Liard, P. E., Savalei, V., & Li, L. (2012). An investigation of the sample performance of two nonnormality corrections for RMSEA. Multivariate Behavioral Research, 47(6), 904–930.

    Article  Google Scholar 

  • Brown, G. T. L. (2004). Teachers’ conceptions of assessment: Implications for policy and professional development. Assessment in Education: Principles, Policy, & Practice, 11(3), 301–318.

    Google Scholar 

  • Brown, G. T. L. (2006). Teachers’ conceptions of assessment: Validation of an abridged version. Psychological Reports, 99, 166–170.

    Article  Google Scholar 

  • Brown, G. T. L. (2011). Teachers’ conceptions of assessment: Comparing primary and secondary teachers in New Zealand. Assessment Matters, 3, 45–70.

    Article  Google Scholar 

  • Brown, G. T. L., & Michaelides, M. P. (2011). Ecological rationality in teachers’ conceptions of assessment across samples from Cyprus and New Zealand. European Journal of Psychology of Education, 26(3), 319–337.

    Article  Google Scholar 

  • Brown, G. T. L., & Remesal, A. (2012). Prospective teachers’ conceptions of assessment: A cross-cultural comparison. The Spanish Journal of Psychology, 15, 75–89.

    Article  Google Scholar 

  • Brown, G. T., & Remesal, A. (2017). Teachers’ conceptions of assessment: Comparing two inventories with Ecuadorian teachers. Studies in Educational Evaluation, 55, 68–74.

    Article  Google Scholar 

  • Brown, G. T. L., Kennedy, K. J., Fok, P. K., Chan, J. K. S., & Yu, W. M. (2009). Assessment for student improvement: Understanding Hong Kong teachers’ conceptions and practices of assessment. Assessment in Education: Principles, Policy & Practice, 16, 347–363.

    Google Scholar 

  • Brown, G. T. L., Lake, R., & Matters, G. (2011). Queensland teachers’ conceptions of assessment: The impact of policy priorities on teacher attitudes. Teaching and Teacher Education, 27, 210–220.

    Article  Google Scholar 

  • Brown, G. T. L., Chaudhry, H., & Dhamija, R. (2015). The impact of an assessment policy upon teachers’ self-reported assessment beliefs and practices: A quasi-experimental study of Indian teachers in private schools. International Journal of Educational Research, 71, 50–64.

    Article  Google Scholar 

  • Brown, G. T. L., Gebril, A., & Michaelides, M. P. (2019). Teachers’ conceptions of assessment: A global phenomenon or a global localism. Frontiers in Education, 4(16). https://doi.org/10.3389/feduc.2019.00016

  • Cheung, G. W., & Rensvold, R. B. (2002). Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling, 9(2), 233–255.

    Article  Google Scholar 

  • Clark, C., & Peterson, P. (1986). Teachers’ thought processes. In M. Wittrock (Ed.), Handbook of research on teaching (pp. 255–296). Macmillan.

    Google Scholar 

  • Daniels, L. M., & Poth, C. A. (2017). Relationships between pre-service teachers’ conceptions of assessment, approaches to instruction, and assessment: An achievement goal theory perspective. Educational Psychology, 37(7), 835–853.

    Article  Google Scholar 

  • Daniels, L. M., Poth, C., Papile, C., & Hutchison, M. (2014). Validating the Conceptions of Assessment-III Scale in Canadian preservice teachers. Educational Assessment, 19(2), 139–158.

    Article  Google Scholar 

  • Datnow, A., & Hubbard, L. (2015). Teacher capacity for and beliefs about data-driven decision making: A literature review of international research. Journal of Educational Change, 17(1), 7–28.

    Article  Google Scholar 

  • Dimitrov, D. M. (2010). Testing for factorial invariance in the context of construct validation. Measurement and Evaluation in Counseling and Development, 43(2), 121–149.

    Article  Google Scholar 

  • Fan, X., & Sivo, S. A. (2007). Sensitivity of fit indices to model misspecification and model types. Multivariate Behavioral Research, 42(3), 509–529.

    Article  Google Scholar 

  • Gebril, A., & Brown, G. T. L. (2014). The effect of high-stakes examination systems on teacher beliefs: Egyptian teachers’ conceptions of assessment. Assessment in Education: Principles, Policy & Practice, 21(1), 16–33.

    Google Scholar 

  • Hammer, J. H. (2016). Construct replicability calculator: A Microsoft Excel-based tool to calculate the Hancock and Mueller (2001) H index. Available from https://DrJosephHammer.com/. Accessed 13 Mar 2023.

  • Hancock, G. R., & Mueller, R. O. (2001). Rethinking construct reliability within latent variable systems. In R. Cudeck, S. du Toit, & D. Sorbom (Eds.) Structural equation modeling: Present and future—A Festschrift in honor of Karl Joreskog (pp. 195–216). Lincolnwood, IL: Scientific Software International.

  • Harris, L. R., & Brown, G. T. L. (2009). The complexity of teachers’ conceptions of assessment: Tensions between the needs of schools and students. Assessment in Education: Principles, Policy & Practice, 16(3), 365–381.

    Google Scholar 

  • Hu, L. T., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1–55.

    Article  Google Scholar 

  • Jung, E., & Yoon, M. (2016). Comparisons of three empirical methods for partial factorial invariance: Forward, backward, and factor-ratio tests. Structural Equation Modeling: A Multidisciplinary Journal, 23(4), 567–584.

    Article  Google Scholar 

  • Kerr, K. A., Marsh, J. A., Ikemoto, G. S., Darilek, H., & Barney, H. (2006). Strategies to promote data use for instructional improvement: Actions, outcomes, and lessons from three urban districts. American Journal of Education, 112(4), 496–520.

    Article  Google Scholar 

  • Kraner, C., & Reeves, T. D. (2018, October). Conceptions of Assessment-III Abridged (COA-IIIA) score reliability and validity among pre-service teachers in the United States. Paper presented at the meeting of the Mid-Western Educational Research Association, Cincinnati, OH.

  • Leighton, J. P., Gokiert, R. J., Cor, M. K., & Heffernan, C. (2010). Teacher beliefs about the cognitive diagnostic information of classroom-versus large-scale tests: Implications for assessment literacy. Assessment in Education: Principles, Policy & Practice, 17(1), 7–21.

    Google Scholar 

  • Little, R. J. (1988). A test of missing completely at random for multivariate data with missing values. Journal of the American Statistical Association, 83, 1098–1202.

    Article  Google Scholar 

  • MacCallum, R. C., Widaman, K. F., Zhang, S., & Hong, S. (1999). Sample size in factor analysis. Psychological Methods, 4, 84–99.

    Article  Google Scholar 

  • Mardia, K. V. (1970). Measures of multivariate skewness and kurtosis with applications. Biometrika, 57, 519–530.

    Article  Google Scholar 

  • Marsh, J. A. (2012). Interventions promoting educators’ use of data: Research insights and gaps. Teachers College Record, 114(11), 1–48.

    Article  Google Scholar 

  • McDonald, R. P. (2013) Test theory: A unified treatment. Mahwah, NJ: Lawrence Earlbaum Associates.

  • Meade, A. W., & Bauer, D. J. (2007). Power and precision in confirmatory factor analytic tests of measurement invariance. Structural Equation Modeling, 14(4), 611–635.

    Article  Google Scholar 

  • Meredith, W. (1993). Measurement invariance, factor analysis and factorial invariance. Psychometrika, 58(4), 525–543.

    Article  Google Scholar 

  • Mullis, I. V., Martin, M. O., Minnich, C. A., Stanco, G. M., Arora, A., Centurino, V. A., & Castle, C. E. (2012). TIMSS 2011 encyclopedia: Education policy and curriculum in mathematics and science. International Association for the Evaluation of Educational Achievement.

    Google Scholar 

  • Olsson, U. H., Foss, T., Troye, S. V., & Howell, R. D. (2000). The performance of ML, GLS, and WLS estimation in structural equation modeling under conditions of misspecification and nonnormality. Structural Equation Modeling, 7(4), 557–595.

    Article  Google Scholar 

  • Pajares, M. F. (1992). Teachers’ beliefs and educational research: Cleaning up a messy construct. Review of Educational Research, 62(3), 307–332.

    Article  Google Scholar 

  • Pellegrino, J. W., Chudowsky, N., & Glaser, R. (Eds.). (2001). Knowing what students know: The science and design of educational assessment. National Academies Press.

    Google Scholar 

  • Prenger, R., & Schildkamp, K. (2018). Data-based decision making for teacher and student learning: A psychological perspective on the role of the teacher. Educational Psychology, 38(6), 734–752.

    Article  Google Scholar 

  • Reeves, T. D., Summers, K. H., & Grove, E. (2016). Examining the landscape of teacher learning for data use: The case of Illinois. Cogent Education, 3(1).

  • Rosseel, Y. (2012). lavaan: An R package for structural equation modeling. Journal of Statistical Software, 48(2), 1–36.

    Article  Google Scholar 

  • Ruiz-Primo, M. A., Shavelson, R. J., Hamilton, L., & Klein, S. (2002). On the evaluation of systemic science education reform: Searching for instructional sensitivity. Journal of Research in Science Teaching, 39(5), 369–393.

    Article  Google Scholar 

  • Satorra, A., & Bentler, P. M. (2001). A scaled difference chi-square test statistic for moment structure analysis. Psychometrika, 66(4), 507–514.

    Article  Google Scholar 

  • Segers, M., & Tillema, H. (2011). How do Dutch secondary teachers and students conceive the purpose of assessment? Studies in Educational Evaluation, 37(1), 49–54.

    Article  Google Scholar 

  • Steiger, J. H., & Lind, J. C. (1980, June). Statistically-based tests for the number of common factors. Paper presented at the Annual Spring Meeting of the Psychometric Society, Iowa City, IA.

  • Tschannen-Moran, M., Hoy, A. W., & Hoy, W. K. (1998). Teacher efficacy: Its meaning and measure. Review of Educational Research, 68(2), 202–248.

    Article  Google Scholar 

Download references

Acknowledgements

The authors thank all pre-service and in-service teachers who participated in the study, as well as Amy Martin, Evelyn Comber, David Walker, Kelly Summers, and Thomas Smith for their feedback provided on an earlier version of this manuscript. We also thank Gavin Brown for his swift and helpful assistance in specifying several of the tested models, and Mary Sanderson for the data analytic support.

Author information

Authors and Affiliations

  1. Northern Illinois University, DeKalb, IL, USA

    Todd D. Reeves, Yasemin Onder & Chris Kraner

Authors
  1. Todd D. Reeves
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yasemin Onder
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chris Kraner
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Todd D. Reeves.

Ethics declarations

Conflict of interest

The authors declare no competing interests.

Additional information

Publisher's note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Reeves, T.D., Onder, Y. & Kraner, C. Validation and invariance of the Conceptions of Assessment-III Abridged (COA-IIIA) among pre-service and in-service teachers in the United States. Educ Asse Eval Acc 35, 419–447 (2023). https://doi.org/10.1007/s11092-023-09407-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11092-023-09407-4

Keywords

Search

Navigation