Skip to main content
SpringerLink
Log in

Additional Evidence Based on the Internal Structure of the Instrument

  • Chapter
  • First Online:
Instrument Development in the Affective Domain

  • 2642 Accesses

Abstract

In many ways, Chap. 5 is an extension of the methods and techniques used to gather evidence based on the internal structure of the instrument and continues to describe the latent variable approach to instrument development. In this chapter, we introduce two common instrument development situations where discrete mathematical structures are encountered: latent class analysis (LCA) and item response theory (IRT). In LCA, both the latent variable and the indicator variable are best represented by a categorical structure. In IRT, the latent trait is continuous, but the items are the items are categorical. IRT has historically been used in educational achievement applications, however, it is gaining popularity in some affective characteristic measurement scenarios. The last section of the chapter is devoted to the topic of measurement invariance and the analytic techniques that allow instrument developers to explore whether the scale functions in the same manner (exhibits the same internal structure) across subgroups. If the items that reflect their latent constructs and the connections between the constructs operate in a fundamentally different way depending on group membership, cross group comparisons become difficult, if not impossible. This chapter discusses the most common statistical techniques (along with examples) for establishing invariance for both scenarios where the indicators are either continuous or categorical.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 149.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 199.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 199.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    Material from Gable et al. (2011) included with permission from Sage Publications.

References

  • Akaike, H. (1987). Factor analysis and AIC. Psychometrika, 52(3), 317–332.

    Article  Google Scholar 

  • American Educational Research Association (AERA), American Psychological Association (APA), & National Council on Measurement in Education (NCME). (1999). The standards for educational and psychological testing. Washington, DC: American Educational Research Association.

    Google Scholar 

  • Andrich, D. (1978a). Application of a psychometric rating model to ordered categories which are scored with successive integers. Applied Psychological Measurement, 2, 581–594.

    Article  Google Scholar 

  • Andrich, D. (1978b). Rating formulation for ordered response categories. Psychometrika, 43, 561–573.

    Article  Google Scholar 

  • Andrich, D. (1978c). Scaling attitude items constructed and scored in the Likert tradition. Educational and Psychological Measurement, 38, 665–680.

    Article  Google Scholar 

  • Beck, C. T., & Gable, R. K. (2001). Further validation of the postpartum depression screening scale. Nursing Research, 50, 155–164.

    Article  PubMed  Google Scholar 

  • Boscardin, C. (2012). Profiling students for remediation using latent class analysis. Advances in Health Sciences, 17, 56–63.

    Google Scholar 

  • Boscardin, C. K., Muthen, B., Francis, D. J., & Baker, E. L. (2008). Early identification of reading difficulties using heterogeneous developmental trajectories. Journal of Educational Psychology, 100(1), 192–208.

    Article  Google Scholar 

  • Bozdogan, H. (1987). Model selection and Akaike’s information criteria (AIC): The general theory and its analytical extensions. Psychometrika, 52(3), 345–370.

    Article  Google Scholar 

  • Brown, T. A. (2006). Confirmatory factor analysis for applied research. New York: The Guilford Press.

    Google Scholar 

  • Collins, L. M., & Lanza, S. T. (2010). Latent class and latent transition analysis with applications in the social, behavioral, and health sciences. Hoboken, NJ: Wiley.

    Google Scholar 

  • Dias, J. G., & Vermunt, J. K. (2006). Bootstrap methods for measuring classification uncertainty in latent class analysis. In A. Rizzi & M. Vichi (Eds.), Proceedings in computational statistics (pp. 31–41). Heidelberg: Springer.

    Google Scholar 

  • Dimitrov, D. M. (2010). Testing factorial invariance in the context of construct validation. Measurement and Evaluation in Counseling and Development, 43(2), 121–149.

    Article  Google Scholar 

  • Embretson, S. E. (1999). Issues in the measurement of cognitive abilities. In S. Embretson & S. Hershberger (Eds.), The new rules of measurement: What every psychologist and educator should know (pp. 1–15). Mahwah, NJ: Lawrence Erlbaum Associates.

    Google Scholar 

  • French, B. F., & Finch, W. H. (2008). Multigroup confirmatory factor analysis: Locating the invariant reference sets. Structural Equation Modeling: A Multidisciplinary Journal, 15, 96–113.

    Article  Google Scholar 

  • Finch, W. H., & Bronk, K. C. (2011). Conducting confirmatory latent class analysis using Mplus. Structural Equation Modeling: A Multidisciplinary Journal, 18(1), 132–151.

    Article  Google Scholar 

  • Franek, M. (2005/2006). Foiling cyberbullies in the new Wild West. Educational Leadership, 63, 39–43.

    Google Scholar 

  • Gable, R. K., Ludlow, L. H., Kite, S. L., McCoach, D. B., & Filippelli, L. P. (2009, April). Development and validation of the survey of knowledge of internet risk and internet behavior. Paper presented at the annual meeting of the American Educational Research Association, San Diego, CA.

    Google Scholar 

  • Gable, R. K., Ludlow, L. H., McCoach, D. B., & Kite, S. L. (2010, October). Construct invariance of the survey of knowledge of internet risk and internet behavior. Paper presented at the Annual Conference of the Northeastern Educational Research Association, Rocky Hill, CT.

    Google Scholar 

  • Gable, R. K., Ludlow, L. H., McCoach, D. B., & Kite, S. L. (2011). Validation of the survey of knowledge of internet risk and internet behavior. Educational and Psychological Measurement, 71(1), 217–230.

    Article  Google Scholar 

  • Gable, R. K., Ludlow, L. H., & Wolf, M. B. (1990). The use of classical and Rasch latent trait models to enhance the validity of affective measures. Educational and Psychological Measurement, 50(4), 869–878.

    Article  Google Scholar 

  • Gable, R. K., & Wolf, M. B. (1993). Instrument development in the affective domain: Measuring attitudes and values in corporate and school settings (2nd ed.). Boston: Kluwer-Nijhoff.

    Book  Google Scholar 

  • Hadzi-Pavlovic, D. (2009). Finding patterns and groupings: I. Introduction to latent class analysis. Acta Neuropsychiatrica, 21(6), 312–313.

    Article  Google Scholar 

  • Hagenaars, J. A., & McCutcheon, A. L. (2002). Applied latent class analysis. Cambridge: Cambridge University Press.

    Book  Google Scholar 

  • Hambleton, R. K., Swaminathan, H., & Rogers, H. J. (1991). Fundamentals of item response theory. Newbury Park, CA: Sage.

    Google Scholar 

  • Hattie, J., Jaeger, R. M., & Bond, L. (1999). Persistent methodological questions in educational testing. Review of Research in Education, 24, 393–446.

    Google Scholar 

  • Helms, B. J., & Gable, R. K. (1989). School situation survey manual. Palo Alto: Consulting Psychologists Press/The Mind Garden.

    Google Scholar 

  • Holland, W., & Wainer, H. (1993). Differential item functioning. Mahwah, NJ: Lawrence Erlbaum Associates.

    Google Scholar 

  • Henson, J. M., Reise, S. P., & Kim, K. H. (2007). Detecting mixtures from structural model differences using latent variable mixture modeling: A comparison of relative model fit statistics. Structural Equation Modeling: A Multidisciplinary Journal, 14(2), 202–226.

    Article  Google Scholar 

  • Johnson, W. J., Dixon, P. N., & Ryan, J. M. (1991, April). Factorial and Rasch analysis of the Charles F. Kettering Ltd. school climate profile. Paper presented at the Annual Meeting of the American Educational Research Association, Chicago, IL.

    Google Scholar 

  • Kite, S. L., Gable, R. K., & Filippelli, L. P. (2010a). Assessment of students’ knowledge of Internet risk and Internet behaviors: Potential threat to bullying and contact by InternetpPredators. Paper presented at the Annual Meeting of the Northeastern Educational Research Association, Rocky Hill, CT.

    Google Scholar 

  • Kite, S. L., Gable, R. K., & Filippelli, L. (2010b). Assessing middle school students’ knowledge of conduct/consequences and their behaviors regarding the use of social networking sites. The Clearing House, 1939-912X, 83, 158–163.

    Google Scholar 

  • Lin, T. H., & Dayton, C. M. (1997). Model selection information criteria for non-nested latent class models. Journal of Educational and Behavioral Statistics, 22(3), 249–264.

    Google Scholar 

  • Lo, Y., Mendell, N. R., & Rubin, D. B. (2001). Testing the number of components in a normal mixture. Biometrika, 88(3), 767–778.

    Article  Google Scholar 

  • Ludlow, L. H., Enterline, S., & Cochran-Smith, M. (2008). Learning to teach for social justice-beliefs scale: An application of Rasch measurement principles. Measurement and Evaluation in Counseling and Development, 20, 194–214.

    Google Scholar 

  • Ludlow, L. H., & Haley, S. M. (1995). Rasch model logits: Interpretation, use, and transformation. Educational and Psychological Measurement, 55, 967–975.

    Article  Google Scholar 

  • Magidson, J., & Vermunt, J. K. (2004). Latent class models. In D. Kaplan (Ed.), The Sage handbook of quantitative methodology for the social sciences (pp. 175–198). Thousand Oaks, CA: Sage.

    Google Scholar 

  • Masters, G. N. (1980). A Rasch model for rating scales (Unpublished doctoral dissertation). Chicago, IL: University of Chicago.

    Google Scholar 

  • Masters, G. N., & Hyde, N. H. (1984). Measuring attitude to school with a latent trait model. Applied Psychological Measurement, 8(1), 39–48.

    Article  Google Scholar 

  • McCoach, D. B. (2002). A validity study of the school attitude assessment survey (SAAS). Measurement and Evaluation in Counseling and Development, 35, 66–77.

    Google Scholar 

  • McCoach, D. B., & Siegle, D. (2003a). The SAAS-R: A new instrument to identify academically able students who underachieve. Educational and Psychological Measurement, 63, 414–429.

    Article  Google Scholar 

  • McCoach, D. B., & Siegle, D. (2003b). The structure and function of academic self-concept in gifted and general education samples. Roeper Review, 25, 61–65.

    Article  Google Scholar 

  • McCutcheon, A. L. (1987). Latent class analysis. Newbury Park, CA: Sage.

    Google Scholar 

  • McKenna, P. (2007). The rise of cyberbullying. New Scientist, 195(2613), 60.

    Article  Google Scholar 

  • Muthén, B., & Muthén, L. K. (2000). Integrating person-centered and variable-centered analyses: Growth mixture modeling with latent trajectory classes. Alcoholism: Clinical and Experimental Research, 24(6), 882–891.

    Google Scholar 

  • Muthen, L. K., & Muthen, B. O. (2010). MPLUS user’s guide (6th ed.). Los Angeles, CA: Muthen & Muthen.

    Google Scholar 

  • Muthén, B. O., & Muthén, L. K. (2000). The development of heavy drinking and alcohol-related problems from ages 18 to 37 in a U.S. national sample. Journal of Studies on Alcohol, 61, 290–300.

    PubMed  Google Scholar 

  • Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory (3rd ed.). New York: McGraw-Hill.

    Google Scholar 

  • Nylund, K. L., Asparouhov, T., & Muthen, B. O. (2007). Deciding on the number of classes in latent class analysis and growth mixture modeling: A Monte Carlo simulation study. Structural Equation Modeling: A Multidisciplinary Journal, 14(4), 535–569.

    Article  Google Scholar 

  • Rasch, G. (1960). Probabilistic models for some intelligence and attainment tests. Copenhagen: Danmarks Paedogogiske Institute.

    Google Scholar 

  • Rasch, G. (1966). An item analysis which takes individual differences into account. British Journal of Mathematical and Statistical Psychology, 19, 49–57.

    Article  PubMed  Google Scholar 

  • Rensvold, R. B., & Cheung, G. W. (2001). Testing for metric invariance using structural equation models: Solving the standardization problem. In C. A. Schriesheim & L. L. Neider (Eds.), Research in management: Equivalence in measurement (pp. 25–50). Greenwich, CT: Information Age Publishing.

    Google Scholar 

  • Sass, D. A. (2011). Testing measurement invariance and comparing latent factor means within a confirmatory factor analysis framework. Journal of Psychoeducational Assessment, 29(4), 347–363.

    Article  Google Scholar 

  • Schwartz, S. A. (1978). A comprehensive system for item analysis in psychological scale construction. Journal of Educational Measurement, 15, 117–123.

    Article  Google Scholar 

  • Sclove, S. L. (1987). Application of model-selection criteria to some problems in multivariate analysis [Special section]. Psychometrika, 52(3), 333–343.

    Article  Google Scholar 

  • Tormakangas, K. (2011). Advantages of the Rasch measurement model in analyzing educational tests: An applicator’s reflection. Educational Research and Evaluation: An International Journal on Theory and Practice, 17(5), 307–320.

    Article  Google Scholar 

  • Tovar, E., & Simon, M. A. (2010). Factorial structure and invariance analysis in sense of belonging scales. Measurement and Evaluation in Counseling and Development, 43(3), 199–217.

    Article  Google Scholar 

  • Vermunt, J. K., & Magidson, J. (2002). Latent class cluster analysis. In J. A. Hagenaars & A. L. McCutcheon (Eds.), Applied latent class analysis (pp. 89–106). Cambridge: Cambridge University Press.

    Chapter  Google Scholar 

  • Vermunt, J. K., & Magidson, J. (2004). Latent class analysis. In M. S. Lewis-Beck, A. Bryman, & T. F. Liao (Eds.), The sage encyclopedia of social sciences research methods (pp. 549–553). Thousand Oakes, CA: Sage Publications.

    Google Scholar 

  • Vuong, Q. H. (1989). Likelihood ratio tests for model selection and non-nested hypotheses. Econometrica, 57(2), 307–333.

    Article  Google Scholar 

  • Weiss, D. J., & Yoes, M. E. (1991). Item response theory. In R. Hambleton & J. Zaal (Eds.), Advances in educational and psychological testing: Theory and applications (pp. 69–95). New York, NY: Kluwer Academic/Plenum Publishers.

    Google Scholar 

  • Wilson, M. (2005). Constructing measures: An item response modeling approach. New York, NY: Taylor & Francis.

    Google Scholar 

  • Wirth, R. J., & Edwards, M. C. (2007). Item factor analysis: Current approaches and future directions. Psychological Methods, 12(1), 58–79.

    Article  PubMed  Google Scholar 

  • Wright, B. D., & Linacre, M. (1998). Winsteps. Chicago, IL: Mesa Press.

    Google Scholar 

  • Wright, B. D., & Masters, G. N. (1982). Rating scale analysis. Chicago, IL: Mesa Press.

    Google Scholar 

  • Wright, B. D., & Stone, M. H. (1979). Best test design. Chicago, IL: Mesa Press.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Educational Psychology Department, University of Connecticut, Glenbrook Road, Unit 3064 249, Storrs, CT, 06269-3064, USA

    D. Betsy McCoach

  2. Alan Shawn Feinstein Graduate School, Johnson and Wales University, Quail Run Rd. 26, Storrs, CT, 06268, USA

    Robert K. Gable

  3. Department of Educational Psychology, University of Connecticut, Glenbrook Road, Unit 2064 249, Storrs, CT, 06269-2064, USA

    John P. Madura

Authors
  1. D. Betsy McCoach
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Robert K. Gable
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. John P. Madura
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to D. Betsy McCoach .

Rights and permissions

Reprints and permissions

Copyright information

© 2013 Springer Science+Business Media New York

About this chapter

Cite this chapter

McCoach, D.B., Gable, R.K., Madura, J.P. (2013). Additional Evidence Based on the Internal Structure of the Instrument. In: Instrument Development in the Affective Domain. Springer, New York, NY. https://doi.org/10.1007/978-1-4614-7135-6_5

Download citation

Publish with us

Policies and ethics

Search

Navigation