Abstract
The aim of the current study is to introduce three assumptions common to psychometric theory and psychometric practice, and to show how alternatives to traditional psychometric approaches can be used to improve psychological measurement. These alternatives are developed by adapting each of these three assumptions. The assumption of structural validity relates to the implementation of mathematical models. The process assumption regards the underlying process generating the observed data. The construct assumption implies that the observed data on its own do not constitute a measurement, but the latent variable that originates the observed data. Nonparametric item response modeling and cognitive psychometric modeling are presented as alternatives for relaxing the first two assumptions, respectively. Network psychometrics is the alternative for relaxing the third assumption. Final remarks sum up the most important conclusions of the study.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data Availability
Data sharing not applicable to this article as no datasets were generated or analysed during the current study.
Materials Availability
Materials sharing not applicable to this article as the study consists of a theoretical/qualitative analysis of the field of psychometrics.
Code Availability
Code sharing not applicable to this article as no datasets were generated or analysed during the current study.
References
Allais M (1953) Le comportement de l'homme rationnel devant le risque: Critique des postulats et axiomes de l'école américaine [The rational man's behavior in the face of risk: Critical of postulates and axioms of the American school]. Econometrica: Journal of the Econometric Society, 503–546. https://doi.org/10.2307/1907921
Bacon, D. (2004). The contributions of reliability and pretests to effective assessment. Practical Assessment, Research & Evaluation, 9(3), 1–8. https://doi.org/10.7275/kbtm-zy59
Baddeley, A. D., & Hitch, G. (1974). Working memory. In G. A. Bower (Ed.), The psychology of learning and motivation (pp. 47–90). Academic Press.
Birnbaum, A. (1968). Some latent trait models and their use in inferring an examinee’s ability. In F. M. Lord & M. R. Novick (Eds.), Statistical theories of mental test scores (pp. 397–479). Addison-Wesley.
Bock, R. D. (1997). A brief history of item theory response. Educational Measurement: Issues and Practice, 16(4), 21–33. https://doi.org/10.1111/j.1745-3992.1997.tb00605.x
Borsboom, D. (2005). Measuring the mind: Conceptual issues in contemporary psychometrics. Cambridge University Press.
Borsboom, D. (2017). A network theory of mental disorders. World Psychiatry, 16(1), 5–13. https://doi.org/10.1002/wps.20375
Borsboom, D., Mellenbergh, G. J., & van Heerden, J. (2003). The theoretical status of latent variables. Psychological Review, 110(2), 203–219. https://doi.org/10.1037/0033-295X.110.2.203
Borsboom, D., Mellenbergh, G. J., & van Heerden, J. (2004). The concept of validity. Psychological Review, 111(4), 1061–1071. https://doi.org/10.1037/0033-295X.111.4.1061
Buck, A. J., & Lady, G. M. (2015). A new approach to model verification, falsification and selection. Econometrics, 3(3), 466–493. https://doi.org/10.3390/econometrics3030466
Buckhalt, J. A. (2002). A short history of g: Psychometrics’ most enduring and controversial construct. Learning and Individual Differences, 13(2), 101–114. https://doi.org/10.1016/S1041-6080(02)00074-2
Campbell, N. R. (1928). An account of the principles of measurement and calculation. Longsman, Green.
Canivez, G. L. (2016). Bifactor modeling in construct validation of multifactored tests: Implications for multidimensionality and test interpretation. In K. Schweizer & C. DiStefano (Eds.), Principles and methods of test construction: Standards and recent advancements (pp. 247–271). Hogrefe.
Canivez, G. L., & Watkins, M. W. (2010). Investigation of the factor structure of the Wechsler Adult Intelligence Scale—Fourth Edition (WAIS–IV): Exploratory and higher order factor analyses. Psychological Assessment, 22(4), 827–836. https://doi.org/10.1037/a0020429
Christensen, A. P., & Golino, H. (2021a). On the equivalency of factor and network loadings. Behavior Research Methods, 53, 1563–1580. https://doi.org/10.3758/s13428-020-01500-6
Christensen, A. P., & Golino, H. (2021b). Factor or network model? Predictions from neural networks. Journal of Behavioral Data Science, 1(1), 85–126. https://doi.org/10.35566/jbds/v1n1/p5
Constantini, G., Richetin, J., Preti, E., Casini, E., Epskamp, S., & Perugini, M. (2019). Stability and variability of personality networks. A tutorial on recent developments in network psychometrics. Personality and Individual Differences, 136, 68–78. https://doi.org/10.1016/j.paid.2017.06.011
Cook, T. D., Campbell, D. T., & Shadish, W. (2002). Experimental and quasi-experimental designs for generalized causal inference. Houghton Mifflin.
Coombs, C. H., Dawes, R. M., & Tversky, A. (1970). Mathematical psychology: An elementary introduction. Prentice-Hall.
Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52(4), 281–302. https://doi.org/10.1037/h0040957
Dey, D. K., Ghosh, S. K., & Mallick, B. K. (2000). Generalized linear models: A Bayesian perspective. CRC Press.
Edwards, D., & Hamson, M. (2007). Guide to mathematical modelling. Industrial Press.
Embretson, S. E. (2010). Measuring psychological constructs: Advances in model-based approaches. American Psychological Association.
Epskamp, S., & Fried, E. I. (2018). A tutorial on regularized partial correlation networks. Psychological Methods, 23(4), 617–634. https://doi.org/10.1037/met0000167
Epskamp, S., Rhemtulla, M., & Borsboom, D. (2017). Generalized network psychometrics: Combining network and latent variable models. Psychometrika, 82(4), 904–927. https://doi.org/10.1007/s11336-017-9557-x
Erdfelder, E., Auer, T. S., Hilbig, B. E., Aßfalg, A., Moshagen, M., & Nadarevic, L. (2009). Multinomial processing tree models: A review of the literature. Journal of Psychology, 217(3), 108–124. https://doi.org/10.1027/0044-3409.217.3.108
Faraway, J. J. (2016). Extending the linear model with R: Generalized linear, mixed effects and nonparametric regression models. CRC Press.
Fechner, G. T. (1860). Elemente der psychophysik [Elements of psychophysics]. Leipzig: Breitkopf & Hartel.
Ferraty, F., & Vieu, P. (2006). Nonparametric functional data analysis: Theory and practice. Springer.
Festinger, L. (1962). A theory of cognitive dissonance. Stanford University Press.
Fried, E. I. (2020). Lack of theory building and testing impedes progress in the factor and network literature. Psychological Inquiry, 31(4), 271–288. https://doi.org/10.1080/1047840X.2020.1853461
Furr, M. (2011). Scale construction and psychometrics for social and personality psychology. Sage.
Guest, O., & Martin, A. E. (2021). How computational modeling can force theory building in psychological science. Perspectives on Psychological Science, 16(4), 789–802. https://doi.org/10.1177/1745691620970585
Golino, H. F., & Demetriou, A. (2017). Estimating the dimensionality of intelligence like data using Exploratory Graph Analysis. Intelligence, 62, 54–70. https://doi.org/10.1016/j.intell.2017.02.007
Golino, H. F., & Epskamp, S. (2017). Exploratory graph analysis: A new approach for estimating the number of dimensions in psychological research. PLoS ONE, 12(6), 1–26. https://doi.org/10.1371/journal.pone.0174035
Gould, S. J. (1996). The mismeasure of man. Norton.
Griffith, D. A., & Akio, S. (1995). Trade-offs associated with normalizing constant computational simplifications for estimating spatial statistical models. Journal of Statistical Computation and Simulation, 51(2–4), 165–183. https://doi.org/10.1080/00949659508811630
Haslbeck, J. M. B., Ryan, O., Robinaugh, D. J., Waldorp, L. J., & Borsboom, D. (2021). Modeling psychopathology: From data models to formal theories. Advance online publication. https://doi.org/10.1037/met0000303
Hays, R. D., Morales, L. S., & Reise, S. P. (2000). Item response theory and health outcomes measurement in the 21st century. Medical Care, 38(9 Suppl), 1128–1142.
Heathcote, A., Brown, S., & Mewhort, D. J. (2000). The power law repealed: The case for an exponential law of practice. Psychonomic Bulletin & Review, 7, 185–207. https://doi.org/10.3758/BF03212979
Heene, M., Kyngdon, A., & Sckopke, P. (2016). Detecting violations of unidimensionality by order-restricted inference methods. Frontiers in Applied Mathematics and Statistics, 2, 1–13. https://doi.org/10.3389/fams.2016.00003
Hull, C. (1943). Principles of behavior. Appleton-Century-Crofts.
Jones, L. V., & Thissen, D. (2006). A history and overview of psychometrics. In C. R. Rao & S. Sinharay (Eds.), Handbook of statistics (Vol. 26, pp. 1–27). Elsevier.
Junker, B. W., & Sijtsma, K. (2000). Latent and manifest monotonicity in item response models. Applied Psychological Measurement, 24(1), 65–81. https://doi.org/10.1177/01466216000241004
Kagan, J. (2005). A time for specificity. Journal of Personality Assessment, 85, 125–127. https://doi.org/10.1207/s15327752jpa8502_03
Kahneman, D., & Tversky, A. (1979). Prospect theory: An analysis of decision under risk. Econometrica, 47(2), 262-292. 0012-9682(197903)47:2<263:PTAAOD>2.0.CO;2-3
Kanazawa, S. (1998). In defense of unrealistic assumptions. Sociological Theory, 16(2), 193–204. https://doi.org/10.1111/0735-2751.00050
Kim, E. J., Namkoong, K., Ku, T., & Kim, S. J. (2008). The relationship between online game addiction and aggression, self-control and narcissistic personality traits. European Psychiatry, 23(3), 212–218. https://doi.org/10.1016/j.eurpsy.2007.10.010
Kish, L. (2004). Statistical design for research. John Wiley & Sons.
Knol, D. L., & Berger, M. P. (1991). Empirical comparison between factor analysis and multidimensional item response models. Multivariate Behavioral Research, 26(3), 457–477. https://doi.org/10.1207/s15327906mbr2603_5
Kruis, J., & Maris, G. (2016). Three representations of the Ising model. Scientific Reports, 6, 1–11. https://doi.org/10.1038/srep34175
LaHuis, D. M., Blackmore, C. E., Bryant-Lees, K. B., & Delgado, K. (2018). Applying Item Response Trees to personality data in the selection context. Organizational Research Methods, Online first, 1-12. https://doi.org/10.1177/1094428118780310
Lauritzen, S. L. (1996). Graphical models. Clarendon Press.
Lee, M. D. (2011). How cognitive modeling can benefit from hierarchical Bayesian models. Journal of Mathematical Psychology, 55(1), 1–7. https://doi.org/10.1016/j.jmp.2010.08.013
Lee, M. D., & Wagenmakers, E. J. (2014). Bayesian cognitive modeling: A practical course. Cambridge University Press.
Levine, M. V. (2003). Dimension in latent variable models. Journal of Mathematical Psychology, 47(4), 450–466. https://doi.org/10.1016/S0022-2496(03)00032-4
Lewandowsky, S., & Farrell, S. (2010). Computational modeling in cognition: Principles and practice. Sage.
Lord, F. M. (1953). An application of confidence intervals and of maximum likelihood to the estimation of an examinee’s ability. Psychometrika, 18(1), 57–76. https://doi.org/10.1007/BF02289028
Manning, C. D., Manning, C. D., & Schütze, H. (1999). Foundations of statistical natural language processing. MIT Press.
Marsman, M., Borsboom, D., Kruis, J., Epskamp, S., van Bork, R., Waldorp, L. J., & van der Maas, & Maris, G. (2018). An introduction to network psychometrics: Relating Ising network models to item response theory models. Multivariate Behavioral Research, 53(1), 15–35. https://doi.org/10.1080/00273171.2017.1379379
McDonald, R. P. (2013). Test theory: A unified treatment. Psychology Press.
McDonald, R. P. (1965). Difficulty factors and nonlinear factor analysis. British Journal of Mathematical and Statistical Psychology, 18(1), 11–23. https://doi.org/10.1111/j.2044-8317.1965.tb00690.x
McNeish, D., & Wolf, M. G. (2020). Thinking twice about sum scores. Behavior Research Methods, 52(6), 2287–2305. https://doi.org/10.3758/s13428-020-01398-0
Mertens, D. M. (2014). Research and evaluation in education and psychology: Integrating diversity with quantitative, qualitative, and mixed methods. Sage.
Michell, J. (1990). An introduction to the logic of psychological measurement. Erlbaum.
Michell, J. (1997). Quantitative science and the definition of measurement in psychology. British Journal of Psychology, 88(3), 355–383. https://doi.org/10.1111/j.2044-8295.1997.tb02641.x
Michell, J. (2000). Normal science, pathological science and psychometrics. Theory & Psychology, 10, 639–667. https://doi.org/10.1177/0959354300105004
Michell, J. (2005). The logic of measurement: A realist overview. Measurement, 38(4), 285–294. https://doi.org/10.1016/j.measurement.2005.09.004
Michell, J. (2008). Conjoint measurement and the Rasch paradox: A response to Kyngdon. Theory & Psychology, 18, 119–124. https://doi.org/10.1177/0959354307086926
Michell, J. (2017). On substandard substantive theory and axing axioms of measurement: A response to Humphry. Theory & Psychology, 27(3), 419–425. https://doi.org/10.1177/0959354317706746
Miyazaki, K., & Hoshino, T. (2009). A Bayesian semiparametric item response model with Dirichlet process priors. Psychometrika, 74(3), 375–393. https://doi.org/10.1007/s11336-008-9108-6
Mokken, R. J. (1971). A theory and procedure of scale analysis. The Hague: Mouton/Berlin: De Gruyter.
Mokken, R. J., & Lewis, C. (1982). A nonparametric approach to the analysis of dichotomous item responses. Applied Psychological Measurement, 6, 417–430. https://doi.org/10.1177/014662168200600404
Myung, I. J., & Pitt, M. A. (2001). Mathematical modeling. MethodologyIn J. Wixted (Ed.), Stevens’ handbook of experimental psychology (Vol. 4, pp. 429–459). Wiley.
Navarro, D. J. (2021). If mathematical psychology did not exist we might need to invent it: A comment on theory building in psychology. Perspectives on Psychological Science, 16(4), 707–716. https://doi.org/10.1177/1745691620974769
Nisbett, R. E., Aronson, J., Blair, C., Dickens, W., Flynn, J., Halpern, D. F., & Turkheimer, E. (2012). Intelligence: New findings and theoretical developments. American Psychologist, 67(2), 130–159. https://doi.org/10.1037/a0026699
Nosofsky, R. M. (1986). Attention, similarity, and the identification–categorization relationship. Journal of Experimental Psychology: General, 115(1), 39–57. https://doi.org/10.1037/0096-3445.115.1.39
Nunnally, J. C. (1978). Psychometric theory. McGraw-Hill.
Plieninger, H. (2021). Developing and applying IR-tree models: Guidelines, caveats, and an extension to multiple groups. Organizational Research Methods, 24(3), 654–670. https://doi.org/10.1177/1094428120911096
Price, D. J. S. (1986). Little science, big science... and beyond. New York: Columbia University Press.
Queirós, A., Faria, D., & Almeida, F. (2017). Strengths and limitations of qualitative and quantitative research methods. European Journal of Education Studies, 3(9), 369–387. https://oapub.org/edu/index.php/ejes/article/view/1017/2934
Rasch, G. (1960). Studies in mathematical psychology: I. Probabilistic models for some intelligence and attainment tests. Danish Educational Research Institute.
Reckase, M. (2009). Multidimensional item response theory. Springer.
Reise, S. P., Ainsworth, A. T., & Haviland, M. G. (2005). Item response theory: Fundamentals, applications, and promise in psychological research. Current Directions in Psychological Science, 14(2), 95–101. https://doi.org/10.1111/j.0963-7214.2005.00342.x
Rosenbaum, P. R. (1987). Comparing item characteristic curves. Psychometrika, 52(2), 217–233. https://doi.org/10.1007/BF02294236
Sandberg, J., & Alvesson, M. (2011). Ways of constructing research questions: Gap-spotting or problematization? Organization, 18(1), 23–44. https://doi.org/10.1177/1350508410372151
Shaughnessy, J. J., Zechmeister, E. B., & Zechmeister, J. S. (2014). Research methods in psychology. Alfred A. Knopf.
Shoenfield, J. R. (2018). Mathematical logic. CRC Press.
Sijtsma, K. (2012). Psychological measurement between physics and statistics. Theory & Psychology, 22(6), 786–809. https://doi.org/10.1177/0959354312454353
Sijtsma, K., & Molenaar, I. W. (2002). Introduction to nonparametric item response theory. Sage.
Stanislaw, H., & Todorov, N. (1999). Calculation of signal detection theory measures. Behavior Research Methods, Instruments, & Computers, 31(1), 137–149. https://doi.org/10.3758/BF03207704
Stanovich, K. E. (2012). How to think straight about psychology. Pearson.
Stevens, S. S. (1946). On the theory of scales of measurement. Science, 103, 667–680. https://www.jstor.org/stable/1671815
Stout, W. (1987). A nonparametric approach for assessing latent trait unidimensionality. Psychometrika, 52(4), 589–617. https://doi.org/10.1007/BF02294821
Straat, J. H., Van der Ark, L. A., & Sijtsma, K. (2013). Comparing optimization algorithms for item selection in Mokken scale analysis. Journal of Classification, 30(1), 75–99. https://doi.org/10.1007/s00357-013-9122-y
Thabane, L., Thomas, T., Ye, C., & Paul, J. (2009). Posing the research question: Not so simple. Canadian Journal of Anesthesia, 56(1), 71–79. https://doi.org/10.1007/s12630-008-9007-4
Thompson, B. (2004). Exploratory and confirmatory factor analysis: Understanding concepts and applications. American Psychological Association.
Townsend, J. T. (2008). Mathematical psychology: Prospects for the 21st century: A guest editorial. Journal of Mathematical Psychology, 52(5), 269–280. https://doi.org/10.1016/j.jmp.2008.05.001
Trendler, G. (2009). Measurement theory, psychology and the revolution that cannot happen. Theory & Psychology, 19(5), 579–599. https://doi.org/10.1177/0959354309341926
Trendler, G. (2013). Measurement in psychology: A case of ignoramus et ignorabimus? A Rejoinder. Theory & Psychology, 23(5), 591–615. https://doi.org/10.1177/0959354313490451
Van Bork, R., Rhemtulla, M., Waldorp, L. J., Kruis, J., Rezvanifar, S., & Borsboom, D. (2021). Latent variable models and networks: Statistical equivalence and testability. Multivariate Behavioral Research, 56(2), 175–198. https://doi.org/10.1080/00273171.2019.1672515
Van der Linden, W. J., & Hambleton, R. K. (2013). Handbook of modern item response theory. Springer.
Van der Maas, H. L., Dolan, C. V., Grasman, R. P., Wicherts, J. M., Huizenga, H. M., & Raijmakers, M. E. (2006). A dynamical model of general intelligence: The positive manifold of intelligence by mutualism. Psychological Review, 113(4), 842–861. https://doi.org/10.1037/0033-295X.113.4.842
Van der Maas, H., Kan, K. J., Marsman, M., & Stevenson, C. E. (2017). Network models for cognitive development and intelligence. Journal of Intelligence, 5(2), 1–17. https://doi.org/10.3390/jintelligence5020016
Von Neumann, J., & Morgenstern, O. (1944). Theory of games and economic behavior. Princeton University Press.
Wang, C., Chang, H. H., & Douglas, J. A. (2013). The linear transformation model with frailties for the analysis of item response times. British Journal of Mathematical and Statistical Psychology, 66(1), 144–168. https://doi.org/10.1111/j.2044-8317.2012.02045.x
Wiberg, M., Ramsay, J. O., & Li, J. (2019). Optimal scores: An alternative to parametric item response theory and sum scores. Psychometrika, 84(1), 310–322. https://doi.org/10.1007/s11336-018-9639-4
Zhang, J., & Stout, W. (1999). The theoretical DETECT index of dimensionality and its application to approximate simple structure. Psychometrika, 64(2), 213–249. https://doi.org/10.1007/BF02294536
Author information
Authors and Affiliations
Contributions
The first author has conceived the presented idea and the main structure of the text. The other authors contributed with writing and critically revising (i.e., suggesting changes for both the idea and the text) the manuscript. The other authors are ordered according to the size of their contribution.
Corresponding author
Ethics declarations
Ethics Approval
Not applicable.
Consent to Participate
Not applicable.
Consent for Publication
Not applicable.
Conflict of Interest
The authors declare no competing interests.
Rights and permissions
About this article
Cite this article
Franco, V.R., Laros, J.A., Wiberg, M. et al. How to Think Straight About Psychometrics: Improving Measurement by Identifying its Assumptions. Trends in Psychol. (2022). https://doi.org/10.1007/s43076-022-00183-6
Accepted:
Published:
DOI: https://doi.org/10.1007/s43076-022-00183-6