Skip to main content

A sequential Moken scaling approach to evaluate response quality in survey research


Careless responding, where participants do not fully engage with item content, is pervasive in survey research. Left undetected, carelessness can compromise the interpretation and use of survey results, including information about participant locations on the construct, item difficulty, and the psychometric quality of the instrument. We present and illustrate a sequential procedure for evaluating response quality in survey research using indicators from Mokken scale analysis (MSA). We use a real data illustration and a simulation study to compare a sequential procedure to a standalone procedure. We also consider how identifying and removing responses with evidence of poor measurement properties affects item quality indicators. Results suggest that the sequential procedure was effective in identifying potentially problematic response patterns that may not always be captured by traditional methods for identifying careless responders but was not always sensitive to specific carelessness patterns. We discuss implications for research and practice.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9


  • Bandalos, D. L. (2018). Measurement Theory and Applications for the Social Sciences.

  • Curran, P. G. (2016). Methods for the detection of carelessly invalid responses in survey data. Journal of Experimental Social Psychology, 66, 4–19.

    Article  Google Scholar 

  • Curran, P. G., & Hauser, K. A. (2019). I’m paid biweekly, just not by leprechauns: Evaluating valid-but-incorrect response rates to attention check items. Journal of Research in Personality, 82, 103849.

    Article  Google Scholar 

  • DeAyala, R. J. (2009). The theory and practice of item response theory. The Guilford Press.

    Google Scholar 

  • Eisele, G., Vachon, H., Lafit, G., Kuppens, P., Houben, M., Myin-Germeys, I., & Viechtbauer, W. (2022). The effects of sampling frequency and questionnaire length on perceived burden, compliance, and careless responding in experience sampling data in a student population. Assessment, 29(2), 136–151.

    Article  PubMed  Google Scholar 

  • Ferrando, P. J. (2007). Factor-analytic procedures for assessing person scalability. Multivariate Behavioral Research, 42, 481–507.

    Article  Google Scholar 

  • Goldammer, P., Annen, H., Stöckli, P. L., & Jonas, K. (2020). Careless responding in questionnaire measures: Detection, impact, and remedies. The Leadership Quarterly, 31(4), 101384.

    Article  Google Scholar 

  • Koopman, L., Zijlstra, B. J. H., & Van Der Ark, L. A. (2020). New advances in Mokken scale analysis Annual Meeting of the Psychometric Society.

    Google Scholar 

  • Meijer, R. R. (1994). The number of Guttman errors as a simple and powerful person-fit statistic. Applied Psychological Measurement, 18(4), 311–314.

    Article  Google Scholar 

  • Meijer, R. R., & Baneke, J. J. (2004). Analyzing psychopathology items: A case for nonparametric item response theory modeling. Psychological Methods, 9(3), 354–368.

    Article  PubMed  Google Scholar 

  • Meijer, R. R., Tendeiro, J. N., & Wanders, R. B. K. (2015). The use of nonparametric item response theory to explore data quality. In S. P. Reise & D. A. Revicki (Eds.), Handbook of item response theory modeling: Applications to typical performance assessment (pp. 85–110). Routledge.

    Google Scholar 

  • Mokken, R. J. (1971). A theory and procedure of scale analysis. De Gruyter.

  • Molenaar, I. W. (1997). Nonparametric models for polytomous responses. In W. J. van der Linden & R. K. Hambleton (Eds.), Handbook of modern item response theory (pp. 369–380). Springer.

    Chapter  Google Scholar 

  • Muraki, E., & Muraki, M. (2018). Generalized partial credit model. In W. J. van der Linden (Ed.), Handbook of item response theory (Vol. 1, pp. 127–138). CRC Press.

    Google Scholar 

  • Niessen, A. S. M., Meijer, R. R., & Tendeiro, J. N. (2016). Detecting careless respondents in web-based questionnaires: Which method to use? Journal of Research in Personality, 63, 1–11.

    Article  Google Scholar 

  • Patton, J. M., Cheng, Y., & Diao, Q. (2019). Detection and treatment of careless responses to improve item parameter estimation. Journal of Educational and Behavioral Statistics, 44(3), 309–341.

    Article  Google Scholar 

  • R Core Team. (2021). R: A language and environment for statistical computing. R Foundation for Statistical Computing.

  • Ryan, R. M., & Deci, E. L. (2020). Intrinsic and extrinsic motivation from a self-determination theory perspective: Definitions, theory, practices, and future directions. Contemporary Educational Psychology, 61, 101860.

    Article  Google Scholar 

  • Silvia, P. J., & Rodriguez, R. M. (2020). Time to renovate the humor styles questionnaire? An item response theory analysis of the HSQ. Behavioral Sciences, 10(11), 173.

    Article  PubMed  PubMed Central  Google Scholar 

  • Straat, J. H., van der Ark, L. A., & Sijtsma, K. (2013). Comparing optimization algorithms for item selection in Mokken scale analysis. Journal of Classification, 30(1), 75–99.

    Article  Google Scholar 

  • Tendeiro, J. N., & Meijer, R. R. (2014). Detection of invalid test scores: The usefulness of simple nonparametric statistics: Detection of invalid test scores. Journal of Educational Measurement, 51(3), 239–259.

    Article  Google Scholar 

  • van Schuur, W. H. (2011). Ordinal Item Response Theory: Mokken Scale Analysis. Sage.

  • Vriesema, C. C., & Gehlbach, H. (2021). Assessing survey satisficing: The impact of unmotivated questionnaire responding on data quality. Educational Researcher, 50(9), 618–627.

    Article  Google Scholar 

  • Walker, A. A., & Wind, S. A. (2020). Identifying Misfitting achievement estimates in performance assessments: An illustration using Rasch and Mokken scale analyses. International Journal of Testing, 20(3), 231–251.

    Article  Google Scholar 

  • Wang, Y., & Wind, S. A. (2020). Psychometric evaluation of a new internalization of learning motivation (ILM) scale. Current Psychology.

  • Wind, S. A., & Wang, Y. (2022). Using Mokken scaling techniques to explore carelessness in survey research. Behavior Research Methods, Online First.

Download references

Author information

Authors and Affiliations


Corresponding author

Correspondence to Stefanie A. Wind.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Open practices statement

The authors do not have permission to share the real data used in this study.

Code used for the analyses with an example is available at the following URL:

This study was not preregistered.

Appendix A

Appendix A

Item number

Item stem


Studying is hardly ever exciting


I learn in school because my parents say I have to


I learn in school because my teachers say I have to


I would not study if my teachers did not make me do it


If my parents do not push me, I would not push myself to learn in school


I study hard to avoid my parents scolding me


I study to avoid being criticized by my parents


I study to avoid being criticized by my parents


If I do not study hard, my parents will punish me


I study so I will not look incompetent in front of others


I work on homework, so my classmates will think I am smart


I study because I want my teacher to think I am smart


I study because I would feel bad about myself if got a bad grade


I feel guilty if I do not learn something well


I feel ashamed if I do not get a good grade on an exam or homework assignment


I study in school because I personally value what I learn


It is important to me that I study regularly/consistently


I am motivated to learn because I find the content meaningful


I study because I am passionate about learning


I am motivated to learn in school because it teaches me how to solve problems


I study because it helps me figure out a purpose in life


I study so I can use what I learn to help others


Learning in school helps me figure out what careers fit my personality


I study because it increases my desire to learn more

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and Permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wind, S.A., Lugu, B. & Wang, Y. A sequential Moken scaling approach to evaluate response quality in survey research. Behav Res (2023).

Download citation

  • Accepted:

  • Published:

  • DOI: