Careless responding, where participants do not fully engage with item content, is pervasive in survey research. Left undetected, carelessness can compromise the interpretation and use of survey results, including information about participant locations on the construct, item difficulty, and the psychometric quality of the instrument. We present and illustrate a sequential procedure for evaluating response quality in survey research using indicators from Mokken scale analysis (MSA). We use a real data illustration and a simulation study to compare a sequential procedure to a standalone procedure. We also consider how identifying and removing responses with evidence of poor measurement properties affects item quality indicators. Results suggest that the sequential procedure was effective in identifying potentially problematic response patterns that may not always be captured by traditional methods for identifying careless responders but was not always sensitive to specific carelessness patterns. We discuss implications for research and practice.
This is a preview of subscription content, access via your institution.
Buy single article
Instant access to the full article PDF.
Price excludes VAT (USA)
Tax calculation will be finalised during checkout.
Bandalos, D. L. (2018). Measurement Theory and Applications for the Social Sciences.
Curran, P. G. (2016). Methods for the detection of carelessly invalid responses in survey data. Journal of Experimental Social Psychology, 66, 4–19. https://doi.org/10.1016/j.jesp.2015.07.006
Curran, P. G., & Hauser, K. A. (2019). I’m paid biweekly, just not by leprechauns: Evaluating valid-but-incorrect response rates to attention check items. Journal of Research in Personality, 82, 103849.
DeAyala, R. J. (2009). The theory and practice of item response theory. The Guilford Press.
Eisele, G., Vachon, H., Lafit, G., Kuppens, P., Houben, M., Myin-Germeys, I., & Viechtbauer, W. (2022). The effects of sampling frequency and questionnaire length on perceived burden, compliance, and careless responding in experience sampling data in a student population. Assessment, 29(2), 136–151.
Ferrando, P. J. (2007). Factor-analytic procedures for assessing person scalability. Multivariate Behavioral Research, 42, 481–507.
Goldammer, P., Annen, H., Stöckli, P. L., & Jonas, K. (2020). Careless responding in questionnaire measures: Detection, impact, and remedies. The Leadership Quarterly, 31(4), 101384. https://doi.org/10.1016/j.leaqua.2020.101384
Koopman, L., Zijlstra, B. J. H., & Van Der Ark, L. A. (2020). New advances in Mokken scale analysis Annual Meeting of the Psychometric Society.
Meijer, R. R. (1994). The number of Guttman errors as a simple and powerful person-fit statistic. Applied Psychological Measurement, 18(4), 311–314. https://doi.org/10.1177/014662169401800402
Meijer, R. R., & Baneke, J. J. (2004). Analyzing psychopathology items: A case for nonparametric item response theory modeling. Psychological Methods, 9(3), 354–368. https://doi.org/10.1037/1082-989X.9.3.354
Meijer, R. R., Tendeiro, J. N., & Wanders, R. B. K. (2015). The use of nonparametric item response theory to explore data quality. In S. P. Reise & D. A. Revicki (Eds.), Handbook of item response theory modeling: Applications to typical performance assessment (pp. 85–110). Routledge.
Mokken, R. J. (1971). A theory and procedure of scale analysis. De Gruyter.
Molenaar, I. W. (1997). Nonparametric models for polytomous responses. In W. J. van der Linden & R. K. Hambleton (Eds.), Handbook of modern item response theory (pp. 369–380). Springer.
Muraki, E., & Muraki, M. (2018). Generalized partial credit model. In W. J. van der Linden (Ed.), Handbook of item response theory (Vol. 1, pp. 127–138). CRC Press.
Niessen, A. S. M., Meijer, R. R., & Tendeiro, J. N. (2016). Detecting careless respondents in web-based questionnaires: Which method to use? Journal of Research in Personality, 63, 1–11. https://doi.org/10.1016/j.jrp.2016.04.010
Patton, J. M., Cheng, Y., & Diao, Q. (2019). Detection and treatment of careless responses to improve item parameter estimation. Journal of Educational and Behavioral Statistics, 44(3), 309–341. https://doi.org/10.3102/1076998618825116
R Core Team. (2021). R: A language and environment for statistical computing. R Foundation for Statistical Computing. https://www.R-project.org/
Ryan, R. M., & Deci, E. L. (2020). Intrinsic and extrinsic motivation from a self-determination theory perspective: Definitions, theory, practices, and future directions. Contemporary Educational Psychology, 61, 101860. https://doi.org/10.1016/j.cedpsych.2020.101860
Silvia, P. J., & Rodriguez, R. M. (2020). Time to renovate the humor styles questionnaire? An item response theory analysis of the HSQ. Behavioral Sciences, 10(11), 173. https://doi.org/10.3390/bs10110173
Straat, J. H., van der Ark, L. A., & Sijtsma, K. (2013). Comparing optimization algorithms for item selection in Mokken scale analysis. Journal of Classification, 30(1), 75–99. https://doi.org/10.1007/s00357-013-9122-y
Tendeiro, J. N., & Meijer, R. R. (2014). Detection of invalid test scores: The usefulness of simple nonparametric statistics: Detection of invalid test scores. Journal of Educational Measurement, 51(3), 239–259. https://doi.org/10.1111/jedm.12046
van Schuur, W. H. (2011). Ordinal Item Response Theory: Mokken Scale Analysis. Sage.
Vriesema, C. C., & Gehlbach, H. (2021). Assessing survey satisficing: The impact of unmotivated questionnaire responding on data quality. Educational Researcher, 50(9), 618–627. https://doi.org/10.3102/0013189X211040054
Walker, A. A., & Wind, S. A. (2020). Identifying Misfitting achievement estimates in performance assessments: An illustration using Rasch and Mokken scale analyses. International Journal of Testing, 20(3), 231–251. https://doi.org/10.1080/15305058.2019.1673758
Wang, Y., & Wind, S. A. (2020). Psychometric evaluation of a new internalization of learning motivation (ILM) scale. Current Psychology. https://doi.org/10.1007/s12144-020-00909-8
Wind, S. A., & Wang, Y. (2022). Using Mokken scaling techniques to explore carelessness in survey research. Behavior Research Methods, Online First. https://doi.org/10.3758/s13428-022-01960-y
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Open practices statement
The authors do not have permission to share the real data used in this study.
Code used for the analyses with an example is available at the following URL:
This study was not preregistered.
Studying is hardly ever exciting
I learn in school because my parents say I have to
I learn in school because my teachers say I have to
I would not study if my teachers did not make me do it
If my parents do not push me, I would not push myself to learn in school
I study hard to avoid my parents scolding me
I study to avoid being criticized by my parents
I study to avoid being criticized by my parents
If I do not study hard, my parents will punish me
I study so I will not look incompetent in front of others
I work on homework, so my classmates will think I am smart
I study because I want my teacher to think I am smart
I study because I would feel bad about myself if got a bad grade
I feel guilty if I do not learn something well
I feel ashamed if I do not get a good grade on an exam or homework assignment
I study in school because I personally value what I learn
It is important to me that I study regularly/consistently
I am motivated to learn because I find the content meaningful
I study because I am passionate about learning
I am motivated to learn in school because it teaches me how to solve problems
I study because it helps me figure out a purpose in life
I study so I can use what I learn to help others
Learning in school helps me figure out what careers fit my personality
I study because it increases my desire to learn more
About this article
Cite this article
Wind, S.A., Lugu, B. & Wang, Y. A sequential Moken scaling approach to evaluate response quality in survey research. Behav Res (2023). https://doi.org/10.3758/s13428-023-02147-9