Skip to main content
SpringerLink
Log in

Using Deterministic, Gated Item Response Theory Model to Detect Test Cheating due to Item Compromise

  • Published:
Psychometrika Aims and scope Submit manuscript

Abstract

The Deterministic, Gated Item Response Theory Model (DGM, Shu, Unpublished Dissertation. The University of North Carolina at Greensboro, 2010) is proposed to identify cheaters who obtain significant score gain on tests due to item exposure/compromise by conditioning on the item status (exposed or unexposed items). A “gated” function is introduced to decompose the observed examinees’ performance into two distributions (the true ability distribution determined by examinees’ true ability and the cheating distribution determined by examinees’ cheating ability). Test cheaters who have score gain due to item exposure are identified through the comparison of the two distributions. Hierarchical Markov Chain Monte Carlo is used as the model’s estimation framework. Finally, the model is applied in a real data set to illustrate how the model can be used to identify examinees having pre-knowledge on the exposed items.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Figure 1.
Figure 2.
Figure 3.
Figure 4.
Figure 5.

Similar content being viewed by others

References

  • Angoff, W.H. (1974). The development of statistical indices for detecting cheaters. Journal of the American Statistical Association, 69, 44–49.

    Article  Google Scholar 

  • Bellezza, F., & Bellezza, S. (1995). Detection of copying on multiple-choice tests: an update. Teaching of Psychology, 22(3), 180–182.

    Article  Google Scholar 

  • Cizek, G.J. (1999). Cheating on tests: how to do it, detect it, and prevent it. Mahway: Lawrence Erlbaum Associates.

    Google Scholar 

  • Drasgow, F., Levine, M.V., & Williams, E.A. (1985). Appropriateness measurement with polychotomous item response models and standardized indices. British Journal of Mathematical & Statistical Psychology, 38(1), 67–86.

    Article  Google Scholar 

  • Drasgow, F., & Levine, M.V. (1986). Optimal detection of certain forms of inappropriate test scores. Applied Psychological Measurement, 10, 59–67.

    Article  Google Scholar 

  • Drasgow, F., Luecht, R.M., & Bennett, R. (2006). Technology and testing. In R.L. Brennan (Ed.), Educational measurement (4th ed., pp. 471–515). Washington: American Council on Education/Praeger Publishers.

    Google Scholar 

  • Dwyer, D.J., & Hecht, J.B. (1996). Using statistics to catch cheaters: methodological and legal issues for students personnel administrators. NASPA Journal, 33(2), 125–135.

    Google Scholar 

  • Frary, R., Tideman, N., & Watts, T. (1977). Indices of cheating on multiple choice tests. Journal of Educational Statistics, 2(4), 235–256.

    Article  Google Scholar 

  • Hanson, B., Harris, D., & Brennan, R. (1987). A comparison of several methods for examining allegations of copying (ACT Research Report No. 87-15). Iowa City: American College Testing.

  • Holand, P. (1996). Assessing unusual agreement between the incorrect answers of two examinees using the K-index: statistical theory and empirical support ETS (Technical Report No. 96-4). Princeton: Educational Testing Service.

  • Levine, M.V., & Rubin, D.B. (1979). Measuring the appropriateness of multiple-choice test scores. Journal of Educational Statistics, 4, 269–290.

    Article  Google Scholar 

  • Lewis, C., & Thayer, D.T. (1998). The power of K-index to detect test copying (Research Report No. 08541). Princeton: Educational Testing Service.

  • Luecht, R.M. (1998). A framework for exploring and controlling risks associated with test item exposure over time. Paper presented at the national council on measurement in education annual meeting, San Diego.

  • Luecht, R.M. (2005). Some useful cost-benefit criteria for evaluating computer-based test delivery models and systems. Journal of Applied Testing Technology. http://www.testpublishers.org/assets/documents/Volum%207%20Some%20useful%20cost%20benefit.pdf.

  • Meijer, R.R. (Ed.) (1996). Person-fit research: theory and applications. Applied Measurement in Education, 9(1), 9–18, [Special issue].

  • Mcleod, L., & Lewis, C. (1999). Detecting item memorization in CAT environment. Applied Psychological Measurement, 23(2), 147–159.

    Article  Google Scholar 

  • Mcleod, L., Lewis, C., & Thissen, D. (2003). A Bayesian method for the detection of item pre-knowledge in computerized adaptive testing. Applied Psychological Measurement, 27(2), 121–137.

    Article  Google Scholar 

  • Nering, M.L. (1996). The effects of person misfit in computerized adaptive testing. Unpublished doctoral dissertation, University of Minnesota, Minneapolis.

  • Nering, M.L. (1997). The distribution of indexes of person fit within the computerized adaptive testing environment. Applied Psychological Measurement, 21, 115–127.

    Article  Google Scholar 

  • Patz, R.J., & Junker, B.W. (1999a). A straightforward approach to Markov chain Monte Carlo methods for item response models. Journal of Educational and Behavioral Statistics, 24, 146–178.

    Google Scholar 

  • Patz, R.J., & Junker, B.W. (1999b). A straightforward approach to Markov Chain Monte Carlo methods for item response models. Journal of Educational and Behavioral Statistics, 24(2), 146–178.

    Google Scholar 

  • Rost, J. (1990). Rasch models in latent classes: an integration of two approaches to item analysis. Applied Psychological Measurement, 14(3), 271–282.

    Article  Google Scholar 

  • Rasch, G. (1960). Probabilistic models of some intelligence and attainment tests. Copenhagen: Danish Institute for Educational Research.

    Google Scholar 

  • Stocking, M.L., Ward, W.C., & Potenza, M.T. (1998). Simulating the use of disclosed items in computerized adaptive testing. Journal of Educational Measurement, 35, 48–68.

    Article  Google Scholar 

  • Segall, D. (2002). An item response model for characterizing test comprise. Journal of Educational and Behavioral Statistics, 27(2), 163–179.

    Article  Google Scholar 

  • Segall, D. (2004). A sharing item response theory model for computerized adaptive testing. Journal of Educational and Behavioral Statistics, 29(4), 439–460.

    Article  Google Scholar 

  • Shu, Z. (2010). Using the deterministic, gated item response model detecting test cheating. Unpublished Dissertation. The University of North Carolina at Greensboro.

  • Sotaridona, L.S., & Meijer, R.R. (2003). Two new statistics to detect answer copying. Journal of Educational Measurement, 40(1), 53–69.

    Article  Google Scholar 

  • Sotaridona, L.S. (2003). Statistical methods for the detection of answer copying on achievement tests. AE Enschede: Twente University Press.

    Google Scholar 

  • Tatsuoka, K. (1996). Use of generalized person-fit indexes, zetas for statistical pattern classification. Applied Measurement in Education, 9(1), 65–75.

    Article  Google Scholar 

  • van der Linden, W.J., & Sotaridona, L.S. (2006). Detecting answer copying when the regular response process follows a known response model. Journal of Educational and Behavioral Statistics, 31(3), 283–304.

    Article  Google Scholar 

  • van der Linden, W.J., & Sotaridona, L.S. (2004). A statistical test for detecting answer copying on multiple-choice tests. Journal of Educational Measurement, 41(4), 361–377.

    Article  Google Scholar 

  • van der Linden, W.J., & Jeon, M. (2012). Modeling answer changes on test items. Journal of Educational and Behavioral Statistics, 37(1), 180–199.

    Article  Google Scholar 

  • Watson, S.A., Iwamoto, C.K., Nungester, R.J., & Luecht, R.M. (1998). The use of response similarity statistics to study examinees who benefit from copying. Paper presented at the national council on measurement in education annual meeting, San Diego.

  • Wollack, J.A. (1997). A nominal response model approach for detecting answer copying. Applied Psychological Measurement, 21(4), 307–320.

    Article  Google Scholar 

  • Wollack, A., & Cohen, A. (1998). Detection of answer copying with unknown item and trait parameters. Applied Measurement in Education, 22(2), 144–152.

    Google Scholar 

  • Wollack, A., Cohen, A., & Serlin, R. (2001). Defining error rate and power for detecting answer copying. Applied Psychological Measurement, 25(4), 385–404.

    Article  Google Scholar 

  • Wollack, J.A. (2006). Simultaneous use of multiple answer copying indexes to improve detection rates. Applied Measurement in Education, 19(4), 265–288.

    Article  Google Scholar 

Download references

Acknowledgements

The real data used in this research are provided by CTB/McGraw-Hill, based in Monterey, CA. An appreciation goes to Dr. Furong Gao and Dr. Jesswlyn Smith at CTB/McGraw-Hill who helped review this research. Further, Dr. Charlie Lewis and Dr. Xueli Xu, both from ETS, are also appreciated because of their kind and wise advice.

Author information

Authors and Affiliations

  1. Educational Testing Service, 660 Rosedale Road, Princeton, NJ, 08541, USA

    Zhan Shu

  2. The University of North Carolina at Greensboro, Greensboro, USA

    Robert Henson & Richard Luecht

Authors
  1. Zhan Shu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Robert Henson
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Richard Luecht
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zhan Shu.

Additional information

Requests for reprints and correspondence concerning this article should be addressed to Zhan Shu, Educational Testing Service, 660 Rosedale Road, Princeton, NJ 08541, USA. E-mail: zshu@ets.org

Rights and permissions

Reprints and permissions

About this article

Cite this article

Shu, Z., Henson, R. & Luecht, R. Using Deterministic, Gated Item Response Theory Model to Detect Test Cheating due to Item Compromise. Psychometrika 78, 481–497 (2013). https://doi.org/10.1007/s11336-012-9311-3

Download citation

  • Received:

  • Revised:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11336-012-9311-3

Key words

Search

Navigation