Abstract
Employment tests have long been scrutinized for psychometric considerations such as validity and reliability, but the extent to which cheating may occur on pre-employment tests has generally been overlooked. With the rise of unproctored, on-demand, online testing, the need has never been greater to focus on design and process considerations that can help mitigate the potential for cheating on employment tests. This paper builds on the limited existing research on the detection of inconsistent test scores on unproctored Internet testing. Job candidates (n = 4,026) completed a computer adaptive cognitive ability test under two conditions: an unproctored screening test followed by a proctored confirmation test. Analyses focused on detecting instances of inconsistent test scores based on comparison of standard errors of measure for the unproctored and proctored test scores (Guo and Drasgow, Int J Sel Assess 18:351–364, 2010). Results revealed a relatively low number of inconsistent scores and are discussed in the context of future research, application, and theory building on the nature of cheating on pre-employment tests.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
where H 0 is the null hypothesis, H a is the alternate hypothesis, θ u is the theta estimate yielded by the unproctored test administration, and θ v is that yielded by the proctored administration.
where d is the pairwise difference by the unproctored test administration, D is the hypothesized difference (0), and σ d is the standard deviation of the sample pairwise differences.
References
Arthur, W., Glaze, R. M., Villado, A. J., & Taylor, J. E. (2010). The magnitude and extent of cheating and response distortion effects on unproctored Internet-based tests of cognitive ability and personality. International Journal of Selection and Assessment, 18, 1–16.
Beaty, J. C., Dawson, C. R., Fallaw, S. S., & Kantrowitz, T. M. (2009). Recovering the scientist-practitioner model: How IOs should respond to UIT. Industrial and Organizational Psychology: Perspectives on Science and Practice, 2, 58–63.
Beaty, J. C., Nye, C., Borneman, M., Kantrowitz, T. M., Drasgow, F., & Grauer, E. (2011). Proctored versus unproctored internet tests: Are unproctored tests as predictive of job performance? International Journal of Selection and Assessment, 19, 1–10.
Cizek, G. J. (1999). Cheating on tests: How to do it, detect it, and prevent it. Mahwah, NJ: Erlbaum.
Drasgow, F., Nye, C. D., Guo, J., & Tay, L. (2009). Cheating on proctored tests: The other side of the unproctored debate. Industrial and Organizational Psychology: Perspectives on Science and Practice, 2, 46–48.
Fallaw, S., Kantrowitz, T. M., & Dawson, C. R. (2012). Global assessment trends report. Technical Report. Alpharetta, GA: SHL.
Fetzer, M., & Grelle, D. (2010). PreVisor ConVerge: The Best Practice for Unproctored/Unsupervised Internet Testing. Alpharetta, GA: White paper, SHL.
Guo, J., & Drasgow, F. (2010). Identifying cheating on unproctored internet tests: The z test and the likelihood ratio test. International Journal of Selection and Assessment, 18, 351–364.
Guo, J., Tay, L., & Drasgow, F. (2009). Consipiraces and test compromise: An evaluation of the resistance of test systems to small-scale cheating. International Journal of Testing, 9, 283–309.
International Testing Commission. (2006). International guidelines on computer-based and internet-delivered testsing. International Journal of Testing, 6, 143–171.
Kantrowitz, T. M., Fetzer, M. S., & Dawson, C. R. (2011). Computer adaptive testing (CAT): A faster, smarter, and more secure approach to pre-employment testing. Journal of Business and Psychology, 26, 227–232.
Kantrowitz, T. M., & Gutierrez. S. (2013). The security of employment testing: Practices that keep pace with evolving organizational demands and technology innovations. The Industrial-Organizational Psychologist, 50, 33–42.
Lievens, F., & Burke, E. (2011). Dealing with the threats inherent in unproctored Internet testing of cognitive ability: Results from a large-scale operational test program. Journal of Occupational and Organizational Psychology, 84, 817–824.
Makransky, G., & Glas, C. (2011). Unproctored internet test verification: Using adaptive confirmation testing. Organizational Research Methods, 14, 608–630.
Maynes, D. (2012). Busted! Tricks can be played by anti-cheaters too. Retrieved April 24, 2014, from http://www.caveon.com/busted-tricks-can-be-played-by-anti-cheaters-too/.
Nye, C. D., Do, B. R., Drasgow, F., & Fine, S. (2008). Two-step testing in employee selection: Is score inflation a problem? International Journal of Selection and Assessment, 16, 112–120.
Pace, V. L., & Borman, W. C. (2006). The use of warnings to discourage faking on noncognitive inventories. In R. Griffith (Ed.), A closer examination of faking behavior. Greenwich, CT: Information Age.
Pearlman, K. (2009). Unproctored Internet testing: Practical, legal, and ethical concerns. Industrial and Organizational Psychology: Perspectives on Science and Practice, 1, 14–19.
Reckase, M. D. (1983). A procedure for decision making using tailored testing. In D. J. Weiss (Ed.), New horizons in testing: Latent trait theory and computerized adaptive testing (pp. 237–254). New York: Academic Press.
ATP Test Security Survey Report (2012). Association of Test Publishers.
Rudner, L. M. (2010). GMAT cheaters beware. Business Week. Retrieved from http://www.businessweek.com/bschools/content/nov2010/bs20101112_437248.htm.
Schmidt, F. L., & Hunter, J. E. (1998). The validity and utility of selection methods in personnel psychology: Practical and theoretical implications of 85 years of research findings. Psychological Bulletin, 124, 262–274.
SHL. (2010). Global cognitive index. Technical Report. Alpharetta, GA: SHL.
Stogner, J. M., Miller, B. L., & Marcum, C. D. (2013). Learning to e-cheat: A criminological test of internet facilitated academic cheating. Journal of Criminal Justice Education, 24, 175–199.
Tippins, N. T. (2009). Internet alternatives to traditional proctored testing: Where are we now? Industrial and Organizational Psychology: Perspectives on Science and Practice, 2, 2–10.
Van der Linden, W. J., & Glas, C. A. W. (Eds.). (2000). Computerized adaptive testing: Theory and practice. Boston, MA: Kluwer Academic Publishers.
Whitley, B. (1998). Factors associated with cheating among college students: A review. Research in Higher Education, 39, 235–274.
Wright, N. A., Meade, A. W., & Gutierrez, S. L. (2014). Using invariance to examine cheating in unproctored ability tests. International Journal of Selection and Assessment, 22, 12–22.
Author information
Authors and Affiliations
Corresponding author
Appendix
Appendix
Sample Item 1
Review the facts below:
An Executive Vice President calls each of her five division managers on Monday morning. She calls them in the same order each week.
-
She calls Laura in one of the first two calls.
-
She calls Pat after Jim.
-
She calls Chris before Sandra.
-
She calls Sandra after the third call.
-
She does NOT call Pat last.
Based on the information above, which of these CANNOT be true?
-
A.
She calls Chris third.
-
B.
She calls Jim second.
-
C.
She calls Jim fourth.
-
D.
She calls Pat third.
-
E.
She calls Chris fourth.
The answer is C. The Executive Vice President does not call Pat last. That means that Jim cannot be called fourth, since we know that she calls Pat after Jim.
Sample Item 2
Review the facts below:
-
Jane drives a red car.
-
Susan drives a blue car.
-
There are no red cars in Ohio.
-
Blue cars get 33 miles per gallon of gasoline.
Based on the information above, which of the following MUST be true?
-
A.
Jane lives in Ohio.
-
B.
Susan lives in Ohio.
-
C.
Red cars get 36 miles per gallon of gasoline.
-
D.
Susan’s car gets 33 miles per gallon of gasoline.
-
E.
Jane and Susan live in the same state.
The correct answer is D. Since blue cars get 33 miles per gallon of gasoline, the fact that Susan drives a blue car means that her car gets 33 miles per gallon of gasoline.
Rights and permissions
About this article
Cite this article
Kantrowitz, T.M., Dainis, A.M. How Secure are Unproctored Pre-Employment Tests? Analysis of Inconsistent Test Scores. J Bus Psychol 29, 605–616 (2014). https://doi.org/10.1007/s10869-014-9365-6
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10869-014-9365-6