Cheating, Reactions, and Performance in Remotely Proctored Testing: An Exploratory Experimental Study
- 889 Downloads
We sought to provide empirical insight and develop theory for a new organizational phenomenon: remote proctoring for Internet-based tests. We examined whether this technology is effective at decreasing cheating and whether it has unintended effects on test-taker reactions, performance, or selection procedures.
Participants (582) were randomly assigned to a webcam proctored or honor code condition and completed two (one searchable, one non-searchable) cognitive ability tests online. Complete data were collected from 295 participants. We indirectly determined levels of cheating by examining the pattern of test-score differences across the two conditions. We directly measured dropout rates, test performance, and participants’ perceived tension and invasion of privacy.
The use of remote proctoring was associated with more negative test-taker reactions and decreased cheating. Remote proctoring did not directly affect test performance or interact with individual differences to predict test performance or test-taker reactions.
Technological advances in selection should be accompanied by empirical evidence. Although remote proctoring may be effective at decreasing cheating, it may also have unintended effects on test-taker reactions. By outlining an initial classification of remote proctoring technology, we contribute to the theoretical understanding of technology-enhanced assessment, while providing timely insight into the practice of Internet-based testing.
We provide timely insight into the development and evaluation of remotely proctored tests. The current study utilizes a unique randomized experimental design in order to indirectly determine levels of cheating across two conditions. Following the results of the current study, we outline an integrative model for future research on remotely proctored tests.
KeywordsSelection Computer-based testing Unproctored Internet testing Technology-enhanced selection
The authors are grateful to Michael Acquah, Cecilia Ramirez, and The George Washington University’s Workplaces and Virtual Environments (WAVE) lab for their assistance in study design and to Shedon Zedeck, Frederick Oswald, and two anonymous reviewers for their insightful comments and feedback on earlier revisions of this manuscript.
- Aiello, J. R. (1993). Computer-based work monitoring: Electronic surveillance and its effects. Journal of Applied Social Psychology, 23(7), 499–507.Google Scholar
- Alder, G. S. (1998). Ethical issues in electronic performance monitoring: A consideration of deontological and teleological perspectives. Journal of Business Ethics, 17(7), 729–743.Google Scholar
- Alder, G. S., & Ambrose, M. L. (2005). An examination of the effect of computerized performance monitoring feedback on monitoring fairness, performance, and satisfaction. Organizational Behavior and Human Decision Processes, 97(2), 161–177.Google Scholar
- Arch, E. C. (1987). Differential responses of females and males to evaluative stress: Anxiety, self-esteem, efficacy, and willingness to participate. In R. Schwarzer, H. Van der Ploeg, & C. Spielberger (Eds.), Advances in test anxiety research (Vol. 5, pp. 97–106). Isse: Swets & Zeitlinger.Google Scholar
- Arthur, W, Jr, Glaze, R. M., Villado, A. J., & Taylor, J. E. (2009). Unproctored internet-based tests of cognitive ability and personality: Magnitude of cheating and response distortion. Industrial and Organizational Psychology, 2(1), 39–45. doi: 10.1111/j.1754-9434.2008.01105.x.CrossRefGoogle Scholar
- Beaty, J. C., Nye, C. D., Borneman, M. J., Kantrowitz, T. M., Drasgow, F., & Grauer, E. (2011). Proctored versus unproctored Internet tests: Are unproctored noncognitive tests as predictive of job performance? International Journal of Selection and Assessment, 19(1), 1–10. doi: 10.1111/j.1468-2389.2011.00529.x.CrossRefGoogle Scholar
- Davidson, R., & Henderson, R. (2000). Electronic performance monitoring: A laboratory investigation of the influence of monitoring and difficulty on task performance, mood state, and self-reported stress levels. Journal of Applied Social Psychology, 30(5), 906–920. doi: 10.1111/j.1559-1816.2000.tb02502.x.CrossRefGoogle Scholar
- DeShon, R. P., & Gillespie, J. Z. (2005). A motivated action theory account of goal orientation. Journal of Applied Psychology, 90(6), 1096–1127.Google Scholar
- DeTienne, K. B. (1993). Big brother or friendly coach? Computer monitoring in the 21st century. The Futurist, 27(5), 33–37.Google Scholar
- Eisenberg, A. (2013). Keeping an eye on online test-takers. The New York Times. Retrieved May 9, 2013, from http://www.nytimes.com/2013/03/03/technology/new-technologies-aim-to-foil-online-course-cheating.html.
- Kolb, K. J., & Aiello, J. R. (1996). The effects of electronic performance monitoring on stress: Locus of control as a moderator variable. Computers in Human Behavior, 12(3), 407–423.Google Scholar
- McCarthy, J. M., Van Iddekinge, C. H., Lievens, F., Kung, M. C., Sinar, E. F., & Campion, M. A. (2013). Do candidate reactions relate to job performance or affect criterion-related validity? A multistudy investigation of relations among reactions, selection test scores, and job performance. Journal of Applied Psychology, 98(5), 701–719. doi: 10.1037/a0034089.PubMedCrossRefGoogle Scholar
- Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010). Running experiments on Amazon Mechanical Turk. Judgment and Decision Making, 5(5), 411–419.Google Scholar
- Raven, J., Raven, C. J., & Court, H. J. (2003). Manual for Raven’s Progressive Matrices and Vocabulary Scales, Section 3. San Antonio, TX: Harcourt Assessment.Google Scholar
- Reynolds, D. H., & Dickter, D. N. (2010). Technology and employee selection. In J. L. Farr & N. T. Tippins (Eds.), Handbook of Employee Selection. Clifton, NJ: Psychological Press.Google Scholar
- Ryan, R. M., & Deci, E. L. (2000). Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. American Psychologist, 55(1), 68–78.Google Scholar
- Thompson, L. F., Sebastianelli, J. D., & Murray, N. P. (2009). Monitoring online training behaviors: Awareness of electronic surveillance hinders E-learners. Journal of Applied Social Psychology, 39(9), 2191–2212.Google Scholar