Journal of Business and Psychology

, Volume 29, Issue 4, pp 555–572 | Cite as

Cheating, Reactions, and Performance in Remotely Proctored Testing: An Exploratory Experimental Study

  • Michael N. Karim
  • Samuel E. Kaminsky
  • Tara S. Behrend
Article

Abstract

Purpose

We sought to provide empirical insight and develop theory for a new organizational phenomenon: remote proctoring for Internet-based tests. We examined whether this technology is effective at decreasing cheating and whether it has unintended effects on test-taker reactions, performance, or selection procedures.

Design/methodology/approach

Participants (582) were randomly assigned to a webcam proctored or honor code condition and completed two (one searchable, one non-searchable) cognitive ability tests online. Complete data were collected from 295 participants. We indirectly determined levels of cheating by examining the pattern of test-score differences across the two conditions. We directly measured dropout rates, test performance, and participants’ perceived tension and invasion of privacy.

Findings

The use of remote proctoring was associated with more negative test-taker reactions and decreased cheating. Remote proctoring did not directly affect test performance or interact with individual differences to predict test performance or test-taker reactions.

Implications

Technological advances in selection should be accompanied by empirical evidence. Although remote proctoring may be effective at decreasing cheating, it may also have unintended effects on test-taker reactions. By outlining an initial classification of remote proctoring technology, we contribute to the theoretical understanding of technology-enhanced assessment, while providing timely insight into the practice of Internet-based testing.

Originality/value

We provide timely insight into the development and evaluation of remotely proctored tests. The current study utilizes a unique randomized experimental design in order to indirectly determine levels of cheating across two conditions. Following the results of the current study, we outline an integrative model for future research on remotely proctored tests.

Keywords

Selection Computer-based testing Unproctored Internet testing Technology-enhanced selection 

References

  1. Aiello, J. R. (1993). Computer-based work monitoring: Electronic surveillance and its effects. Journal of Applied Social Psychology, 23(7), 499–507.Google Scholar
  2. Alder, G. S. (1998). Ethical issues in electronic performance monitoring: A consideration of deontological and teleological perspectives. Journal of Business Ethics, 17(7), 729–743.Google Scholar
  3. Alder, G. S., & Ambrose, M. L. (2005). An examination of the effect of computerized performance monitoring feedback on monitoring fairness, performance, and satisfaction. Organizational Behavior and Human Decision Processes, 97(2), 161–177.Google Scholar
  4. Alge, B. J. (2001). Effects of computer surveillance on perceptions of privacy and procedural justice. Journal of Applied Psychology, 86(4), 797–804. doi:10.1037/0021-9010.86.4.797.PubMedCrossRefGoogle Scholar
  5. Alge, B. J., Ballinger, G. A., Tangirala, S., & Oakley, J. (2006). Information privacy in organizations: Empowering creative and extrarole performance. Journal of Applied Psychology, 91(1), 221–232. doi:10.1037/0021-9010.91.1.221.PubMedCrossRefGoogle Scholar
  6. Arch, E. C. (1987). Differential responses of females and males to evaluative stress: Anxiety, self-esteem, efficacy, and willingness to participate. In R. Schwarzer, H. Van der Ploeg, & C. Spielberger (Eds.), Advances in test anxiety research (Vol. 5, pp. 97–106). Isse: Swets & Zeitlinger.Google Scholar
  7. Arthur, W, Jr, Glaze, R. M., Villado, A. J., & Taylor, J. E. (2009). Unproctored internet-based tests of cognitive ability and personality: Magnitude of cheating and response distortion. Industrial and Organizational Psychology, 2(1), 39–45. doi:10.1111/j.1754-9434.2008.01105.x.CrossRefGoogle Scholar
  8. Bauer, T. N., Truxillo, D. M., Sanchez, R. J., Craig, J. M., Ferrara, P., & Campion, M. A. (2001). Applicant reactions to selection: Development of the selection procedural justice scale (SPJS). Personnel Psychology, 54(2), 387–419. doi:10.1111/j.1744-6570.2001.tb00097.x.CrossRefGoogle Scholar
  9. Beaty, J. C., Nye, C. D., Borneman, M. J., Kantrowitz, T. M., Drasgow, F., & Grauer, E. (2011). Proctored versus unproctored Internet tests: Are unproctored noncognitive tests as predictive of job performance? International Journal of Selection and Assessment, 19(1), 1–10. doi:10.1111/j.1468-2389.2011.00529.x.CrossRefGoogle Scholar
  10. Behrend, T. S., Sharek, D. J., Meade, A. W., & Wiebe, E. N. (2011). The viability of crowdsourcing for survey research. Behavior Research Methods, 43(3), 800–813. doi:10.3758/s13428-011-0081-0.PubMedCrossRefGoogle Scholar
  11. Bond, C. F., & Titus, L. J. (1983). Social facilitation: A meta-analysis of 241 studies. Psychological Bulletin, 94(2), 265–292. doi:10.1037//0033-2909.94.2.265.PubMedCrossRefGoogle Scholar
  12. Buchanan, T., & Smith, J. L. (1999). Using the internet for psychological research: personality testing on the world wide web. British Journal of Psychology, 90(1), 125–144. doi:10.1348/000712699161189.PubMedCrossRefGoogle Scholar
  13. Buhrmester, M. D., Kwang, T., & Gosling, S. D. (2011). Amazon’s Mechanical Turk: A new source of inexpensive, yet high-quality, data? Perspectives on Psychological Science, 6(1), 3–5. doi:10.1177/1745691610393980.CrossRefGoogle Scholar
  14. Chalykoff, J., & Kochan, T. A. (1989). Computer-aided monitoring: Its influence on employee job satisfaction and turnover. Personnel Psychology, 42(4), 807–834. doi:10.1111/j.1744-6570.1989.tb00676.x.CrossRefGoogle Scholar
  15. Davidson, R., & Henderson, R. (2000). Electronic performance monitoring: A laboratory investigation of the influence of monitoring and difficulty on task performance, mood state, and self-reported stress levels. Journal of Applied Social Psychology, 30(5), 906–920. doi:10.1111/j.1559-1816.2000.tb02502.x.CrossRefGoogle Scholar
  16. Deci, E. L., Eghrarl, H., Patrick, B. C., & Leone, D. R. (1994). Facilitating internalization: The self-determination theory perspective. Journal of Personality, 62(1), 119–142. doi:10.1111/j.1467-6494.1994.tb00797.x.PubMedCrossRefGoogle Scholar
  17. DeShon, R. P., & Gillespie, J. Z. (2005). A motivated action theory account of goal orientation. Journal of Applied Psychology, 90(6), 1096–1127.Google Scholar
  18. DeTienne, K. B. (1993). Big brother or friendly coach? Computer monitoring in the 21st century. The Futurist, 27(5), 33–37.Google Scholar
  19. Drasgow, F., Nye, C. D., Guo, J., & Tay, L. (2009). Cheating on proctored tests: The other side of the unproctored debate. Industrial and Organizational Psychology, 2(1), 46–48. doi:10.1111/j.1754-9434.2008.01106.x.CrossRefGoogle Scholar
  20. Eisenberg, A. (2013). Keeping an eye on online test-takers. The New York Times. Retrieved May 9, 2013, from http://www.nytimes.com/2013/03/03/technology/new-technologies-aim-to-foil-online-course-cheating.html.
  21. Foster, D. (2009). Secure, online, high-stakes testing: Science fiction or business reality? Industrial and Organizational Psychology, 2(1), 31–34. doi:10.1111/j.1754-9434.2008.01103.x.CrossRefGoogle Scholar
  22. George, C. E., Lankford, J. S., & Wilson, S. E. (1992). The effects of computerized versus paper-and-pencil administration on measures of negative affect. Computers in Human Behavior, 8(2–3), 203–209. doi:10.1016/0747-5632(92)90004-X.CrossRefGoogle Scholar
  23. Hembree, R. (1988). Correlates, causes, effects, and treatment of test anxiety. Review of Educational Research, 58(1), 47–77. doi:10.3102/00346543058001047.CrossRefGoogle Scholar
  24. Hong, S.-M., & Page, S. (1989). A psychological reactance scale: Development, factor structure and reliability. Psychological Reports, 64(3c), 1323–1326. doi:10.2466/pr0.1989.64.3c.1323.CrossRefGoogle Scholar
  25. Kolb, K. J., & Aiello, J. R. (1996). The effects of electronic performance monitoring on stress: Locus of control as a moderator variable. Computers in Human Behavior, 12(3), 407–423.Google Scholar
  26. Landers, R. N., & Sackett, P. R. (2012). Offsetting performance losses due to cheating in unproctored Internet-based testing by increasing the applicant pool. International Journal of Selection and Assessment, 20(2), 220–228. doi:10.1111/j.1468-2389.2012.00594.x.CrossRefGoogle Scholar
  27. Leary, M. R. (1983). A brief version of the fear of negative evaluation scale. Personality and Social Psychology Bulletin, 9(3), 371–375. doi:10.1177/0146167283093007.CrossRefGoogle Scholar
  28. Lievens, F., & Burke, E. (2011). Dealing with the threats inherent in unproctored Internet testing of cognitive ability: Results from a large-scale operational test program. Journal of Occupational and Organizational Psychology, 84(4), 817–824. doi:10.1348/096317910X522672.CrossRefGoogle Scholar
  29. Macan, T. H., Avedon, M. J., Paese, M., & Smith, D. E. (1994). The effects of applicants’ reactions to cognitive ability tests and an assessment center. Personnel Psychology, 47(4), 715–738. doi:10.1111/j.1744-6570.1994.tb01573.x.CrossRefGoogle Scholar
  30. McCarthy, J. M., Van Iddekinge, C. H., Lievens, F., Kung, M. C., Sinar, E. F., & Campion, M. A. (2013). Do candidate reactions relate to job performance or affect criterion-related validity? A multistudy investigation of relations among reactions, selection test scores, and job performance. Journal of Applied Psychology, 98(5), 701–719. doi:10.1037/a0034089.PubMedCrossRefGoogle Scholar
  31. McNall, L. A., & Roch, S. (2007). Effects of electronic monitoring types on perceptions of procedural justice, interpersonal justice, and privacy. Journal of Applied Social Psychology, 37(3), 25. doi:10.1111/j.1559-1816.2007.00179.x.CrossRefGoogle Scholar
  32. Mead, A. D., & Drasgow, F. (1993). Equivalence of computerized and paper-and-pencil cognitive ability tests: A meta-analysis. Psychological Bulletin, 114(3), 449–458. doi:10.1037//0033-2909.114.3.449.CrossRefGoogle Scholar
  33. Moore, C., Detert, J. R., Treviño, L. K., Baker, V. L., & Mayer, D. M. (2012). Why employees do bad things: Moral disengagement and unethical organizational behavior. Personnel Psychology, 65(1), 1–48. doi:10.1111/j.1744-6570.2011.01237.x.CrossRefGoogle Scholar
  34. Naglieri, J. A., Drasgow, F., Schmit, M., Handler, L., Prifitera, A., Margolis, A., et al. (2004). Psychological testing on the internet: New problems, old issues. American Psychologist, 59(3), 150–162. doi:10.1037/0003-066X.59.3.150.PubMedCrossRefGoogle Scholar
  35. Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010). Running experiments on Amazon Mechanical Turk. Judgment and Decision Making, 5(5), 411–419.Google Scholar
  36. Prabhakar, S., Pankanti, S., & Jain, A. K. (2003). Biometric recognition: Security and privacy concerns. IEEE Security and Privacy, 1(2), 33–42. doi:10.1109/MSECP.2003.1193209.CrossRefGoogle Scholar
  37. Raven, J., Raven, C. J., & Court, H. J. (2003). Manual for Raven’s Progressive Matrices and Vocabulary Scales, Section 3. San Antonio, TX: Harcourt Assessment.Google Scholar
  38. Reynolds, D. H., & Dickter, D. N. (2010). Technology and employee selection. In J. L. Farr & N. T. Tippins (Eds.), Handbook of Employee Selection. Clifton, NJ: Psychological Press.Google Scholar
  39. Reynolds, D. H., & Weiner, J. A. (2009). Online Recruiting and Selection: Innovations in Talent Acquisition. Chichester: Wiley-Blackwell.CrossRefGoogle Scholar
  40. Ryan, R. M., & Deci, E. L. (2000). Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. American Psychologist, 55(1), 68–78.Google Scholar
  41. Schleifer, L. M., & Shell, R. L. (1992). A review and reappraisal of electronic performance monitoring, performance standards and stress allowances. Applied Ergonomics, 23(1), 45–53. doi:10.1016/0003-6870(92)90010-S.Google Scholar
  42. Smither, J. W., Reilly, R. R., Millsap, R. E., Pearlman, K., & Stoffey, R. W. (1993). Applicant reactions to selection procedures. Personnel Psychology, 46(1), 49–76. doi:10.1111/j.1744-6570.1993.tb00867.x.CrossRefGoogle Scholar
  43. Stanton, J. M. (2000). Reactions to employee performance monitoring: Framework, review, and research directions. Human Performance, 13(1), 85–113. doi:10.1207/S15327043HUP1301_4.CrossRefGoogle Scholar
  44. Stone, E. F., Gueutal, H. G., Gardner, D. G., & McClure, S. (1983). A field experiment comparing information-privacy values, beliefs, and attitudes across several types of organizations. Journal of Applied Psychology, 68(3), 459–468. doi:10.1037//0021-9010.68.3.459.CrossRefGoogle Scholar
  45. Tendeiro, J. N., Meijer, R. R., Schakel, L., & Meij, A. M. M.-D. (2012). Using cumulative sum statistics to detect inconsistencies in unproctored Internet testing. Educational and Psychological Measurement, 73(1), 143–161. doi:10.1177/0013164412444787.CrossRefGoogle Scholar
  46. Thompson, L. F., Sebastianelli, J. D., & Murray, N. P. (2009). Monitoring online training behaviors: Awareness of electronic surveillance hinders E-learners. Journal of Applied Social Psychology, 39(9), 2191–2212.Google Scholar
  47. Tippins, N. T. (2009). Internet alternatives to traditional proctored testing: Where are we now? Industrial and Organizational Psychology, 2(1), 2–10. doi:10.1111/j.1754-9434.2008.01097.x.CrossRefGoogle Scholar
  48. Tippins, N. T., Beaty, J. C., Drasgow, F., Gibson, W. M., Pearlman, K., Segall, D. O., et al. (2006). Unproctored internet testing in employment settings. Personnel Psychology, 59(1), 189–225. doi:10.1111/j.1744-6570.2006.00909.x.CrossRefGoogle Scholar
  49. Vandewalle, D. (1997). Development and validation of a work domain goal orientation instrument. Educational and Psychological Measurement, 57(6), 995–1015. doi:10.1177/0013164497057006009.CrossRefGoogle Scholar
  50. Watson, A. M., Thompson, L. F., Rudolph, J. V., Whelan, T. J., Behrend, T. S., & Gissel, A. L. (2013). When big brother is watching: Goal orientation shapes reactions to electronic monitoring during online training. Journal of Applied Psychology,. doi:10.1037/a0032002.PubMedGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2014

Authors and Affiliations

  • Michael N. Karim
    • 1
  • Samuel E. Kaminsky
    • 1
  • Tara S. Behrend
    • 1
  1. 1.Department of Organizational Sciences and CommunicationThe George Washington UniversityWashingtonUSA

Personalised recommendations