Commercial content moderators review user-generated content (UGC) to ensure the posts meet platform policies, guidelines, community standards, and regional laws. While the majority of posted UGC is deemed acceptable, a large amount of content remains, which is classified as non-compliant and may include gore, violence, suicide, child sexual abuse material (CSAM), and pornography, to name a few. Because of this, content moderators have a greater prevalence of their nervous system activating a startle response, which can impact emotional, psychological, and physiological processes. Prior research on content moderators has failed to explore moderators’ initial reactions to content from the start of employment through tenure as the subjection to material and habituation increases. This study takes an in-depth look at moderators’ experiences from recruiting, through training, and production to better understand the content moderators’ startle response and factors that enable startle habituation. The current study sample consisted of 78 total respondents—38 content moderators in the Philippines employed by TaskUs Inc. and 40 in Estonia employed by Sutherland Global Services. Employee tenure ranged from 0 to 6 months. Succeeding our analysis, transparency, understanding, and preparedness were major themes identified as the critical factors found within both companies when exploring the activation of the startle response and facilitation of habituation following content exposure. These themes were prevalent in the employment life-cycle's recruiting, training, and production phases.
- Commercial content moderation
- Startle response
- Startle habituation
- Employee life cycle
This is a preview of subscription content, access via your institution.
Tax calculation will be finalised at checkout
Purchases are for personal use onlyLearn about institutional subscriptions
Arsht A, Etcovitch D (2018) The human cost of online content moderation. Harvard Law Review Online, Harvard University, Cambridge, MA, USA. Retrieved from https://jolt.law.harvard.edu/digest/the-human-cost-ofonline-contentmoderation
Beg M (2015) Socio-physicochemical interpretation of theory on emotions. https://doi.org/10.13140/RG.2.1.1825.4241
Bradley MM, Codispoti M, Cuthbert BN, Lang PJ (2001). Emotion and motivation I: defensive and appetitive reactions in picture processing. Emotion 276–298
Carmi E (2019) The hidden listeners: regulating the line from telephone operators to content moderators. Int J Commun 13:440–458
Newton C (2019) The trauma floor. The Verge. Retrieved from https://www.theverge.com/2019/2/25/18229714/cognizant-facebook-content-moderator-interviews-trauma-working-conditions-arizona
Chandrasekharan E, Gandhi C, Mustelier MW, Gilbert E (2019) Crossmod: a cross-community learning-based system to assist reddit moderators. In: Proceedings of the ACM on human-computer interaction, Vol 3(CSCW), pp 1–30
Chen A (2014) The laborers who keep dick pics and beheadings out of your facebook feed. Wired. Retrieved from https://www.wired.com/2014/10/content-moderation/
Chen GD, Radziwon KE, Kashanian N, Manohar S, Salvi R (2014) Salicylate-induced auditory perceptual disorders and plastic changes in nonclassical auditory centers in rats. Neural Plast 2014:658741
Chotiner I (2019) The underworld of online content moderation. The New Yorker. Retrieved from https://www.newyorker.com/news/q-and-a/the-underworld-of-online-content-moderation
Dang B, Riedl MJ, Lease M (2018) But who protects the moderators? The case of crowdsourced image moderation. Cornell University
De la Casa LG, Mena A, Ruiz-Salas JC (2016) Effect of stress and attention on startle response and prepulse inhibition. Physiol Behav 165:179–186
Deuter CE, Kuehl LK, Blumenthal TD, Schulz A, Oitzl MS, Schachinger H (2012) Effects of cold pressor stress on the human startle response. PloS One 7(11)
Gorwa R, Binns R, Katzenbach C (2020) Algorithmic content moderation: technical and political challenges in the automation of platform governance. Big Data Soc 7(1):2053951719897945
Hartman ME, Ladwig MA, Ekkekakis P (2021) Contactless differentiation of pleasant and unpleasant valence: asssessment of the acoustic startle eye blink response with infrared reflectance oculography. Behav Res Meth 1–13
Kiger ME, Varpio L (2020) Thematic analysis of qualitative data: Amee Guide no. 131. Med Teach 42(8):846–854. https://doi.org/10.1080/0142159x.2020.1755030
Lang PJ, Bradley MM, Cuthbert BN (1990) Emotion, attention, and the startle reflex. Psychol Rev 97(3):377–395
Maguire M, Delahunt B (2017) Doing a thematic analysis: a practical, step-by-step guide for learning and teaching scholars. All Ireland J Teach Learn High Educ 9(3):3351–33514. http://ojs.aishe.org/index.php/aishe-j/article/view/335
Martin WL, Murray PS, Bates PR (2012) The effects of startle on pilots during critical events: a case study analysis. In: Aviation psychology and applied human factors—working toward sezo accidents Conference 2012. Griffith University, pp 387–394
Mukhopadhyay B (2020) Warning: the (open) secret lives of content moderators. Cassandra voices. https://cassandravoices.com/society-culture/society/warning-the-open-secret-lives-of-content-moderators/
Nielsen Norman Group (2000) Why you only need to test with 5 users. Nielsen Norman Group. Retrieved from https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/#:~:text=Not%20true
Phelps BJ, Doyle-Lunders L, Harsin-Waite A, Hofman N, Knutson LM (2012) Demonstrating habituation of a startle response to loud noise. Behav Anal Today 13(1):17–19. https://doi.org/10.1037/h0100714
Reagh ZM, Knight DC (2013) Negative, but not positive emotional images modulate the startle response independent of conscious awareness. Emotion 13(4):782
Roberts ST (2016) Commercial content moderation: digital laborers’ dirty work. Western Lib 12
Schachter S, Singer J (1962) Cognitive, social, and physiological determinants of emotional state. Psychol Rev 69(5):379–399. https://doi.org/10.1037/h0046234
Silbermann J (2020) They’re looking out for the internet. But who’s looking out for them? Onlabor. https://onlabor.org/theyre-looking-out-for-the-internet-but-whos-looking-out-for-them/
Steiger M (2020) Building a resilient workforce: programming for commercial content moderation staff.
Steiger M, Bharucha TJ, Venkatagiri S, Riedl MJ, Lease M (2021) The psychological well-being of content moderators: the emotional labor of commercial moderation and avenues for improving support. In Proceedings of the 2021 CHI conference on human factors in computing systems, pp 1–14
Vaidyanathan U, Malone SM, Miller MB, McGue M, Iacono WG (2014) Heritability and molecular genetic basis of acoustic startle eye blink and affectively modulated startle response: a genome-wide association study. Psychophysiology 51(12):1285–1299
Valsamis B, Schmid S (2011) Habituation and prepulse inhibition of acoustic startle in rodents. J Vis Exp 55:e3446
The authors express deep gratitude for those content moderators who participated in the research and so willingly shared their experiences. With this knowledge, we can better support the well-being of all reviewers. Additionally, the authors would like to thank content moderators worldwide for the tremendous and much-needed work that they do safeguarding the internet and social media platforms for the end-user.
The research study was conducted without any external funding (Table 1).
Editors and Affiliations
TaskUs Inc. and Sutherland Global Services employ the authors and content moderators interviewed for the study.
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Bharucha, T., Steiger, M.E., Manchanda, P., Mere, R., Huang, X. (2024). Content Moderator Startle Response: A Qualitative Study. In: Yang, XS., Sherratt, R.S., Dey, N., Joshi, A. (eds) Proceedings of Eighth International Congress on Information and Communication Technology. ICICT 2023. Lecture Notes in Networks and Systems, vol 695. Springer, Singapore. https://doi.org/10.1007/978-981-99-3043-2_18
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-3042-5
Online ISBN: 978-981-99-3043-2