Skip to main content

Advertisement

SpringerLink
Log in

Using Mechanical Turk to Study Family Processes and Youth Mental Health: A Test of Feasibility

  • Original Paper
  • Published:
Journal of Child and Family Studies Aims and scope Submit manuscript

Abstract

Extensive evidence links youth mental health to family functioning, highlighting the need to document causal pathways. This will require longitudinal studies, but traditional methods for longitudinal clinical research have several limitations, including high cost and resource demands, underrepresentation of fathers, and attrition bias. We tested whether an online alternative might address limitations of—and thus provide a useful complement to—traditional methods. We used the Mechanical Turk (MTurk) survey program to obtain reports from parents (N = 177) on family functioning, the parents’ own symptoms, their children’s behavioral and emotional problems, and parenting stress, with assessments in three consecutive months. Parents provided largely high-quality data (e.g., passed consistency checks); measures showed acceptable psychometrics at each time-point; and correlations among study measures paralleled those observed in prior research. Compared to prior studies using traditional longitudinal methods, the MTurk method was (a) much lower in cost and resource requirements, (b) successful in enrolling fathers, (c) comparable in participant attrition, and (c) similar in attrition bias, participant race/ethnicity, and enrollment of single parents. Overall, findings suggest that MTurk is a viable tool with its own set of strengths and limitations, and a potentially useful complement to traditional longitudinal methods. In particular, MTurk might be a cost-effective first step in generating causal hypotheses about family processes and youth mental health, for later testing via more traditional methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Abidin, R. (1986). Parenting stress index (2nd ed.). Charlottesville, VA: Pediatric Psychology Press.

    Google Scholar 

  • Abidin, R. R. (1995). Parenting stress index: Professional manual (3rd ed.). Odessa, FL: Psychological Assessment Resources.

    Google Scholar 

  • Achenbach, T. M. (1991). Integrative guide for the 1991 CBCL/4-18, YSR, and TRF profiles. Burlington: Department of Psychiatry, University of Vermont.

    Google Scholar 

  • Adler, N. E., Epel, E. S., Castellazzo, G., & Ickovics, J. R. (2000). Relationship of subjective and objective social status with psychological and physiological functioning: Preliminary data in healthy, White women. Health Psychology, 19, 586–592.

    Article  PubMed  Google Scholar 

  • Arbisi, P. A., & Ben-Porath, Y. S. (1995). An MMPI-2 infrequent response scale for use with psychopathological populations: The infrequency-psychopathology scale. Psychological Assessment, 7, 424.

  • Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012). Evaluating online labor markets for experimental research: Amazon.com’s Mechanical Turk. Political Analysis, 20, 351–368.

    Article  Google Scholar 

  • Berry, J. O., & Jones, W. H. (1995). The parental stress scale: Initial evidence. Journal of Social and Personal Relationships, 12, 463–472.

    Article  Google Scholar 

  • Bianchi, S. M., & Milkie, M. A. (2010). Work and family research in the first decade of the 21st century. Journal of Marriage and Family, 72, 705–725.

    Article  Google Scholar 

  • Bögels, S. M., & Brechman-Toussaint, M. L. (2006). Family issues in child anxiety: Attachment, family functioning, parental rearing and beliefs. Clinical Psychology Review, 26, 834–856.

    Article  PubMed  Google Scholar 

  • Brannan, A. M., Heflinger, C. A., & Bickman, L. (1997). The Caregiver Strain Questionnaire: Measuring the impact on the family of living with a child with serious emotional disturbance. Journal of Emotional and Behavioral Disorders, 5, 212–222.

  • Burstein, M., Ginsburg, G. S., & Tein, J.-Y. (2010). Parental anxiety and child symptomatology: An examination of additive and interactive effects of parent psychopathology. Journal of Abnormal Child Psychology, 38, 897–909.

    Article  PubMed Central  PubMed  Google Scholar 

  • Cassileth, B. R., Lusk, E. J., Strouse, T. B., Miller, D. S., Brown, L. L., Cross, P. A., & Tenaglia, A. (1984). Psychosocial status in chronic illness. The New England Journal of Medicine, 311, 506–511.

    Article  PubMed  Google Scholar 

  • Chandler, J., Mueller, P., & Paolacci, G. (2013). Nonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers. Behavior Research Methods, 46, 112–130.

    Article  Google Scholar 

  • Chorpita, B. F., & Barlow, D. H. (1998). The development of anxiety: The role of control in the early environment. Psychological Bulletin, 124, 3–21.

    Article  PubMed  Google Scholar 

  • Chorpita, B. F., Brown, T. A., & Barlow, D. H. (1998). Perceived control as a mediator of family environment in etiological models of childhood anxiety. Behavior Therapy, 29, 457–476.

    Article  Google Scholar 

  • Cohen, S., Kamarck, T., & Mermelstein, R. (1983). A global measure of perceived stress. Journal of Health and Social Behavior, 24, 385–396.

    Article  PubMed  Google Scholar 

  • Conger, R. D., Elder, G. H, Jr, Lorenz, F. O., Simons, R. L., & Whitbeck, L. B. (1994). Families in troubled times. New York: De Gruyter.

    Google Scholar 

  • Cooper, C. E., Mclanahan, S. S., Meadows, S. O., & Brooks-gunn, J. (2010). Family structure transitions and maternal parenting stress. Journal of Marriage and Family, 71(3), 558–574.

    Article  Google Scholar 

  • Costa, N. M. & Weems, C. F., Pellerin, K., & Dalton, R. (2006). Parenting stress: An examination of specificity to internalizing and externalizing symptoms. Journal Of Psychopathology and Behavioral Assessment, 28, 113–122.

  • Costello, E. J., Compton, S. N., Keeler, G., & Angold, A. (2003). Relationships between poverty and psychopathology: A natural experiment. JAMA, 290, 2023–2029.

    Article  PubMed  Google Scholar 

  • Cotter, R. B., Burke, J. D., Stouthamer-Loeber, M., & Loeber, R. (2005). Contacting participants for follow-up: How much effort is required to retain participants in longitudinal studies? Evaluation and Program Planning, 28, 15–21.

    Article  Google Scholar 

  • Crawford, A. M., & Manassis, K. (2001). Familial predictors of treatment outcome in childhood anxiety disorders. Journal of the American Academy of Child and Adolescent Psychiatry, 40, 1182–1189.

  • Day, R. D., & Lamb, M. E. (2004). Measuring and conceptualizing fatherhood involvement. Mahwah, NJ: Erlbaum.

  • De Graaf, R., Bijl, R. V., Smit, F., Ravelli, A., & Vollebergh, W. A. (2000). Psychiatric and socio-demographic predictors of attrition in a longitudinal. American Journal of Epidemiology, 152, 1039–1047.

    Article  PubMed  Google Scholar 

  • De Los Reyes, A., & Kazdin, A. E. (2005). Informant discrepancies in assessment in childhood psychopathology: A critical review. Psychological Bulletin, 131, 483–509.

    Article  PubMed  Google Scholar 

  • Dearing, E., Taylor, B. A., & McCartney, K. (2004). Implications of family income dynamics for women’s depressive symptoms during the first 3 years after childbirth. American Journal of Public Health, 94, 1372–1377.

    Article  PubMed Central  PubMed  Google Scholar 

  • Derogatis, L. R., Lipman, R. S., & Covi, L. (1973). SCL-90: An outpatient psychiatric rating scale-preliminary report. Psychopharmacology Bulletin, 9, 13–17.

  • Duhig, A. M., Renk, K., Epstein, M. E., & Phares, V. (2000). Interparental agreement on internalizing, externalizing, and total behavior problems: A meta-analysis. Clinical Psychology: Science and Practice, 7, 435–453.

    Google Scholar 

  • Dunst, C. J., Trivette, C. M., & Deal, A. G. (1988). Enabling and empowering families: Principles and guidelines for practice. Cambridge, MA: Brookline Books.

    Google Scholar 

  • Eisenberg, N., Zhou, Q., Spinrad, T. L., Valiente, C., & Fabes, R. A. (2005). Relations among positive parenting, children’s effortful control, and externalizing problems: A three-wave longitudinal study. Child Development, 76, 1055–1071.

    Article  PubMed Central  PubMed  Google Scholar 

  • Elander, J., & Rutter, M. (1996). Use and development of the Rutter parents’ and teachers’ scales. International Journal of Methods in Psychiatric Research, 6, 63–78.

    Article  Google Scholar 

  • Eriksson, K., & Simpson, B. (2010). Emotional reactions to losing explain gender differences in entering a risky lottery. Judgment and Decision Making, 5, 159–163.

  • Gibson, E., Piantadosi, S., & Fedorenko, K. (2011). Using Mechanical Turk to obtain and analyze English acceptability judgments. Language and Linguistics Compass, 5, 509–524.

  • Ginsburg, G. S., Siqueland, L., Masia-Warner, C., & Hedtke, K. A. (2004). Anxiety disorders in children: Family matters. Cognitive and Behavioral Practice, 11, 408–412.

    Article  Google Scholar 

  • Goodman, J. S. (1996). Assessing the non-random sampling effects of subject attrition in longitudinal research. Journal of Management, 22, 627–652.

  • Goodman, R. (1997). The strengths and difficulties questionnaire: A research note. Journal of Child Psychology and Psychiatry, 38, 581–586.

    Article  PubMed  Google Scholar 

  • Goodman, S. H., Rouse, M. H., Connell, A. M., Broth, M. R., Hall, C. M., & Heyward, D. (2011). Maternal depression and child psychopathology: A meta-analytic review. Clinical Child and Family Psychology Review, 14, 1–27.

    Article  PubMed  Google Scholar 

  • Goodman, R., & Scott, S. (1999). Comparing the strengths and difficulties questionnaire and the child behavior checklist: Is small beautiful? Journal of Abnormal Child Psychology, 27, 17–24.

    Article  PubMed  Google Scholar 

  • Griffin, C., Guerin, S., Sharry, J., & Drumm, M. (2010). A multicentre controlled study of an early intervention parenting programme for young children with behavioural and developmental difficulties. International Journal of Clinical Health and Psychology, 10, 279–294.

    Google Scholar 

  • Hammen, C. (2009). Adolescent depression: Stressful interpersonal contexts and risk for recurrence. Current Directions in Psychological Science, 18, 200–204.

    Article  PubMed Central  PubMed  Google Scholar 

  • Horton, J. J., & Chilton, L. B. (2010). The labor economics of paid crowdsourcing. In Proceedings from EC’10: The 11th ACM conference on electronic commerce (pp. 209–218). New York, NY: ACM.

  • Kessler, R. C., Little, R. J. A., & Groves, R. M. (1995). Advances in strategies for minimizing and adjusting for survey nonresponse. Epidemiological Review, 17, 192–204.

    Google Scholar 

  • Lamb, M. E. (Ed.). (2010). The role of the father in child development (5th ed.). Hoboken NJ: Wiley.

    Google Scholar 

  • Lease, M., Hullman, J., Bigham, J. P., Bernstein, M. S., Kim, K., Lasecki, W., et al. (2013). Mechanical Turk is not anonymous. Social Science Research Network. doi:10.2139/ssrn.2228728.

  • Mason, W., & Suri, S. (2012). Conducting behavioral research on Amazon’s Mechanical Turk. Behavior Research Methods, 44, 1–23.

    Article  PubMed  Google Scholar 

  • Mason, W. A., & Watts, D. J. (2009). Financial incentives and the “Performance of crowds.” In Proceedings of the Human Computation Workshop. Paris: ACM.

  • McQuaid, E., & Barakat, L. (2012). Introduction to special section: Advancing research on the intersection of families, culture, and health outcomes. Journal of Pediatric Psychology, 37(8), 827–831.

    Article  PubMed Central  PubMed  Google Scholar 

  • Miller, R. B. & Hollist, C. S. (2007). Attrition bias. Faculty Publications, Department of Child, Youth, and Family Studies. http://digitalcommons.unl.edu/famconfacpub/45

  • Paolacci, G., & Chandler, J. (2014). Inside the Turk: Understanding Mechanical Turk as a participant pool. Current Directions in Psychological Science, 23, 184–188.

  • Paolacci, G., Chandler, J., & Ipeirotis, P. (2010). Running experiments on Amazon Mechanical Turk. Judgment and Decision Making, 5, 411–419.

    Google Scholar 

  • Pe’er, E., Vosgerau, J., & Acquisti, A. (2014). Reputation as a sufficient condition for data quality on Amazon Mechanical Turk. Behavior Research Methods, 46, 1023–1031.

    Article  Google Scholar 

  • Phares, V. (1992). Where’s poppa? The relative lack of attention to the role of fathers in child and adolescent psychopathology. American Psychologist, 47, 656–664.

    Article  PubMed  Google Scholar 

  • Phares, V. (1997). Accuracy of informants: Do parents think that mother knows best? Journal of Abnormal Child Psychology, 25, 215–226.

    Article  Google Scholar 

  • Phares, V., Fields, S., Kamboukos, D., & Lopez, L. (2005). Still looking for Poppa. American Psychologist, 60, 735–736.

  • Rand, D. G. (2012). The promise of Mechanical Turk: How online labor markets can help theorists run behavioral experiments. Journal of Theoretical Biology, 299, 172–179.

    Article  PubMed  Google Scholar 

  • Rapee, R. M. (1997). Potential role of childrearing practices in the development of anxiety and depression. Clinical Psychology Review, 17, 47–67.

    Article  PubMed  Google Scholar 

  • Rubin, K. H., & Mills, R. S. (1990). Maternal beliefs about adaptive and maladaptive social behaviors in normal, aggressive, and withdrawn preschoolers. Journal of Abnormal Child Psychology, 18, 419–435.

    Article  PubMed  Google Scholar 

  • Schleider, J. L., Chorpita, B. F., & Weisz, J. R. (2014). Relation between parent psychiatric symptoms and youth problems: Moderation through family structure and youth gender. Journal of Abnormal Child Psychology, 42, 195–204.

    Article  PubMed  Google Scholar 

  • Schleider, J. L., Patel, A., Krumholz, L., Chorpita, B. F., & Weisz, J. R. (2015). Relations between parent symptomatology and youth problems: Multiple mediation through family income and parent–youth stress. Child Psychiatry and Human Development, 46, 1–9.

  • Seiffge-Krenke, I., & Kollmar, F. (1998). Discrepancies between mothers’ and fathers’ perceptions of sons’ and daughters’ problem behaviour: A longitudinal analysis of parent-adolescent agreement on internalising and externalising problem behaviour. Journal of Child Psychology, Psychiatry, and Allied Disciplines, 39, 687–697.

  • Shaffer, A., Lindhiem, O., Kolko, D. J., & Trentacosta, C. J. (2013). Bidirectional relations between parenting practices and child externalizing behavior: A cross-lagged panel analysis in the context of a psychosocial treatment and 3-year follow-up. Journal of Abnormal Child Psychology, 41, 199–210.

    Article  PubMed Central  PubMed  Google Scholar 

  • Shapiro, D. N., Chandler, J., & Mueller, P. A. (2013). Using Mechanical Turk to study clinical populations. Clinical Psychological Science, 1, 213–220.

    Article  Google Scholar 

  • Smedje, H., Broman, J.-E., Hetta, J., & von Knorring, A.-L. (1999). Psychometric properties of a Swedish version of the “strengths and difficulties questionnaire”. European Child and Adolescent Psychiatry, 8, 63–70.

    Article  PubMed  Google Scholar 

  • Snowden, L. R., & Cheung, F. K. (1990). Use of inpatient mental health services by members of ethnic minority groups. American Psychologist, 45, 347–355.

    Article  PubMed  Google Scholar 

  • Strohschein, L. (2005). Parental divorce and child mental health trajectories. Journal of Marriage and Family, 67, 1286–1300.

    Article  Google Scholar 

  • Suri, S., Goldstein, D. G., & Mason, W. A. (2011). Honesty in an online labor market. In L. von Ahn & P. G. Ipeirotis (Eds.), Papers from the 2011 AAAI workshop. Menlo Park, CA: AAAI Press.

    Google Scholar 

  • Trivette, C. M., Dunst, C. J., Deal, A., Hamer, A., & Propst, S. (1990). Assessing family strengths and family functioning style. Topics in Early Childhood Special Education, 10, 16–35.

    Article  Google Scholar 

  • Van Oort, F. V. A., Verhulst, F. C., Ormel, J., & Huizink, A. C. (2010). Prospective community study of family stress and anxiety in (pre)adolescents: The TRAILS study. European Child and Adolescent Psychiatry, 19, 483–491.

    Article  PubMed Central  PubMed  Google Scholar 

  • Veit, C., & Ware, J. (1983). The structure of psychological distress and well-being in general populations. Journal of Consulting and Clinical Psychology, 51, 730–742.

    Article  PubMed  Google Scholar 

  • Ventura, S. J., & Bachrach, C. A. (2000). Nonmarital childbearing in the United States, 1940–1999. National Vital Statistics Reports, 48, 1–40.

    Google Scholar 

  • Vespa, J., Lewis, J. M., & Kreider, R. M. (2013). America’s families and living arrangements: 2012. Washington, DC: U.S. Census Bureau.

  • Warren, S. L., Huston, L., Egeland, B., & Sroufe, L. A. (1997). Child and adolescent anxiety disorders and early attachment. Journal of the American Academy of Child and Adolescent Psychiatry, 36, 637–644.

    Article  PubMed  Google Scholar 

  • Williams, A. W., Ware, J. E., & Donald, C. A. (1981). A model of mental-health, life events, and social supports applicable to general populations. Journal of Health and Social Behavior, 22, 324–336.

    Article  Google Scholar 

  • Yancey, A. K., Ortega, A. N., & Kumanyika, S. K. (2006). Effective recruitment and retention of minority research participants. Annual Review of Public Health, 27, 1–28.

    Article  PubMed  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Psychology Department, Harvard University, 33 Kirkland Street, Cambridge, MA, 02138, USA

    Jessica L. Schleider & John R. Weisz

Authors
  1. Jessica L. Schleider
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. John R. Weisz
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jessica L. Schleider.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Schleider, J.L., Weisz, J.R. Using Mechanical Turk to Study Family Processes and Youth Mental Health: A Test of Feasibility. J Child Fam Stud 24, 3235–3246 (2015). https://doi.org/10.1007/s10826-015-0126-6

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10826-015-0126-6

Keywords

Search

Navigation