Social Psychiatry and Psychiatric Epidemiology

, Volume 44, Issue 8, pp 685–692 | Cite as

Assessing local instrument reliability and validity: a field-based example from northern Uganda

  • Theresa S. BetancourtEmail author
  • Judith Bass
  • Ivelina Borisova
  • Richard Neugebauer
  • Liesbeth Speelman
  • Grace Onyango
  • Paul Bolton


This paper presents an approach for evaluating the reliability and validity of mental health measures in non-Western field settings. We describe this approach using the example of our development of the Acholi psychosocial assessment instrument (APAI), which is designed to assess depression-like (two tam, par and kumu), anxiety-like (ma lwor) and conduct problems (kwo maraco) among war-affected adolescents in northern Uganda. To examine the criterion validity of this measure in the absence of a traditional gold standard, we derived local syndrome terms from qualitative data and used self reports of these syndromes by indigenous people as a reference point for determining caseness. Reliability was examined using standard test–retest and inter-rater methods. Each of the subscale scores for the depression-like syndromes exhibited strong internal reliability ranging from α = 0.84–0.87. Internal reliability was good for anxiety (0.70), conduct problems (0.83), and the pro-social attitudes and behaviors (0.70) subscales. Combined inter-rater reliability and test–retest reliability were good for most subscales except for the conduct problem scale and prosocial scales. The pattern of significant mean differences in the corresponding APAI problem scale score between self-reported cases vs. noncases on local syndrome terms was confirmed in the data for all of the three depression-like syndromes, but not for the anxiety-like syndrome ma lwor or the conduct problem kwo maraco.


war adolescents northern Uganda mental health validity culture 



A number of talented and dedicated collaborators made this work possible. We are endlessly grateful to all the local research assistants who carried out these interviews in Northern Uganda and to the youth and caregivers who participated in the interviews. We are also grateful to World Vision and War Child Holland for their collaboration and support of this field research. This publication was also made possible by Grant #1K01MH077246-01A2 from the National Institute of Mental Health. Additional thanks go to Seganne Musisi, James Okello, Margarita Alegria and Sidney Atwood for their input and review of study design/and or findings.


  1. 1.
    Achenbach TM (1991) Manual for the child behavior checklist and 1991 profile. Department of Psychiatry, University of Vermont, BurlingtonGoogle Scholar
  2. 2.
    Achenbach TM (1991) Manual for the youth self report and 1991 profile. Department of Psychiatry, University of Vermont, BurlingtonGoogle Scholar
  3. 3.
    Achenbach TM, Dumenci L (2001) Advances in empirically based assessment: revised cross-informant syndromes and new DSM-oriented scales for the CBCL, YSR, and TRF: comment on Lengua, Sadowksi, Friedrich, and Fischer (2001). J Consult Clin Psychol 69:699–702PubMedCrossRefGoogle Scholar
  4. 4.
    Achenbach TM, McConaughy SH, Howell CT (1987) Child/adolescent behavioral and emotional problems: implications of cross-informant correlations for situational specificity. Psychol Bull 101:213–232PubMedCrossRefGoogle Scholar
  5. 5.
    Aidoo M, Harpham T (2001) The explanatory models of mental health amongst low-income women and health care practitioners in Lusaka, Zambia. Health Policy Plan 16:206–213PubMedCrossRefGoogle Scholar
  6. 6.
    Bass JK, Bolton PA, Murray LK (2007) Do not forget culture when studying mental health. Lancet 370:918–919PubMedCrossRefGoogle Scholar
  7. 7.
    Bass JK, Ryder RW, Lammers CM, Mukaba TN, Bolton PA (2008) Post-partum depression in Kinshasa, Democratic Republic of Congo: validation of a concept using a mixed-methods cross-cultural approach. Trop Med Int Health 13:1–9CrossRefGoogle Scholar
  8. 8.
    Bernal G (2006) Intervention development and cultural adaptation research with diverse families. Fam Process 45:143–151PubMedCrossRefGoogle Scholar
  9. 9.
    Betancourt TS, Speelman L, Onyango G, Bolton P (2008) A qualitative study of psychosocial problems of war-affected youth in northern Uganda. J Transcult Psychiatry (in press)Google Scholar
  10. 10.
    Bhagwanjee A, Parekh A, Paruk Z, Petersen I, Subedar H (1998) Prevalence of minor psychiatric disorders in an adult African rural community in South Africa. Psychol Med 28:1137–1147PubMedCrossRefGoogle Scholar
  11. 11.
    Bhui K, Craig T, Mohamud S, Warfa N, Stansfeld SA, Thornicroft G, Curtis S, McCrone P (2006) Mental disorders among Somali refugees: developing culturally appropriate measures and assessing socio-cultural risk factors. Soc Psychiatry Psychiatr Epidemiol 41:400–408PubMedCrossRefGoogle Scholar
  12. 12.
    Bolton P (2001) Cross-cultural validity and reliability testing of a standard psychiatric assessment instrument without a gold standard. J Nerv Ment Dis 189:238–242PubMedCrossRefGoogle Scholar
  13. 13.
    Bolton P (2001) Local perceptions of the mental health effects of the Rwandan genocide. J Nerv Ment Dis 189:243–248PubMedCrossRefGoogle Scholar
  14. 14.
    Bolton P, Bass J, Betancourt T, Speelman L, Onyango G, Clougherty KF, Neugebauer R, Murray L, Verdeli H (2007) Interventions for depression symptoms among adolescent survivors of war and displacement in northern Uganda: a randomized controlled trial. JAMA 298:519–527PubMedCrossRefGoogle Scholar
  15. 15.
    Bolton P, Wilk CM, Ndogoni L (2004) Assessment of depression prevalence in rural Uganda using symptom and function criteria. Soc Psychiatry Psychiatr Epidemiol 39:442–447PubMedCrossRefGoogle Scholar
  16. 16.
    Cronbach LJ, Meehl PE (1955) Construct validity in psychological tests. Psychol Bull 52:281–302PubMedCrossRefGoogle Scholar
  17. 17.
    Derluyn E, Broekaert E, Schuyten G, De Temmerman E (2004) Post-traumatic stress in former Ugandan child soldiers. Lancet 363:861–863PubMedCrossRefGoogle Scholar
  18. 18.
    Duhig AM, Renk K, Epstein MK, Phares V (2000) Interparental agreement on internalizing, externalizing, and total behavior problems: a meta-analysis. Clin Psychol Sci Pract 7:435–453CrossRefGoogle Scholar
  19. 19.
    Geertz C (1973) Interpretation of cultures. Basic Books, New YorkGoogle Scholar
  20. 20.
    Goldstein JM, Simpson JC (1995) Validity: definitions and applications to psychiatric research. In: Tsuang MT, Tohen M, Zahner GEP (eds) Textbook in psychiatric epidemiology. Wiley, New York, pp 229–242Google Scholar
  21. 21.
    Hollifield M, Warner TD, Nityama L, Krakow B, Jenkins J, Kesler J, Stevenson J, Westermeyer J (2002) Measuring trauma and health status in refugees: a critical review. JAMA 288: 611–621PubMedCrossRefGoogle Scholar
  22. 22.
    Kagee A (2004) Do South African former detainees experience post-traumatic stress? Circumventing the demand characteristics of psychological assessment. Transcult Psychiatry 41: 323–336PubMedCrossRefGoogle Scholar
  23. 23.
    Kessler RC, Foster CL, Saunders WB, Stang PE (1995) Social consequences of psychiatric disorders, I: educational attainment. Am J Psychiatry 152:1026–1032PubMedGoogle Scholar
  24. 24.
    Kleinman A (1988) Rethinking psychiatry: from cultural category to personal experience. Free Press, New YorkGoogle Scholar
  25. 25.
    MacMullin C, Loughry M (2004) Investigating psychosocial adjustment of former child soldiers in Sierra Leone and Uganda. J Refug Stud 17:460–472CrossRefGoogle Scholar
  26. 26.
    Mollica R, Lopez Cardoza B, Osofsky HJ, Ager A, Salama P (2004) Mental health in complex emergencies. Lancet 364: 2058–2067PubMedCrossRefGoogle Scholar
  27. 27.
    Nubukpo P, Preux PM, Houinato D, Radji A, Grunitzky EK, Avode G, Clement JP (2004) Psychosocial issues in people with epilepsy in Togo and Benin (West Africa) I: anxiety and depression measured using Goldberg’s scale. Epilepsy Behav 5:722–727PubMedCrossRefGoogle Scholar
  28. 28.
    Patel V, Simunyu E, Gwanzura F, Lewis G, Mann A (1997) The shona symptom questionnaire: the development of an indigenous measure of common mental disorders in Harare. Acta Psychiatr Scand 95:469–475PubMedCrossRefGoogle Scholar
  29. 29.
    Reich W, Earls F (1987) Rules for making psychiatric diagnoses in children on the basis of multiple sources of information: preliminary strategies. J Abnorm Child Psychol 15:601–616PubMedCrossRefGoogle Scholar
  30. 30.
    Relief Web (2006) Child soldiers global report: United Nations Office for Coordination of Humanitarian AffairsGoogle Scholar
  31. 31.
    Seedat S, Nyamai C, Njenga F, Vythilingum B, Stein DJ (2004) Trauma exposure and post-traumatic stress symptoms in urban African schools: survey in Cape Town and Nairobi. Br J Psychiatry 184:169–175PubMedCrossRefGoogle Scholar
  32. 32.
    Shrout PE (1995) Reliability. In: Tsuang MT, Tohen M, Zahner GE (eds) Textbook in psychiatric epidemiology. Wiley, New York, pp 213–242Google Scholar
  33. 33.
    UN OCHA (2007) Forgotten and neglected emergencies: UgandaGoogle Scholar
  34. 34.
    Verdeli H, Clougherty K, Onyango G, Lewandowski E, Speelman L, Betancourt TS, Neugebauer R, Stein TR, Bolton P (2008) Group interpersonal psychotherapy for depressed youth in IDP camps in Northern Uganda: adaptation and training. Child Adolesc Psychiatr Clin N Am 17:605–624PubMedCrossRefGoogle Scholar
  35. 35.
    Weisz JR, Sigman M, Weiss B, Mosk J (1993) Parent reports of behavioral and emotional problems among children in Kenya, Thailand, and the United States. Child Dev 64:98–109PubMedCrossRefGoogle Scholar
  36. 36.
    Zandi T, Havenaar JM, Limburg-Okken AG, van Es H, Sidali S, Kadri N, van den Brink W, Kahn RS (2008) The need for culture sensitive diagnostic procedures: a study among psychotic patients in Morocco. Soc Psychiatry Psychiatr Epidemiol 43: 244–250PubMedCrossRefGoogle Scholar

Copyright information

© Steinkopff Verlag Darmstadt 2009

Authors and Affiliations

  • Theresa S. Betancourt
    • 1
    Email author
  • Judith Bass
    • 2
  • Ivelina Borisova
    • 1
  • Richard Neugebauer
    • 3
  • Liesbeth Speelman
    • 4
  • Grace Onyango
    • 5
  • Paul Bolton
    • 6
  1. 1.Dept. of Population and International Health, François-Xavier Bagnoud Center for Health and Human RightsHarvard School of Public HealthBostonUSA
  2. 2.Dept. of Mental HealthJohn Hopkins Bloomberg School of Public HealthBaltimoreUSA
  3. 3.Division of EpidemiologyNYS Psychiatric InstituteNew YorkUSA
  4. 4.War Child Holland HeadquartersAmsterdamThe Netherlands
  5. 5.World Vision InternationalKampalaUganda
  6. 6.Center for Refugee and Disaster ResponseJohns Hopkins Bloomberg School of Public HealthBaltimoreUSA

Personalised recommendations