The Importance of Establishing Reliability and Validity of Assessment Instruments for Mental Health Problems: an Example from Somali Children and Adolescents Living in Three Refugee Camps in Ethiopia

Abstract

Assessing mental health problems cross-culturally for children exposed to war and violence presents a number of unique challenges. One of the most important issues is the lack of validated symptom measures to assess these problems. The present study sought to evaluate the psychometric properties of two measures to assess mental health problems: the Achenbach Youth Self-Report and the Child Posttraumatic Stress Disorder Symptom Scale. We conducted a validity study in three refugee camps in Eastern Ethiopia in the outskirts of Jijiga, the capital of the Somali region. A total of 147 child and caregiver pairs were assessed, and scores obtained were submitted to rigorous psychometric evaluation. Excellent internal consistency reliability was obtained for symptom measures for children and their caregivers. Validation of study instruments based on local case definitions was obtained for the caregivers but not consistently for the children. Sensitivity and specificity of study measures were generally low, indicating that these scales would not perform adequately as screening instruments. Combined test–retest and inter-rater reliability was low for all scales. This study illustrates the need for validation and testing of existing measures cross-culturally. Methodological implications for future cross-cultural research studies in low- and middle-income countries are discussed.

This is a preview of subscription content, log in to check access.

References

  1. Achenbach, T. M., McConaughy, S. H., & Howell, C. T. (1987). Child/adolescent behavioral and emotional problems: implications of cross-informant correlations for situational specificity. Psychological Bulletin, 10, 213–232.

    Google Scholar 

  2. Achenbach, T. M., & Rescorla, L. A. (2001). Manual for the ASEBA school-age forms & profiles. VT, Burlington: University of Vermont, Research.

    Google Scholar 

  3. Applied Mental Health Research Group. (2013). DIME module 2. http://www.jhsph.edu/research/centers-and-institutes/center-for-refugee-and-disaster-response/response_service/AMHR/dime/VOT_DIME_MODULE2_FINAL.pdf2013

  4. Barenbaum, J., Ruchkin, V., & Schwab-Stone, M. (2004). The psychosocial aspects of children exposed to war: practice and policy initiatives. Journal of Child Psychology and Psychiatry, 45(1), 41–62. doi:10.1046/j.0021-9630.2003.00304.x

    PubMed  Article  Google Scholar 

  5. Bass, J. K., Ayash, C., Betancourt, T. S., Haroz, E. E., Verdeli, H., Neugebauer, R., et al. (2013). Mental health problems of displaced war-affected adolescents in northern Uganda: patterns of agreement between self and caregiver assessment. Journal of Child and Family Studies. doi:10.1007/s10826-013-9755-9

    Google Scholar 

  6. Bass, J. K., Ryder, R. W., Lammers, M. C., Mukaba, T. N., & Bolton, P. (2008). Post-partum depression in Kinshasa, Democratic Republic of Congo: validation of a concept using a mixed-methods cross-cultural approach. Tropical Medicine International Health, 13(12), 1534–1542. doi:10.1111/j.1365-3156.2008.02160.x

    PubMed  Article  Google Scholar 

  7. Betancourt, T. S., Bass, J., Borisova, I., Neugebauer, R., Speelman, L., Onyango, G., & Bolton, P. A. (2009). Assessing local instrument reliability and validity: a field-based example from northern Uganda. Social Psychiatry and Psychiatric Epidemiology, 44(8), 685–692. doi:10.1007/s00127-008-0475-1

  8. Bolton, P. (2001). Cross-cultural validity and reliability testing of a standard psychiatric assessment instrument without a gold standard. Journal of nervous and mental disease, 189(4), 238–242. doi:10.1097/00005053-200104000-00005

    PubMed  Article  Google Scholar 

  9. Bolton, P., Michalopoulos, L., Ahmed, M. A., Murray, L. K., & Bass, J. (2013). The mental health and psychosocial problems of survivors of torture and genocide in Kurdistan, Northern Iraq: a brief qualitative study. Torture, 23(1), 1–14.

    PubMed  Google Scholar 

  10. Bolton, P., Wilk, C. M., & Ndogoni, L. (2004). Assessment of depression prevalence in rural Uganda using symptom and function criteria. Social Psychiatry and Psychiatric Epidemiology, 39(6), 442–447. doi:10.1007/s00127-004-0763-3

    PubMed  Article  Google Scholar 

  11. Federal Democratic Republic of Ethiopa Ministry of Health. (2012). National Mental Health Strategy (2012/13 - 2015/6).

  12. Foa, E. B., Johnson, K. M., Feeny, N. C., & Treadwell, K. R. H. (2001). The Child PTSD Symptom Scale: a preliminary examination of its psychometric properties. Journal of Clinical Child Psychology, 30(3), 376–384. doi:10.1207/S15374424JCCP3003_9

    PubMed  Article  Google Scholar 

  13. Hobfoll, S. E., Canetti, D., Hall, B. J., Brom, D., Palmieri, P. A., Johnson, R. J., et al. (2011). Are community studies of psychological trauma's impact accurate? A study among Jews and Palestinians. [Research Support, N.I.H., Extramural]. Psychological Assessment, 23(3), 599–605. doi:10.1037/a0022817

    PubMed Central  PubMed  Article  Google Scholar 

  14. Hollifield, M., Warner, T. D., Lian, N., Krakow, B., Jenkins, J. H., Kesler, J., et al. (2002). Measuring trauma and health status in refugees: a critical review. JAMA: Journal of the American Medical Association, 288(5), 611–621. doi:10.1001/jama.288.5.611

    PubMed  Article  Google Scholar 

  15. The International Rescue Committee & UNHCR (2009). Assessment report on the situation of children in Kebribeyeh and Sheder refugee camps.

  16. Kleinman, A. (1988). Rethinking psychiatry: from cultural category to personal experience. New York City: Free Press.

    Google Scholar 

  17. Kohrt, B. A., & Hruschka, D. J. (2010). Nepali concepts of psychological trauma: the role of idioms of distress, ethnopsychology and ethnophysiology in alleviating suffering and preventing stigma. Culture, Medicine and Psychiatry, 34(2), 322–352. doi:10.1007/s11013-010-9170-2

    PubMed  Article  Google Scholar 

  18. Lustig, S. L., Kia-Keating, M., Knight, W. G., Geltman, P., Ellis, H., Kinzie, J. D., et al. (2004). Review of child and adolescent refugee mental health. Journal of the American Academy of Child & Adolescent Psychiatry, 43(1), 24–36. doi:10.1097/00004583-200401000-00012

    Article  Google Scholar 

  19. Meyer, S., Murray, L. K., Puffer, E. S., Larsen, J., & Bolton, P. (2013). The nature and impact of chronic stressors on refugee children in Ban Mai Nai Soi camp, Thailand. Global Public Health, 8(9), 1027–1047. doi:10.1080/17441692.2013.811531

    PubMed  Article  Google Scholar 

  20. Mollica, R. F., Cardozo, B. L., Osofsky, H. J., Raphael, B., Ager, A., & Salama, P. (2004). Mental health in complex emergencies. The Lancet, 364(9450), 2058–2067. doi:10.1016/S0140-6736(04)17519-3

    Article  Google Scholar 

  21. Murray, L. K., Bass, J., Chomba, E., Imasiku, M., Thea, D., Semrau, K., et al. (2011). Validation of the UCLA child post traumatic stress disorder-reaction index in Zambia. International Journal of Mental Health Systems, 5, doi:10.1186/1752-4458-5-24

  22. Phan, T., & Silove, D. (1999). An overview of indigenous descriptions of mental phenomena and the range of traditional healing practices amongst the Vietnamese. Transcultural Psychiatry, 36(1), 79–94. doi:10.1177/136346159903600105

    Article  Google Scholar 

  23. Puffer, E. S., Larsen, J., Murray, L. K., Meyer, S., & Bolton, P. (2011). Qualitative findings: community perspectives on children’s mental health in Somali refugee camps in Ethiopia. Report prepared for the International Rescue Committee and Bill and Melinda Gates Foundation.

  24. Rasmussen, A., Katoni, B., Keller, A. S., & Wilkinson, J. (2011). Posttraumatic idioms of distress among Darfur refugees: Hozun and Majnun. Transcultural Psychiatry, 48(4), 392–415. doi:10.1177/1363461511409283

    PubMed  Article  Google Scholar 

  25. Reed, R. V., Fazel, M., Jones, L., Panter-Brick, C., & Stein, A. (2012). Mental health of displaced and refugee children resettled in low-income and middle-income countries: risk and protective factors. The Lancet, 379(9812), 250–265. doi:10.1016/S0140-6736(11)60050-0

    Article  Google Scholar 

  26. Sagi-Schwartz, A., Seginer, R., & Abdeen, Z. (2008). Chronic exposure to catastrophic war experiences and political violence: links to the well-being of children and their families: introduction to the special issue. International Journal of Behavioral Development, 32(4), 257–259. doi:10.1177/0165025408092219

    Article  Google Scholar 

  27. Silove, D., Bateman, C. R., Brooks, R. T., Fonseca, C. A. Z., Steel, Z., Rodger, J., et al. (2008). Estimating clinically relevant mental disorders in a rural and an urban setting in postconflict Timor Leste. Archives of General Psychiatry, 65(10), 1205–1212. doi:10.1001/archpsyc.65.10.1205

    PubMed  Article  Google Scholar 

  28. StataCorp (2012). STATA version 12.1. College Station, TX.

  29. UNHCR (2012). Jijiga population statistics (as of 25 September, 2012).

  30. United Nations. (2012). Report of the United Nations High Commissioner for Refugees. New York: United Nations.

    Google Scholar 

  31. Women's Refugee Commission. (2012). In search of safety and solutions: Somali refugee adolescent girls at Sheder and Aw Barre Camps, Ethiopia.

  32. Yan, H. (2013). State of Syria: exodus reaches 1 million; seesaw battles rage on. http://www.cnn.com/2013/03/06/world/meast/syria-civil-war/index.html. Accessed September 4 2013.

Download references

Acknowledgments

The present research was made possible by a grant from the Bill and Melinda Gates Foundation: Changing Lives: A Learning Initiative to End Violence Against Women and Children in Emergencies. Dr. Hall was supported by the National Institute of Mental Health T32 in Psychiatric Epidemiology T32MH014592-35 and through the Fogarty Global Health Fellows Program Consortium comprised of the University of North Carolina and John Hopkins Bloomberg School of Public Health, Morehouse and Tulane (1R25TW009340-01). We thank the dedicated study interviewers, Sarah Katherine Baird, IRC Community Well-Being Initiative Manager, Aden Abdi Hiss, IRC Transportation Officer, Asyia Abdulahi Elabe, IRC Caring for Child Survivors Officer, Tensay Tefera IRC CYPD-Child, Youth and Protection Department Manager, Shewaye, IRC CYPD Coordinator, Ayalew, IRC Child Protection Manager, Anjuli Shivshanker, IRC Research and Evaluation Officer, for her assistance with data collection, and the children and caregivers who participated in this study.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Brian J. Hall.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Hall, B.J., Puffer, E., Murray, L.K. et al. The Importance of Establishing Reliability and Validity of Assessment Instruments for Mental Health Problems: an Example from Somali Children and Adolescents Living in Three Refugee Camps in Ethiopia. Psychol. Inj. and Law 7, 153–164 (2014). https://doi.org/10.1007/s12207-014-9188-9

Download citation

Keywords

  • PTSD
  • Assessment
  • Reliability
  • Validity
  • Children
  • Adolescents
  • Refugees