Background

The process of collecting robust scientific evidence of ‘what works’ in humanitarian programming, and of ensuring contextual relevance in local communities, can be very challenging. In this paper, we present a case analysis of a longitudinal, interdisciplinary research project conducted with Syrian refugee and Jordanian non-refugee youth living in urban communities near active war zones. Responding to calls to strengthen the evidence base for public health interventions in humanitarian settings, [1, 2] our research team worked at the invitation of Mercy Corps to evaluate one of its youth-focused humanitarian programmes - a brief, scalable, community-based intervention aimed at improving the mental health and psychosocial wellbeing of war-affected adolescents. Subsequent to the completion of a randomized controlled trial, the team engaged in a systematic reflection on the research experience. We reviewed the scientific contributions of the study, the challenges in its implementation, and the strategies adopted to address them.

Humanitarian context and programme response

Worldwide, there are now 70.8 million forcibly displaced people, over half of whom are children under 18 years of age [3]. The Syrian crisis has created the world’s largest refugee crisis since World War II, with over 5.5 million people forced to flee their country since 2011 [4]. To-date, 656,700 Syrians are taking refuge in neighboring Jordan, 81% of whom live as urban refugees outside demarcated camps [5]. For children and youth, conflicts such as the Syrian crisis present urgent challenges for survival, health, and development pathways [6] – an entire generation of young Syrians is deemed at-risk for developmental and health consequences of exposure to trauma and toxic stress [7]. In 2013, the No Lost Generation initiative, a vigorous humanitarian response led by UNICEF, Mercy Corps, Save the Children, and World Vision, launched a number of youth-focused, community-based interventions to reach war-affected adolescents in crisis settings in the Middle East region [8].

As part of this initiative, Mercy Corps implemented the Advancing Adolescents programme with conflict-affected youth in Jordan, Lebanon, Iraq, Syria, and Turkey. This psychosocial intervention offers structured activities (16 sessions, across 8 weeks) to both Syrian refugee and host-community youth, with the goal of alleviating profound stress, enhancing psychosocial support, strengthening learning skills, and building resilience and social cohesion [9]. The program draws on a profound stress attunement framework to promote capacities for mediating extreme and prolonged stress, following the International Child Development Programme developed in Holland for global use [10]. Key elements of the program are common to other psychosocial interventions, including group-based sessions (developing technical, vocational, crafts, or fitness skills) run by trained, lay community workers. The latter were Jordanian and Syrian men and women, aged 21+, recruited from local community-based NGOs; prior to implementation, they completed the 16-day Coaches Foundation Training Programme delivered by Mercy Corps. The Mercy Corps training manual covers practicing healthy communication; defining profound stress, its impact on the human brain, and principles of attunement; developing gender equity and adolescent protection; building psychosocial resilience; and enhancing skills of creative facilitation for effective technical training [10].

Research study

Our study aimed to (a) test impacts of Advancing Adolescents on mental health and psychosocial wellbeing, and (b) develop a useful and sensitive toolkit for interdisciplinary evaluation. This would address important questions for humanitarian practice and policy (Does the intervention work, for whom, and for how long? What are the best ways to generate robust evidence of what works in humanitarian settings?). Before this, programme impact evaluations in the region had been wholly based on the self-reports of beneficiaries - sole reliance on self-reports can lead to response bias, especially where programme implementations and evaluations are done by the same agency, and sole reliance on beneficiaries, in the absence of a control group, can lead to unreliable evidence of impacts. In response to the need for an innovative research design to track stress alleviation and learning skills for beneficiaries and non-beneficiaries, we used an interdisciplinary approach embedded in a randomized controlled trial: we measured how Advancing Adolescents impacted stress in the mind, the body, and the brain (in terms of felt experiences, physiological regulation, and cognitive function), for both refugee and non-refugee adolescents.

We formed a research consortium after consulting with local community leaders and obtaining formal approval from the prime minister’s office to work in northern Jordan, drawing together international and regional staff at Mercy Corps; scholars from the UK, USA, and Canada; local community-based organizations running youth centers; and an independent NGO (Taghyeer, or We Love Reading) with research and advocacy experience in disadvantaged communities. Taghyeer anchored the project locally, by engaging communities in research design and implementation, and by recruiting a core research team of six Syrian and Jordanian women with university and professional degrees. We conducted our 18-month project (2015–17) in four northern cities (Irbid, Jarash, Mafraq, and Zarqa), receiving a significant influx of Syrian refugees. We implemented a quasi-experimental trial and then a randomized controlled trial with a waitlist control condition. We collected data at three time-points: baseline (T1), 10-weeks post-intervention (T2), and after 11 months (T3). We secured a gender-balanced sample of refugee and non-refugee youth (n = 817, 12–18 years) at T1, representing 20% of targeted beneficiaries and 48% of youth enrolled in Advancing Adolescents in Jordan at the time of study. With high levels of relocation in the targeted population, we retained 533 youth at T2 and 302 youth at T3. Our findings and methodologies, published in journals within the fields of biological sciences, child psychology and psychiatry, and child development, are summarized in Table 1.

Table 1 Summary of the research question, outcomes of interest, and findings for interdisciplinary evaluation of program impacts in a humanitarian setting
Full size table

Discussion

Scientific importance

Our study was the first to go beyond self-reports in measuring the effectiveness of a brief, scalable community-based program seeking to improve the psychosocial wellbeing of war-affected adolescents. Notably, it included biomarkers to track the cumulative effects of stress in the body: measuring cortisol concentrations in ~ 100 strands of hair, cut close to the scalp, reveals stress physiology in hair growth over the past month, with repeated sampling over time yielding cortisol secretion data akin to a stress diary. This helped us understand the processes of recovery from past trauma and current insecurity, showing that even brief interventions to improve day-to-day environments can protect adolescent health and wellbeing. Stress biomarkers were relatively simple and objective ways of helping to measure changes over time. They conclusively demonstrated that the Mercy Corps program had beneficial impacts for adolescents – not only did it improve self-reports of insecurity, stress, and poor mental health, but it changed and regulated physiological stress in the body.

In addition, our study tested cognitive skills and measured resilience – also a first for Syrian refugee youth. We found no evidence of program impacts for either outcome. This was an important lesson: programs that demonstrably alleviate profound stress, but not poverty per se, do not necessarily boost cognitive functioning, while programs that target individual-level changes may not change family and community environments enough to move the needle for psychosocial resilience. In order to boost learning skills and resilience in settings of protracted crises, humanitarian programs may need to broaden the package of youth-focused interventions with sustained support in home and school environments. We see that humanitarian programs do good in crisis settings, but often achieve only limited good given the multiple challenges of operating in such contexts. For this reason, evidence of impact or non-impact truly matters, helping to calibrate expectations and to move towards a more evidence-based model of humanitarian practice.

The field of mental health and psychosocial support (MHPSS) is poised to consolidate robust evidence on what works in humanitarian settings; specifically, it must assess why program impacts (often measured using self, parent, or teacher reports) are mostly small, variable, or nil, especially over the long-term [17]. Psychosocial interventions for children and adolescents are one of the most frequently implemented, yet poorly evaluated, interventions in crisis settings [1, 18, 19]. This proves frustrating for humanitarians looking for credible, actionable evidence in MHPSS programming. Our study introduced innovative, practical ways to generate robust scientific evidence and it also created strong partnerships between researchers, practitioners, and local communities. Syrian and Jordanian research team members were trained in research methods and analysis, engaged fully and regularly with US and UK partners to share cultural insights and benefit from opportunities created by international, collaborative exchange. We adopted a study design that prioritized both scientific rigor and local community engagement, used interdisciplinary methods of assessment to track multisystemic signatures of profound stress alleviation, and engaged in wide-ranging dissemination activities to promote the awareness of results and uptake of recommendations [20].

Challenges

We faced challenges with respect to study design, methods, and dissemination (Table 2).

Table 2 Ethical, logistical, and temporal challenges of research in a humanitarian setting
Full size table

Study design

A first set of challenges involved the ethical implementation of the randomized controlled trial and the logistics involved in sampling representative populations groups and tracking a mobile refugee population. Ethically, we could not randomly allocate participants to intervention or control groups, unless the latter were wait-listed to another cycle of program implementation. While Advancing Adolescents was implemented on a rolling basis, the Mercy Corps regional office was nearing the end of a funding cycle, unsure of its ability to extend programming for another year. Concerns were raised regarding random selection of participants into intervention and waitlist control conditions in a such a resource-uncertain and time-constrained situation. In response, we chose to initially implement an experimental trial, with non-randomized controls, and only moved to a fully randomized trial once longer-term funding was guaranteed to Mercy Corps for program implementation. At all stages, we relied on Syrian/Jordanian communities and research/humanitarian staff to help us design an ethical, robust, and culturally-compelling means of randomization.

Methods

A second set of challenges was related to methods. We began the study with a toolkit comprising psychometric and socio-demographic questionnaires, stress biomarkers, and cognitive tests – then made adjustments in response to pilot studies and fieldwork experience. We wanted to ensure that youth, local communities, and humanitarian staff were fully on board with the science of objective measurement, without placing undue burden for the young people who volunteered their participation in conditions of poverty, intense heat, and unreliable electricity. We were implementing, at three time-points, in four different cities, a fairly long and demanding study protocol, featuring multiple psychometric instruments to include internationally- and regionally-validated scales, relevant to war-affected youth. We augmented the survey with a culturally-grounded measure of resilience and dropped measures that proved less relevant or too burdensome for participants and fieldworkers (such as salivary cortisol). Collecting biological samples (blood spots, cheek swabs, hair cortisol) and testing cognitive skills (RACER tests of working memory and inhibitory control) in difficult field conditions was demanding – but worthwhile to thoroughly assess logistical feasibility and scientific value in humanitarian settings.

Dissemination

We also faced challenges in reconciling the different demands and time-frames for dissemination of results. As scholars with multiple teaching, research, and administration responsibilities, we took 18 months to fully analyse all field and lab data. Senior researchers met regularly in person or via skype, discussing and interpreting data to articulate meaningful results, relevant insights, and succinct conclusions. Local research team members were also in continuous dialogue with senior researchers, locally and internationally. In this way, we crystallised the work of an interdisciplinary research consortium through publishing multiple, peer-reviewed academic papers. However, Mercy Corps needed a bottom line regarding program effectiveness, without delay, to influence No Lost Generation funding and policy in timely fashion. We sought to accommodate such needs with preliminary reports, but given that rigorous conclusions could not be drawn until the completion of all analyses and the peer-review of scientific evidence, we slowed down the ability of Mercy Corps to share results publically, at least beyond verbal form, and their global influence on youth-focused humanitarian programming. Interdisciplinary work takes time, but can eventually engage different sectors in policy circles, educational sectors, and the media; most recently, an independent filmmaker released a 35-min documentary film of Syrian refugees and the science of resilience, based on our research work [21].

Research strategies

In response to the above challenges, our research strategies featured many elements to secure engagement and partnership (Table 3).

Table 3 Research strategies for engagement and partnership
Full size table

Local ownership

Through engagement with local partners and communities, we fostered a strong sense of local ownership. Issues of research design and study implementation were clearly open for discussion with research partners and communities. Local staff remarked that team de-briefings were both unusual and refreshing: they could voice concerns, influence scope and delivery, offer viewpoints and solutions. For example, the ethical issue of randomization was debated in back-and-forth conversations with Syrian and Jordanian families, who themselves provided a fair and transparent solution to secure equi-probability of assignment: after completing baseline assessments, each participant would draw one lollipop from an opaque bag, and a coin flip would randomize two lollipop colours to intervention and control groups. Community engagement was also key to avoid sampling biases: while only 14 families declined study participation, we faced significant sample attrition (n = 817 at baseline and n = 302 after 11 months) due to defunct phone contact numbers and youth who travelled, fell ill, got married, or were busy working. At T3, we offered transport compensation to families who had moved away in order to remove barriers to participation.

Engaging youth

Young people were keen to be included in a project that featured stress biomarkers, as this was cutting-edge science; in particular, Syrian refugees welcomed an opportunity to leverage science as a means of telling their story [22]. We also made participation attractive, with well-appreciated measures: we offered toiletries such as combs and toothbrushes as compensation for young people’s participation time, and we hired local male and female hairdressers to style young people’s hair during cortisol sampling. We chose, also, to discontinue experimental testing of cognitive skills at T3, in light of difficulties experienced by youth due to the long duration of testing. Finally, we adapted and validated the Child Youth Resilience Measure (CYRM, 12-items) in response to an explicit request from young people, who asked why assessment was largely focused on traumatic exposures and posttraumatic stress, and did not include measures of strength and hope [12].

Sharing findings

Our ultimate goal was the uptake of rigorous research to improve health-related programming in humanitarian settings. To this end, we presented at many events designed to bring together scholars, humanitarian practitioners, and policy-makers, continuing strong engagement with in-country partners well past the project funding dates. This process informed programmatic decisions: given dual evidence of psychosocial and physiological benefits, Mercy Corps integrated elements of the Advancing Adolescents stress-attunement approach into its regional livelihood interventions, and - given null impacts for resilience – turned their attention to engaging whole families, rather than individual youth, into resilience-building efforts. Sharing results with Syrian and Jordanian communities also created palpable excitement about the achievements of science in helping people understand what can be done, practically, to address profound stress in vulnerable populations. While some humanitarian and local staff had been sceptical about the need for randomization and the value of including biological markers, young people themselves – once these concepts were explained - welcomed this type of scientific enquiry; they valued playing a role in advancing science, especially with respect to measuring stress (in hair samples) to assess biological impacts.

We found that biological data collection worked well and provided sharp results: specifically, high cortisol levels revealed a body in a state of high alert; chronically low levels indicated blunted sensitivity to environmental challenges; and mixed profiles showed dysregulation [13, 14]. In contexts of war and forced displacement, this has value for understanding how the alleviation of profound stress may protect adolescent health and developmental trajectories. Because we leveraged stress physiology into our research design, evidence from this study has been promoted as proof-of-concept within the fields of humanitarian practice and global mental health. And because we used RACER tests to experimentally assess executive function, we were able to contribute to debates on the relative impacts of trauma exposure versus socio-economic deprivation on youth’s cognitive function [16].

Finally, the project created several professional development opportunities. For example, a Syrian researcher was able to travel to the United States to visit laboratories used for cortisol analysis. Both Syrian/Jordanian and Western senior scientists went to global dissemination meetings and academic conferences, prompting the funders of the study to commend our North-South collaboration as a model partnership. The project prompted new collaborations, including a study on the genetics of stress and resilience [23, 24], research on how an educational intervention for war-affected youth develops, and research on the intergenerational transmission of trauma (forthcoming). Finally, the work directly informed evaluation strategies for organisations such as We Love Reading and Noor Hussein Foundation, in addition to Mercy Corps.

Conclusions

Reflection on the process of implementing this interdisciplinary evaluation study established four important lessons. First, we learnt that high-quality scientific research is feasible, and can be useful and ethical, in humanitarian settings: with the support of a local research team and active engagement with their community, we were able to realize an ambitious research program to assess the bio-psychosocial and cognitive signatures of profound stress alleviation. Second, we found young people in the context of a humanitarian crisis to be engaged, motivated, and informed regarding the value of scientific research seeking to determine effective means of supporting them in dealing with the profound stress of conflict and displacement.

Third, we learnt that sustained partnerships between scholars, humanitarians, funders, and beneficiaries across multiple locales produces a strong sense of research ownership – back-and-forth discussions of complex challenges and heterogeneous findings help grow the research agenda. Finally, we learnt that opportunities for frequent, meaningful conversations are strongly needed – and that serious dedication to one’s work does not preclude dialogue and humour. Working with multiple, diverse partners often entails bridging different needs and priorities: local communities want their concerns met and recognized; humanitarian organizations are focused on delivering care, and need research findings to support that effort; scholars task themselves with producing research that informs this engagement and meets a high standard for academic scrutiny. Positioned in this mix are the funding agencies disbursing limited resources, the state confronting a challenging crisis, and the media in pursuit of newsworthy stories. Sound partnerships between these multiple actors requires time, funding, good communication, reflexive evaluation, and an ethos of building trust, respect, and meaningful engagement.