Methods for conducting systematic reviews of risk factors in low- and middle-income countries
- 2.1k Downloads
Rates of youth violence are disproportionately high in many low- and middle-income countries [LMICs] but existing reviews of risk factors focus almost exclusively on high-income countries. Different search strategies, including non-English language searches, might be required to identify relevant evidence in LMICs. This paper discusses methodological issues in systematic reviews aiming to include evidence from LMICs, using the example of a recent review of risk factors for child conduct problems and youth violence in LMICs.
We searched the main international databases, such as PsycINFO, Medline and EMBASE in English, as well as 12 regional databases in Arabic, Chinese, English, French, Spanish, Portuguese and Russian. In addition, we used internet search engines and Google Scholar, and contacted over 200 researchers and organizations to identify potentially eligible studies in LMICs.
The majority of relevant studies were identified in the mainstream databases, but additional studies were also found through regional databases, such as CNKI, Wangfang, LILACS and SciELO. Overall, 85 % of eligible studies were in English, and 15 % were reported in Chinese, Spanish, Portuguese, Russian or French. Among eligible studies in languages other than English, two-thirds were identified only by regional databases and one-third was also indexed in the main international databases.
There are many studies on child conduct problems and youth violence in LMICs which have not been included in prior reviews. Most research on these subjects in LMICs has been produced in the last two-three decades and mostly in middle-income countries, such as China, Brazil, Turkey, South Africa and Russia. Based on our findings, it appears that many studies of child conduct problems and youth violence in LMICs are reported in English, Chinese, Spanish and Portuguese, but few such studies are published in French, Arabic or Russian. If non-English language searches and screening had not been conducted in the current review, 15 % of eligible studies would have been missed. Although there are benefits to non-English language searches and the inclusion of non-English studies in meta-analyses, systematic reviewers also need to consider the resources required to incorporate multi-lingual research.
KeywordsDatabases Meta-analysis Systematic review Low- and middle-income countries Conduct problems Violence
Oppositional Defiant Disorder
low- and middle-income countries
Eighty-two percent of the world’s population, including 90 % of 2.2 billion children, live in low- and middle-income countries (LMICs) . Key requirements for responding to the health needs of populations in LMICs is to determine the extent of the problems and identify relevant risk factors . Risk factors are defined as characteristics of people or their environment associated with an increased probability of an adverse outcome . Preventive interventions aim to reduce the occurrence of risk factors and mitigate their impact, as well as to promote protective factors that reduce the probability of adverse outcomes in high risk contexts . In the first survey on mental health research priorities in LMICs, stakeholders and researchers identified prevalence studies and studies of risk factors as the most needed type of research . This paper discusses methods for conducting systematic reviews of risk factor studies in LMICs.
Violence is a major global health problem, and its levels are much higher in many LMICs than in high-income countries (HICs) [6, 7, 8]. As adolescence and early adulthood is a peak phase for many problem behaviours , the World Health Organization has characterized youth violence as a priority focus area due to its large contribution to the global burden of injury and premature death .
Given the high economic and health costs of youth crime and violence, and lack of existing reviews of evidence in LMICs, we conducted a systematic review of correlates and risk factors for perpetrating interpersonal violence and crime among young people 10 to 29 years old. The outcomes of interest included violent and non-violent crime, gang membership, carrying weapons, fighting, and other types of physical violence, such as assault and intimate partner violence. This review defined youth as 10–29 years old, following the World Health Organization definition of youth violence . We also searched for studies examining childhood conduct problems, which often precede youth crime and violence. Severe conduct problems are related to lifelong adverse outcomes, including unemployment, mental health problems, violent and non-violent crime and substance abuse [12, 13, 14]. The conduct problems considered in the review were general conduct problems, aggression, bullying, Oppositional Defiant Disorder (ODD) and Conduct Disorder (CD) among children 0 to 18 years old. We excluded studies that examined measures that combined conduct problems with symptoms of attention deficit hyperactivity disorder.
Several extensive reviews have been carried out to examine risk factors for conduct problems, crime and violence in high-income countries [15, 16, 17, 18]. While some of the previous reviews did not explicitly exclude research from LMICs, no review we are aware of in this field has actively focused on LMICs. Some reviews even made an explicit decision to focus on only HIC (or “Western”) countries . Most previous reviews have also had relatively exclusive methodological eligibility criteria, which may have resulted in a predominance of studies from HICs. Furthermore, they might not have captured some of the more recent research in LMICs. Therefore, there is a critical need to conduct systematic reviews to synthesise and evaluate evidence on these topics from LMICs [4, 19].
There are particular challenges for conducting systematic reviews aiming to locate and synthesize evidence from LMICs, including the range of languages in which studies may be reported and potential lack of inclusion of study reports in the large international databases usually searched in systematic reviews. An examination of the indexing of publications in psychiatry suggested that the international medical bibliographic databases tend to over-represent US journals compared to European ones . Another examination, focusing on five large Chinese biomedical databases, suggested that less than 6 % of the 2500 journals in those five databases were indexed by MEDLINE .
Empirical results from the current review, synthesizing findings on correlates, risk and protective factors will be reported separately in future publications. The current article describes key methodological issues involved in searching for evidence available from LMICs to contribute to “the science of research synthesis” . A specific empirical question we address in this paper is what proportion of the studies found by conducting searches and screening in seven different languages would have been retrieved if we had only searched the main medical and social science databases in English, as is often standard practice in systematic reviews. Are standard practices likely to yield a large proportion of the globally available evidence?
In this systematic review, we aimed to search for all evidence on risk factors for childhood externalising behaviours and youth crime and violence in LMICs. Drawing on a combination of free-text search terms, Medical Subject Headings and database-specific subject headings, we developed a sensitive search strategy for multiple electronic databases (see Additional file 1). The key concepts that orientated the search procedures were “low- and middle-income countries”, “children and youth”, and “conduct problems, crime and violence”. It is considered essential to search multiple international databases for a systematic review , particularly in a cross-disciplinary area of research . In August-September 2013 we searched PsycINFO, MEDLINE, EMBASE, CINAHL, EconLit, Criminal Justice Abstracts, Russian Academy of Sciences Bibliographies, Sociological Abstracts & Social Services Abstracts, Applied Social Sciences Index and Abstracts, International Bibliography of the Social Sciences, ERIC, Web of Science, Global Health Library, National Criminal Justice Reference Service Abstracts Database, CENTRAL, JOLIS, World Bank, Open Grey, and Google Scholar, without restrictions on study years or languages. The searches of these databases were conducted in English.
Databases searched with non-English terms
Index Medicus for the Eastern Mediterranean Region
King Saud University Repository
Index Medicus Afro
Revue de Médicine tropicale
Agence Universitaire de la Francophonie
Spanish and Portuguese
The review protocol was made public online in advance of the review. A full description of study selection criteria can be found in a supplementary online file (see Additional file 2). In brief, to be eligible, studies has to be conducted in a LMIC, and report an estimate of the strength of association between an eligible outcome and a potential correlate, or enough numerical information to calculate an association. The correlate and behavioural outcome had to be measured at the level of the individual. For example, studies at the neighborhood level comparing rates of violence across neighbourhoods with different levels of poverty were excluded. To restrict the review to more population-based generalizable studies, we only included studies with at least 100 participants recruited through random, stratified probability, or total population sampling within households, or two or more institutions in the community, such as schools, or maternity hospitals in birth cohort studies. Although we were most interested to review evidence on risk factors, which by definition precede the outcome and should be evaluated in longitudinal studies, results on correlates measured in cross-sectional studies were also eligible for the review because we did not expect to find many eligible longitudinal studies in LMICs. We also included case–control studies, for instance comparing a group of people in a young offenders’ institution and a matched group of youth in the general population.
Searching and screening
After screening with a specially designed form (see Additional file 3), 519 articles were identified as eligible for the review. Eligible studies may have included more than one eligible outcome for the review. Most of the eligible studies included in the review were English-language studies identified in the mainstream databases. Most of these were peer-reviewed academic journal publications, and 11 dissertations. Grey literature searches beyond indexed databases made a small contribution to this review, including ten additional reports (five from South Africa and one each from Brazil, Chile, China, Mexico, and Jamaica) not published in academic journals at the time but identified by email communication or internet searches. Ten additional publications were identified in reference harvesting (references from other included studies). Finally, 11 publications were chapters of the book describing the Second International Self-Report Delinquency Study, including the only studies identified from Armenia, Slovenia and Surinam.
Overall, 85 % of eligible studies were in English, and 15 % were reported in Chinese, Spanish, Portuguese, Russian, French, or German and translated to English for use in the review. Of the 77 studies reported in Chinese, Spanish, Portuguese, Russian and French, 25 eligible texts were identified both by well-known subscription databases, such as PsycINFO, Medline or EMBASE, as well as regional databases. Therefore, the regional database searches in foreign languages identified an additional 52 publications. Of these, the Chinese databases CNKI and Wangfang located 37 additional unique eligible studies, SciELO search identified seven, LILACS a further five, and elibrary.ru another three texts. The studies included from the regional databases report on research from China (37), Brazil (4), Russia (3), Colombia (3), Chile (2), Peru (2), and Mexico (1). Thus, most of the studies identified in the regional databases described research from countries also represented in the international databases – with the exception of Peru, as the only two publications available from there came from Lilacs. Finally, we also identified 39 potentially eligible articles in languages that we did not have the resources to screen in full-text (13 in Polish, 9 in Croatian, Serbian and Bosnian, 8 in Turkish, 4 in Hungarian, 4 in Lithuanian, and 2 in Persian). We were able to include one longitudinal study in German, identified through mainstream databases, but did not carry out any German-language searches.
Included study characteristics
Most of the studies identified by the review were based on cross-sectional studies (442). The review also included 76 reports on longitudinal studies, in which the risk factor preceded the outcome. Correlates and risk factors reported included, among others, the gender and age of the child/young person, psychological characteristics, substance use, health indicators, school performance, parenting factors such as parenting styles and attitudes, family socioeconomic status, and urban versus rural residence. An individual eligible study could include results on several relevant outcomes. Of the 351 studies of 0–18 year olds, 179 included results for general conduct problems, 146 for aggression, 35 for bullying perpetration and 16 for CD/ODD. Among 174 studies focusing on 10–29 year olds, 72 studies examined fighting, 45 carrying weapons, 48 other violence, 61 combined measures of non-violent and violent crime, 9 sexual or intimate partner violence and 8 gang membership.
As the included studies have not been fully coded yet, we cannot assess their quality. From the full-text screening, we know that a number of studies did not clearly specify the ages of participants, only providing the mean age of the sample or their stage of education (e.g., “secondary school students”). Clear information on sampling strategy was missing in 30 % of included studies. We chose to include such studies in our review but coded that sampling was “not clear”. In particular, we note that most of the reports in Chinese tended to be brief (3–4 pages), and many did not contain some important information, such as the sampling strategy.
Even with several restrictions on study design, such as a minimum sample size of 100 participants in community samples, we found a large number of studies providing evidence on correlates of childhood conduct problems, and, to a smaller extent, youth crime and violence in LMICs.
The publication dates of eligible studies retrieved for our review show that an increasing number of studies are being carried out in LMICs. In our review, 86 % of eligible studies were published since 2000. Our findings also suggest that only a few relevant studies have been carried out in some LMIC regions, such as South Asia. The locations of included studies suggest that the bulk of LMIC evidence is produced in several middle-income countries, such as China and Brazil, and very little in low-income countries. Although it is possible that different search methods might have located more studies in other parts of the world, this finding is similar to that of other reviews in international mental health literature [19, 26, 27].
Studies of language bias suggest that the majority of systematic reviews tend to only include studies in English . We explored a number of regional databases in Spanish, Portuguese, Chinese, French, Arabic and Russian that uniquely hosted 10 % of the eligible studies. In addition, we screened and included studies in languages other than English, which were identified in the mainstream databases (5 % of the eligible studies). Combined, foreign language studies accounted for 15 % of the total. Hence, if the non-English language searches and screening had not been conducted in the current review, 15 % of eligible studies would have been missed. Based on our findings, it appears that many studies examining child conduct problems and youth violence in the areas of child psychology, psychiatry and criminology in LMICs are reported in Chinese, Spanish and Portuguese, but few studies are likely to be reported in French, Arabic or Russian (although there is a number of studies from Russia in English). The project is currently at the stage of coding and analyses of the eligible studies, so it is not yet possible to determine whether the inclusion of non-English studies will impact the findings.
The influence of only searching in English has been examined in relation to systematic reviews of intervention evaluations. Two examinations of pairs of RCT reports in German and English suggested that programme evaluations with significant results were more likely to be published in English language journals than in non-English language journals [29, 30], suggesting that multi-lingual searches are needed to prevent bias. However, several other retrospective analyses of systematic reviews suggested that excluding non-English trials would not have substantially altered review conclusions [31, 32]. Including studies in languages other than English is clearly advantageous to increase the number of studies in a review, and thereby increase statistical power in meta-analyses, and external validity of the evidence. Further research is required concerning the potential impact of including articles from non-English journals on the power of the meta-analyses as well as the potential systematic bias introduced by only searching and screening in English.
Researchers also need to consider the resources required to incorporate multi-lingual research. In particular, to carry out screening and selection of studies that are not in English, team members are required who are fluent in those languages, and they need to be either fully familiar with the selection criteria, or translate study methods for others to evaluate, which can require considerable resources or decrease consistency in study screening. Hence, reviewers need to balance the use of resources with the gains made in terms of the number of additional eligible studies found by searching in languages other than English. It has been proposed that, for randomized trials, initial eligibility screening might be carried out even without knowledge of the relevant language, using English-language abstracts, and indicators such as the number of authors listed and presence of a CONSORT participant flowchart .
Most of our efforts in this review focussed on searching for published studies, but we also contacted over 200 researchers and practitioners to try to locate grey literature, and used Google Scholar to search for unpublished reports. Grey literature is considered important to minimize publication bias , although several researchers suggest that non-peer reviewed studies tend to be lower quality than studies published in peer reviewed journals . It is possible that other ways of searching for grey literature are necessary, to find high quality studies of correlates and risk factors in non-peer-reviewed reports. For example, researchers may consult studies carried out by international organizations, such as Global school-based student health survey coordinated by the World Health Organization. Although such reports may not directly examine risk factors, correlates can be calculated from the available data.
Future reviews of studies in LMICs could make valuable contributions by further investigating the proportion of studies identified in different databases. An extensive list of regional LMIC databases has been created by the Norwegian Satellite of the Cochrane Collaboration Effective Practice and Organisation of Care Group.1 Furthermore, there are a growing number of databases collecting impact evaluations of programmes and policies in LMICs. Extensive lists of relevant databases have been collected by the Campbell Collaboration International Development Coordinating group2 and the Center for Global Development.3
Guidelines for search strategies for observational studies are much less developed than for identifying evaluation studies such as randomized trials . Campbell Collaboration guidelines recommend that “High sensitivity should be sought, which may result in relatively low precision” . However, in a large project this may lead to an overwhelming number of citations. We believe our searches were very sensitive, but certainly not precise: less than 1 % of the unique database citations screened was eventually included in the review. For comparison, Sampson et al.  calculated that a sample of 94 MEDLINE-indexed systematic reviews had 3 % precision on average, although there was a wide range of precision in the reviews. It may be difficult to maintain sensitivity and identify all relevant studies that include correlates or risk factors with a more specific search strategy. We had considered using additional search terms for study design for example, but decided not to do so in order to maximise the chances of locating relevant studies. As part of developing guidance for future reviews of risk and protective factors, it would be useful to propose peer-reviewed search filters for studies of correlates.
There are many studies on conduct problems and youth crime and violence in LMICs which have not been included in prior reviews, which, for various reasons, have focused on HICs. However, most research on these subjects in LMICs has been produced in a handful of countries. Searches in languages other than English, especially in Chinese, Spanish and Portuguese, as well as the use of less well-known regional databases, such as CNKI, Wangfang, LILACS and SciELO, can contribute to systematic reviews of risk factor studies in LMICs. Future systematic review guidance should include advice on identifying and including studies reported in languages other than English.
We thank Tomas Allen and Isla Kuhn for vital help in developing the search strategy, Antonia Concha Errazuriz, Bruno Dalponte, Dong Yiqun, Franziska Mager, Lana Ghuneim, Lana Yoo, Lídia Maria de Oliveira Morais, Maria Paula Godoy, Simón Escoffier Martínez, Sze Long Mui, Yan Zhang, Zehang Chen, Li Jiawei, Ma Li, Li Tianqing, Wei Junfan, Zheng Anqing for searches, screening and translation in languages other than English. We thank Jim Derzon for examining his database of longitudinal studies to identify studies relevant to this review. We are thankful to many authors who kindly shared their manuscripts and provided clarifications. We are grateful to the Wellcome Trust [089963/Z/09/Z] and Bernard van Leer Foundation for financial support for the research.
- 1.Unicef. Statistics and monitoring. 2012. http://www.unicef.org/statistics. Accessed 31 October 2012.
- 3.Kraemer HC, Kraemer-Lowe KK, Kupfer DJ. To your health: How to understand what research tells us about risk. New York: Oxford University Press; 2005.Google Scholar
- 8.Haugen GA, Boutros V. The Locust Effect: Why the End of Poverty Requires the End of Violence. New York: Oxford University Press; 2014.Google Scholar
- 10.WHO. Chapter 2. Youth violence. 2002.Google Scholar
- 11.Krug EG et al. eds. World report on violence and health. Geneva: World Health Organization; 2002.Google Scholar
- 15.Farrington DP, Welsh BC. Saving children from a life of crime: Early risk factors and effective interventions. New York: Oxford University Press; 2007.Google Scholar
- 16.Hawkins JD, Herrenkohl T, Farrington DP, Brewer D, Catalano RF, Harachi TW. A review of predictors of youth violence. 1998.Google Scholar
- 18.Ellis L, Beaver KM, Wright J. Handbook of crime correlates. San Diego, CA: Academic Press; 2009.Google Scholar
- 20.Nieminen P. Bias against European journals in medical publication databases. The Lancet (North American edition).353(9164):1592. doi: 10.1016/S0140-6736(99)00415-8.
- 23.Lefebvre C, Manheimer E, Glanville J. Chapter 6. Searching for studies. In: Higgins J, Green S, editors. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0. www.cochrane-handbook.org: The Cochrane Collaboration; 2011.
- 24.Löhönen J, Isohanni M, Nieminen P, Miettunen J. Coverage of the bibliographic 569 databases in mental health research. Nord J Psychiatry. 2010;64(3):181–8.Google Scholar
- 26.Sharan ,Saxena S. World mental health: role of journals edited in developing countries. J Indian Assoc Child Adolesc Ment Health. 2006;2(1):1–8.Google Scholar
- 27.Hughes K, Bellis MA, Hardcastle KA, Butchart A, Dahlberg LL, Mercy JA, et al. Global development and diffusion of outcome evaluation research for interpersonal and self-directed violence prevention from 2007 to 2013: A systematic review. Aggress Violent Behav. 2014;19(6):655–62.CrossRefGoogle Scholar
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.