Introduction

Nationally representative mental health surveys play a critical role in providing quality data to help understand and track the mental health of a population. Most contemporary mental health surveys collect data on more than disorder prevalence alone to gain better insight into the impacts of mental disorders and the types of interventions or services required. For example, surveys conducted as part of the World Health Organization (WHO)’s World Mental Health Survey (WMH) Initiative also collect data on disease burden, relevant risk factors, comorbidities, service use and unmet treatment needs.[1].

On a broad scale, global estimates of the burden of mental and other disorders rely on population-based prevalence surveys as a key input. At a national level, these surveys help to shape the public narrative around mental health within a country and play a fundamental role in influencing mental health policy, planning and funding [2]. In fact, planning and costing tools such as the mental health module within the OneHealth systems planning tool (OHT) [3] and Australia’s National Mental Health Service Planning Framework (NMHSPF) [4, 5] draw heavily on estimates generated by population-based prevalence surveys to model mental health care resource requirements.

While such mental health surveys provide critical insight into the mental health status and needs of a population, it is important to recognise their limitations. These include the ongoing exclusion of key vulnerable groups through sampling methodologies and growing rates of non-response. To date, there has been no review synthesising information on the impacts of non-response or excluded populations in large mental health surveys. Understanding these systematically excluded subpopulations is critical to identifying potential underreporting of population mental health needs and to ensure equity of access and tailored service planning for the highest needs groups.

This review aims to (1) identify which groups are commonly excluded from large mental health surveys in high-income OECD (Organisation for Economic Co-operation and Development) countries, (2) identify survey response rates and describe the characteristics of non-responders, and (3) describe any efforts to examine, supplement or adjust for the impacts of non-response and missing or underrepresented populations.

Methods

We conducted a targeted literature review of national mental health surveys to extract detail on survey sampling methods by drawing on previous systematic reviews of prevalence data.

Survey search strategy and inclusion criteria

The Global Burden of Disease Study 2019 (GBD 2019) Data Input Sources Tool [6] was used to screen data sources for prevalence of mental disorders in high-income OECD countries published from 2005 onwards. The GBD inputs are obtained through a systematic review of studies and data sources on the prevalence and burden of mental disorders from 204 countries between 1990 and 2019 [7]. Additional information on the GBD review methodology is available online [7].

From this list, sources were included for further consideration if they described a population-based mental health survey that:

  • Ended between January 2005 and December 2019, inclusive; included a nationally representative sample of adults; surveyed multiple classes of mental disorders (e.g. depressive disorders, anxiety disorders, substance use disorders, eating disorders, personality disorders, conduct or impulse control disorders, attention-deficit/hyperactivity disorder, autism spectrum disorders, psychotic disorders); and

  • was completed as either a stand-alone mental health survey or conducted as part of a general health survey.

  • Sources were excluded from further review if the survey they described:

  • was limited to only one gender;

  • focused on adults in a narrow age cohort only (e.g. young adults, older adults);

  • was a follow-up of a previously surveyed cohort and therefore not a random sample from the general population;

  • had specific aims other than determining prevalence of mental disorders that otherwise restricted the eligibility criteria;

  • focused exclusively on one class of mental disorder, e.g. depressive disorders only;

  • restricted the sample to one region within a country;

  • sampled populations across multiple countries as part of the same survey.

A list of relevant surveys was then compiled from the included GBD sources. Where multiple iterations of a survey were conducted for different years, only the most recent version included in GBD was selected.

Search for published survey methods

A Google search was then conducted from May to September 2021 using the name (or the title of the GBD source where no name was given) and year of each survey as search terms to collect academic and grey literature sources associated with each survey, such as methodology reports and results studies. The first 50 search results were scanned for publications that described the survey in their methodology or findings. Where websites or webpages devoted to listing or archiving a survey’s publications were identified in the search results, these were thoroughly checked. Forward and backward snowballing [8] was conducted using references from the original GBD publication(s) and Google search results as the start set. Forward snowballing was performed using the online citation index scite [9], while backward snowballing was performed by hand until saturation was reached (i.e. no new relevant sources were identified). Google searching and snowballing was preferred to a search of scientific databases given that a large proportion of the data of interest lay in grey literature. Government and non-government reports, government webpages, journal articles and manuscripts were reviewed. Online translations of non-English sources were considered for review if they could be verified by a fluent language reader. News articles, commentaries, blog posts, abstracts only and non-government webpages were excluded.

A full-text review of all relevant publications was conducted to identify the primary source of data for the survey methodology. Where multiple publications described the methodology in comparable detail, the earliest published or available resource was selected. Supplementary sources were also included where they provided additional information on: (1) the survey design; (2) survey response rate; (3) excluded populations; (4) characteristics associated with response as judged by the authors of relevant sources; (5) survey adjustments for excluded or underrepresented populations; (6) supplementary or parallel mental health surveys of non-responders or excluded populations; or (7) additional analyses on the association between survey response and disorder prevalence. Where supplementary surveys or additional analyses were explicitly identified in the primary publication, an additional search was conducted using the same Google search and snowballing method to identify relevant publications.

Supplementary non-response surveys were included for data extraction if they collected data on mental health, including symptom measures or diagnosis history. Since resource limitations may restrict the number of additional surveys conducted over time, supplementary surveys of excluded populations conducted for both past and current iterations of the primary survey (up to a maximum of two) were included for data extraction so long as they collected information on mental health, including a diagnostic interview, symptom measures or diagnosis history. Non-response and supplementary surveys focused solely on substance use disorders, mental health treatment or self-harm, suicide attempts and suicidal ideation were not included.

Data extraction and synthesis

Survey information extracted from relevant sources included country, survey year, survey design, sample size, psychiatric disorders of interest, psychological assessment instruments and response rate. Information extracted on psychiatric disorders excluded indicators of self-harm; suicidal thoughts; suicide attempts; smoking; use of tobacco products; nicotine use, dependence or withdrawal; alcohol or drug use where this was not characterised as harmful, problematic or a substance use disorder; and other general health and well-being measures (e.g. quality of life). Next, data were extracted to describe populations that were identified as underrepresented or less likely to respond, as well as groups that were excluded from participating in the survey because of the sampling methodology or eligibility criteria.

Finally, information was extracted from the relevant literature on additional efforts undertaken to survey non-responders or excluded groups, or to investigate the impacts of response rate on prevalence estimates. This included data on the general design and methodology of the supplementary survey or analysis as well as the year, sample size, psychological assessment measures used, response rate and key findings.

Where relevant data were missing from existing literature or questions remained regarding the sampling methodology, exclusion criteria or supplementary surveys, up to three contributing authors of included sources were contacted for more information.

Following a preliminary review of the data, categories for frequently excluded groups were established through an iterative process, beginning with the categories identified in the source literature and grouping these based on similar characteristics and reasons for exclusion. This process facilitated comparisons between surveys. The data extraction and synthesis were conducted and reviewed by EW, IP and CP.

Results

Survey characteristics

Twenty-two sources from the GBD Data Input Sources Tool described surveys that met our criteria. After accounting for sources that used the same survey, 16 individual surveys [11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29] conducted in 14 countries were included for analysis (Figure 1). Table 1 provides an overview of the methods for each included survey. Fourteen surveys sampled adults only [11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26], while 2 surveys also included children and adolescents under the age of 15 years [27,28,29]. Fifteen of the surveys were conducted in person [11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29] and 2 was mailed to participants [21], who could mail back the completed survey or complete it online. Three surveys explicitly described attempts to account for the possible underrepresentation of specific groups in their sampling methodologies [11, 13, 20, 27, 29], while 14 applied post-stratification weighting to adjust for sampling bias and non-response and improve representativeness [11,12,13,14,15,16,17, 19,20,21,22,23,24,25,26,27,28,29]. All surveys included measures of anxiety and depressive disorders [11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29], 4 included psychotic disorders [12, 14, 19, 20, 26], 2 included personality disorders [12, 14, 19], 2 included conduct or impulse control disorders [19, 22], 1 included autism spectrum disorders [12, 14] and 13 included substance use disorders [11,12,13,14,15,16, 18,19,20, 22,23,24,25,26,27,28,29]. All 13 surveys that included substance use disorders included measures of alcohol use disorders [11,12,13,14,15,16, 18,19,20, 22, 24,25,26,27,28,29], while 10 also collected information on other drug use disorders [11,12,13,14,15, 18, 19, 20, 22, 24, 27, 28, 29]. Most surveys were conducted by government agencies and/or academic collaboratives.

Fig. 1
figure 1

Identification of national mental health surveys [10]

Full size image
Table 1 Overview of included nationally representative adult mental health surveys
Full size table

Excluded populations

Details about inclusion and exclusion criteria were identified for all included surveys; however, the amount of information provided was highly variable. Based on available published data, we identified 13 populations or groups that were commonly excluded from national mental health surveys (Table 2). These populations are listed below in order from most to least commonly excluded:

  • Homeless people with no fixed address (14 of 16 surveys).

  • People in hospitals and health facilities (14 of 16 surveys.)

  • People residing in correctional facilities (14 of 16 surveys).

  • People in residential care facilities (12 of 16 surveys).

  • Military personnel on base or abroad (12 of 16 surveys).

  • Non-local language speakers (11 of 16 surveys).

  • People residing in temporary housing (11 of 16 surveys).

  • Short-term overseas visitors (11 of 16 surveys).

  • People living on islands, in remote areas or in specific territories (9 of 16 surveys).

  • People living in other non-private dwellings (e.g. migratory worker dormitories, detention facilities, monasteries, etc.) (8 of 16 surveys).

  • People living in educational institutions (7 of 16 surveys).

  • Non-residents (7 of 16 surveys).

  • People with a cognitive impairment (7 of 16 surveys).

Table 2 Populations covered in national mental health surveys
Full size table

Information was missing and could not be obtained on the inclusion or exclusion of some of these groups for ten of the surveys, two of which only had data available on the eligibility of one population (see Table 2).

It is worth noting that many surveys only sampled people living in private dwellings, and this often constituted the reason some groups were or may have been excluded. Furthermore, the location-based criteria were often only applicable to the time period in which the fieldwork was conducted and/or whether the individual had a fixed residential address [19, 30, 31]. For instance, a student with a fixed address outside of their educational institution would have been eligible for the 2014 survey in England [30] or the 2010–2011 survey in Poland [31] if they had returned home during the sampling period in that area.

Fewer than half of the surveys had literature that discussed the eligibility of people with a cognitive impairment [13, 20, 25, 29, 32, 33]. In two instances where authors were able to be contacted, it was suggested that while there was no formal cognitive assessment conducted as part of the sampling process, there would likely be a degree of self-selection or implicit exclusion where individuals were unable to understand and respond to the survey questions [34, 35]. In the case of the Adult Psychiatric Morbidity Survey (APMS) conducted in England in 2014, personal communication with the survey team stated that individuals with a cognitive impairment were likely to be excluded unless the impairment was mild [30]. As such, we considered that this exclusion criterion was applied on a case-by-case basis.

Response rate, underrepresented groups and non-responder characteristics

Survey response rates ranged from 36.3% [23] to 80% [27] (Table 1). Ten of the included surveys had sources that compared the characteristics of people in the survey sample to those of the general population [11, 13, 14, 16,17,18,19,20,21, 25, 26]. Analysis of these findings showed that males [14, 17, 19, 20, 25] and younger adult cohorts [14, 17, 19] were the most commonly underrepresented groups. Literature on four surveys also reported on factors associated with response rate [13, 16, 19, 24]. However, no common factors influencing response rate were identified between the surveys.

Supplementary surveys of excluded populations

Attempts to capture excluded populations through supplementary surveys were identified for four of the primary surveys included in our analysis (Table 3):

  • The 2007 NSMHWB in Australia

    • Supplementary survey population: people living in rural and remote areas [42,43,44,45,46].

  • The Canadian Community Health Survey- Mental Health (CCHS-MH).

    • Supplementary survey population: military personnel [47,48,49,50].

  • The APMS 2014 in England.

    • Supplementary survey populations: ethnic minorities [51], homeless adults [52], people in correctional facilities [53] and residents of institutions catering to people with mental illness [54].

  • The 2014 NSDUH in the USA.

    • Supplementary survey populations: military personnel [55,56,57] and people in correctional facilities [58,59,60,61].

Table 3 Supplementary surveys of populations excluded from national mental health surveys
Full size table

Supplementary survey findings from Australia showed that while the overall rate of moderate psychological distress in rural and remote areas was similar to that reported for the general household survey [42], distress scores indicating caseness were >10% more prevalent in very remote locations relative to other rural and remote regions [46]. Data from the supplementary surveys conducted in Canada showed that major depressive disorder, generalised anxiety disorder and panic disorder were more prevalent in Canadian military personnel than in members of the civilian population [47,48,49,50]. While there were no direct comparisons between the military surveys and the 2014 NSDUH findings in the USA, comparisons made to earlier survey results in the USA also indicated higher rates of mental disorders in active-duty soldiers [55, 56].

The supplementary survey of people in correctional facilities in England and Wales showed that significant neurotic symptoms and functional psychosis were far more common among prisoners than members of the general population [53]. While comparisons between surveys in the USA were limited by differences in the measures used, the results from the National Inmate Survey (NIS-3) also showed significantly higher rates of severe psychological distress among inmates when compared to the general population [58, 59].

The supplementary survey of homeless people conducted in Great Britain showed that the prevalence rates of common mental disorders, psychotic disorders, and alcohol and non-cannabinoid drug dependence were significantly higher in this group than the general population [52, 62]. In contrast, the supplementary survey of ethnic minority populations found that common mental disorders were more prevalent in people who were interviewed in English and would therefore not have been excluded from the general household survey [51].

Lastly, the supplementary survey of residents of institutions catering to people with mental illness in Great Britain showed a high proportion of people with severe disorders including schizophrenia, delusional disorder, schizoaffective disorder and affective psychoses, particularly in hospital settings [54]. Neurotic disorders, however, were more common in residential settings and comparisons to the general household survey data suggest that while people with these disorders do spend time in hospital, their length of stay in those facilities is likely to be relatively short in most cases [54].

Supplementary surveys of non-responders

The 2007 National Survey of Mental Health and Wellbeing (NSMHWB) in Australia and the Netherlands Mental Health Survey and Incidence Study (NEMESIS-2) attempted to contact non-responders for information on their mental health (Table 4). In both instances, an abridged version of the original survey was used, which allowed for some comparison between responders and non-responders [11, 13, 19].

Table 4 Non-response surveys and response-based analyses
Full size table

The NSMHWB Non-response Follow-up Survey (NRFUS) found that psychological distress, as measured by the Kessler Psychological Distress Scale (K10), was higher among non-responders than in the responding NSMHWB sample [13]. However, applying the NRFUS scores to the broader population of people who did not respond to the NSMHWB did not significantly increase the overall K10 score for the survey [13]. The NEMESIS-2 non-response survey found that compared to responders, non-responders were significantly more likely to have recent mood and anxiety problems as well as at least one impulse control symptom in childhood [19]. Despite the reduced survey content, participation rates for these follow-up surveys remained poor, with only 26.1% [19] to 40% [11, 13] of people who were contacted responding.

Another survey in Japan, the WMHJ2, analysed the association between response rate and mental disorder prevalence in different regions and conducted additional sampling of an area with a low response rate [18]. The findings showed no association between disorder prevalence and regional response rate; a lower disorder prevalence was reported for the area that was re-sampled when a higher response rate was achieved [18].

Discussion

Our review of national mental health surveys found that people not living in private dwellings in the community, especially those who are homeless, in health or correctional facilities, were commonly excluded from sampling, while males and young people were commonly underrepresented among survey respondents. Supplementary surveys of excluded populations and non-responders, where available, indicated higher prevalence of mental health problems in many of these groups. The common exclusion of key vulnerable groups and relatively high rates of non-response in these surveys have important implications for how results are used and interpreted.

The findings of this review show that most national mental health surveys recently conducted in high-income countries employ a similar approach and therefore have similar limitations concerning excluded populations and non-responders. This is not entirely surprising as six of the surveys included for review were either part of the WMH Survey Initiative or rooted in a previous WMH survey conducted in that country [11, 13, 18, 22,23,24,25,26]. Furthermore, given the recognition and prominence of the WMH Initiative, other independent surveys may have adopted similar sampling methodologies.

Four surveys included for review explicitly attempted to quantify the proportion of the population that would be excluded according to their methodology, with estimates ranging between 1% and 3.5% [15, 20, 25, 27, 29]. However, since countries may differ considerably with respect to the size of some excluded groups (e.g. military personnel, non-local language speakers), these figures may not be more widely representative. While the exclusion of such a small proportion of the total population may have had little to no impact on the overall survey results, as was sometimes articulated in the literature [19], excluded groups may have accounted for a far larger proportion of people in a particular subpopulation with specific risk factors and service needs (e.g. minority group, age cohort, or clinical severity group) [11, 63, 64]. In that instance, a difference in disease prevalence between included and excluded individuals could significantly affect the accuracy of disorder and service need estimates generated for that subpopulation. It may also impact the estimates of disease burden, particularly where individuals with more severe disorders are undersampled.

For example, data from 12 high-income OECD countries in 2019 showed that the proportion of people 80 years of age or older receiving care in long-term (i.e. residential) facilities (excluding hospitals) ranged between 10-19% [65]. Therefore, while this group may represent a relatively small proportion of a country’s total population, they would account for a far larger proportion of the elderly population. Research also suggests that mental disorders are more common among people residing in residential care relative to their community-dwelling counterparts [66, 67], yet only one of the surveys that met our inclusion criteria sampled this population [27, 29]. The exclusion of people in residential care from surveys may therefore result in an underestimate of both the disorder prevalence for this age group and the resources required to meet the needs of those living in these settings. As the populations in high-income countries continue to age, the impact of this group’s exclusion is likely to become even more profound [68].

Data from the supplementary surveys indicated that many of the other groups commonly excluded from population-based mental health surveys are also especially vulnerable to psychological distress and mental illness, including prisoner populations [53, 58, 59], military personnel [48,49,50, 57], homeless people [52, 62], people in hospitals and other health facilities [54], and people living in very remote areas [44,45,46]. It is particularly important to note the small number of surveys that included psychotic and personality disorders as well as the overrepresentation of individuals with a severe disorder among homeless populations [52, 62], prisoner populations [53] and people in hospitals and health facilities [54]. Individuals with severe mental disorders have high service needs, with estimates from England indicating that hospital admissions for these disorders account for 7% of total bed days for all health conditions [69, 70]. It is therefore imperative that service planning adequately account for the needs of these groups. One approach to gaining a better understanding of the size, distribution and needs of these populations is by conducting surveys through existing mental health services, like the Survey of High Impact Psychosis in Australia [71] and its predecessor the Study on Low Prevalence Disorders [72]. In countries with patient registers like Denmark, data on these populations are instead typically derived from admissions data [73, 74], however both methodologies are limited in that they rely on individuals with severe disorders being in contact with health services.

The lack of clear criteria around the inclusion or exclusion of people with cognitive impairment was another important finding of this review. Research has shown that individuals with a cognitive impairment or intellectual disability have a higher prevalence of mental disorders than those without, and have more complex service needs as a result of their comorbid conditions [75, 76]. As such, ambiguity regarding the inclusion of these individuals in national mental health surveys may complicate attempts to interpret and use survey data with respect to understanding and planning for mental health services overall and specifically for this population.

Survey results may be skewed if non-responders have a different mental health profile. Recent research shows that the response rates for health surveys in high-income countries have generally declined over time [77,78,79] and this trend was noted where previous iterations of the mental health surveys included in this review could be identified [14, 15, 19, 80,81,82,83,84]. Few studies attempted to capture non-responders or people in excluded groups through supplementary surveys or additional survey rounds, thereby limiting the generalisability of their results. In some cases, supplementary surveys also used different mental health measures, thus preventing the direct comparison or integration of results to those of the broader population surveys.

Similar to previous findings, supplementary analyses of non-response did not show a consistent relationship between survey response and mental health status. The first NEMESIS survey conducted in 1996 found that non-responders who agreed to complete the 12-item General Health Questionnaire (GHQ-12) had slightly better average mental health scores when compared to those who participated in the full diagnostic interview [80]. On the other hand, high levels of mental distress were associated with increased rates of attrition from the Nord-Trøndelag Health Study (HUNT) in Norway [85] and previous psychiatric diagnoses were tied to lower rates of participation in the population-based PART (Psykisk hälsa, Arbete och RelaTioner) study on mental health in Sweden [86]. Finally, while considerable evidence has linked heavy alcohol consumption, alcohol-related problems and financial support or hospital treatment for substance use disorders to lower rates of survey response [85,86,87,89,90], the opposite relationship has also been found [91,92,93,94].

Strategies adopted for the purposes of improving response rates included financial incentives, i.e. money or vouchers (Response Rate (RR): 57.3-65.1%) [12, 19, 20, 25], pre-survey notification (RR: 43.4-80%) [12, 15, 18, 19, 20, 24, 25, 27, 28], follow-up contact attempts (RR: 57.3-80%) [13, 15, 19,20,21, 25,26,27] and interviewing or contacting people face-to-face (60- 65.1%) [13, 19]. Interestingly, the survey that was mailed to potential participants had one of the lowest response rates (44.3%) [21] of the surveys reviewed, while the survey with the highest response rate (80%) had one of the shortest mental health questionnaires [27, 29], supporting previous findings that shorter questionnaires improve participation [95].

The underrepresentation of males and young people in our findings has also been reported in other health and health-related surveys [96,97,98,99,100]. One of the surveys included in our review oversampled younger people to ensure adequate representation [11, 13]. No surveys specifically adjusted their sampling methodology to account for sex or gender; however, nearly all surveys applied sampling weights to adjust for lower response rates and reflect the demographics of the general population [11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29]. It is possible that there may also be other populations that are underrepresented in survey findings but that were not specifically mentioned in the related literature, such as shift workers, fly in–fly out workers and long-haul transport drivers, all of whom have been found to be at elevated risk of poor mental health [101,102,103,104]. Other more specific issues with survey sampling methodologies, such as those described by Kalton et al.[105], were outside the scope of this review.

Planners and policymakers should be aware of the limitations of the mental health surveys that they use to understand mental health service needs and resource requirements. Ideally, efforts to sample non-responders and previously excluded populations would be included in all national mental health survey designs, or in complementary studies that use a standardised methodology to allow direct comparisons. However, given the significant additional resources required, sampling these populations may not always be feasible. In such instances, end users of survey results could consider other available data on the level of need within excluded populations to adjust estimates where appropriate. Adjustments of this kind were recently made to the prevalence estimates informing the mental health service demand and resource requirement modelling for adequate care in Australia’s NMHSPF (more information available in the NMHSPF V4.1 Technical Appendices [4]). Behan and Kennelly [106] used a similar approach to incorporate homeless and prison populations into their estimates for the cost of schizophrenia in Ireland. Given the increasing rates of non-response, groups conducting future surveys may also want to give greater consideration to factors that could influence participation, such as survey length and method of delivery, in their survey design [95].

Strengths and limitations

The lack of information available on which groups were included and excluded from many of the surveys was a key finding and limitation of this review. It suggests that more consistent reporting of these criteria in survey methodologies would be highly beneficial to improving our understanding of prevalence rates and facilitating comparisons between surveys. Another important limitation of this review was that the search for supplementary surveys of non-responders and excluded populations was restricted to those explicitly referred to in the primary survey literature. As a result, other potentially relevant surveys would not have been identified. For example, the First Nations Regional Health Survey conducted in Canada between 2008 and 2010 sampled First Nations communities that would have been excluded from the national mental health survey, but was not identified through our search criteria [107]. Unfortunately, a broader search of supplementary surveys was not feasible given the large number of excluded groups that were identified. Finally, as the review was limited to surveys conducted in high-income OECD countries, the findings may not be applicable to other settings.

Conclusion

This study showed that there are key populations that are often excluded from national mental health surveys in high-income countries (e.g. persons who are homeless, in hospitals or health facilities, or correctional institutions). The exclusion of these populations, and the few attempts to follow-up survey non-responders, may limit the generalisability of survey findings and result in underestimates of need for care. Collectively, our findings suggest the need for more inclusive sampling methods, or targeted population surveys, to strengthen the accuracy of prevalence estimates drawn from these surveys to inform policy and service planning decisions. If there are resource restrictions that limit the feasibility of these options, consideration should be given to whether prevalence estimates may be adjusted to account for any exclusions. Additionally, clear reporting on who is included and/or excluded in survey sampling procedures may ensure greater accuracy in the interpretation of survey findings.