Introduction

Several studies have attempted to clarify the theoretical approaches to assessing the quality of mental health services. However, to date, no review study has synthesized the theoretical, methodological, and measurement instruments used in such assessment. This integrative review study aims to address this gap by collectively synthesizing relevant theories, methodological issues, and measurement instruments used to assess mental health service quality. In doing so, it increases understanding of the conceptualization of mental health service quality assessment.

More specifically, the study aims to synthesize evidence on the theoretical frameworks, methodological considerations, and measurement instruments used to assess the quality of mental health services from consumers’ perspectives. The evidence compiled and analyzed in this study would inform clinicians and quality of care evaluators regarding the best approaches to measure improved mental health care. Further, the evidence provides an interpretation and collectively appraises the theoretical and methodological issues, which informs stakeholders and policymakers about the best way to monitor and evaluate population-based mental health services.

Concept of Quality

The quality of mental health services in this integrative review is defined as the degree to which services provided to consumers and populations increase the likelihood of desired outcomes and whether they are evidential and conform with best practice (Kelley and Hurst 2006); World Health Organization [WHO] (2003). The concept of quality also encompasses delivering equitable, consumer-centred, cost-effective mental health services (Funk et al. 2009).

The concept of improving the quality of health services has increasingly gained international consideration (Gaebel 1997; Hermann and Mattke 2004; Taube and Berzina-Novikova 2018; World Health Organization [WHO] 2003). Several efforts have been made to assess health care system performance and health care quality globally. For example, the WHO has played a leadership role in developing countries in monitoring quality checklists, glossaries, and documentation of health care policy, health care programs, and institutions (Gaebel 1997). Consistently, mental health services have also responded to the need to measure the quality of mental health services delivery.

Additionally, several international professional associations, multinational agencies, and government institutions have developed guidelines to govern the measurement of quality care delivery in mental health services. Agencies such as the Council of the European Union (Taube and Berzina-Novikova 2018), Royal Australian and New Zealand College of Psychiatrists (1982), the Canadian Psychiatry Association, the German Society of Psychiatry, Psychotherapy and Nervous Diseases (DGPPN 1993), and the Committee on Quality Assurance of the American Psychiatry Association have supported quality assurance in mental health services through position statements and guidelines (Gaebel 1997).

These professional associations and agencies have reviewed several criteria, concepts, and instruments required to implement quality assurance in psychiatric institutions and in regard to treatment delivery (Gaebel 1997). Recently, the Organisation for Economic Co-operation and Development promulgated quality assurance indicators to assist countries in assessing health service outcomes, including those of mental health services (Hermann and Mattke 2004; Taube and Berzina-Novikova 2018).

Moreover, several studies highlight the significant implications of quality assurance in mental health on stakeholder groups, such as consumers, families, providers, and policymakers (Gaebel 1997; World Health Organization [WHO] 2003). For instance, while quality assurance in mental health assists program managers and clinicians to assess the effectiveness and efficiency of services, it also informs consumers about service outcomes (World Health Organization [WHO] 2003).

Methods

Methodology

An integrative review was conducted incorporating a wide range of approaches, including defining concepts, reviewing theories, and analyzing methodological issues (Hopia et al. 2016; Whittemore and Knafl 2005). This review combines data from the theoretical and methodological literature using a five-stage process: (a) problem identification, (b) literature search, (c) data evaluation, (d) data analysis; and (e) presentation of findings (Hopia et al. 2016; Whittemore and Knafl 2005).

Data Sources and Search Strategy

We searched literature across six databases, PsycINFO, EMBASE, Medline, Scopus, Web of Science, and Google Scholar. We applied a three-stage process, including the initial search in Medline and EMBASE. In the second stage, we analyzed the keywords and search items retrieved to inform further searching across the remaining databases. The final stage of the search process involved hand searching reference lists of all articles for additional relevant articles. The keywords, items, and Boolean operators used to search data sources were:

(“theor*” OR “model*” OR “dimension*” OR “concept*” or “framework*” OR “component*”) AND (“*Patient Satisfaction” OR “patient” OR “consumer” or “consumer”) AND (“psychiatry care” OR “mental health” OR “*Mental Health Services”) AND (“Efficacy*” OR “effective*” OR “appropriate*” or “acceptab*” or affordab* or * or efficien* or access* or equit*) or AND (“relevan*” OR “empath*” OR “Qualit*” OR “satisfact*” OR “improvem*” OR “*Quality of Health Care” or “Outcome Assessment” OR “*treatment outcome” OR “recovery” OR “person-centred outcome”).

Eligibility Criteria

Articles retained focused on theoretical, conceptual, and methodological issues regarding consumer perspectives in assessing mental health service quality. The inclusion criteria were peer-reviewed articles published in the English language, which described the dimensions and domains used to measure mental health service quality as well as those validating instruments used for such outcomes. Similarly, studies of all designs and those that were not limited to a particular year of publication were included.

Studies were excluded if they are published in a language other than English, used empirical data to measure the outcome of mental health services, measured the administrative aspects of quality control, national regulatory documents on quality assurance, assessed the clinical status of consumers, as well as quality measures that target service providers.

Data Management and Extraction

Endnote X8 was used to manage all aspect of the articles, particularly screening and handling duplicate references. A predefined data extraction form was developed to extract data from all included articles (see “Appendix 1”). The form was developed using manuals from Joanna Briggs Institute (Pearson et al. 2014) and Cochrane (Moher et al. 2009), and variables from the literature on conceptual and methodological reporting guidelines. The subheadings in this form include study details, methods, study setting, objectives, article category and subject area (i.e., theory of mental health service quality, theory history, theory components/dimensions, intersection of theories, methodological issues, and measurement instruments), additional information and recommendations, and additional references to check. Two reviewers independently managed the data extraction process.

Selection of Studies

The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flowchart for systematic reviews (Moher et al. 2009) was used to illustrate the selection processes (see Fig. 1). The selection process of studies had three stages (Pearson et al. 2014). First, two authors independently screened the titles of retrieved articles and identified those meeting the selection criteria. Second, the authors reviewed all the titles and abstracts and agreed on those needing full-text screening. Accordingly, in the full-text screening, all the articles that met the inclusion criteria were approved and included in the final synthesis.

Fig. 1
figure 1

Flow chart of studies included in the review

Full size image

Data Synthesis

The extracted data were analyzed using thematic analysis (Braun and Clarke 2006; Mayring 2014; Pearson et al. 2014). First, the extracted data were open coded (Braun and Clarke 2006). The emerging codes were further grouped into subthemes and themes of commonality. These themes and subthemes are used to present the review findings (see Table 2).

Results

Study Characteristics

A total of 834 articles from all search sources was identified, after which 26 duplicates were removed. Next, 808 titles and abstracts were assessed for eligibility. A total of 54 full-text articles were identified, and an additional seven were identified from the reference lists. Of these, 31 were excluded and 30 were included in the final synthesis (see Fig. 1).

Of the 30 articles included, 16 contained measurement instruments used to assess mental health service quality, while 14 addressed theoretical and methodological issues. Similarly, 28 targeted mental health services alone and two focused on general health care. Further, ten articles targeted global issues, 10 focused on the United States, and three each targeted Australia and Italy (see Table 1). The review identified several themes and subthemes (see Table 2).

Table 1 Study characteristics
Full size table
Table 2 The key emerging themes
Full size table

Theory for Measuring Mental Health Service Quality

Two theoretical frameworks were identified to explain and measure mental health service quality. The theoretical frameworks are Programme Theory (Birleson et al. l2001) and the Donabedian model (Duckers and Thormar 2015; Grabowski et al. 2010; Kilbourne et al. 2018; McGrath and Tempier 2003; Salzer et al. 1997; Watkins et al. 2010). The programme theory explains the connections between programme operations and outcome (Birleson et al. 2001). The programme theory is predominantly explained and measured using three major dimensions, namely program operations, intermediate and final outcomes (Birleson et al. 2001). Conversely, the Donabedian model has three major components, which include structure, process, and outcome (Duckers and Thormar 2015; Grabowski et al. 2010; Kilbourne et al. 2018; McGrath and Tempier 2003; Salzer et al. 1997; Sixma et al. 1998; Watkins et al. 2010). The Donabedian model has extensively been applied in measuring the quality of general health services and has been adapted to the mental health specialty (Grabowski et al. 2010). The current review adapts the components of the Donabedian model to explain mental health service quality assessment.

Components of Theory on Mental Health Service Quality

Structure

Several articles emphasized an interrelated explanation of structure as a component of measuring mental health service quality (Donabedian 2005; Duckers and Thormar 2015; Kilbourne et al. 2018; McGrath and Tempier 2003; Salzer et al. 1997; Schroder et al. 2007; Watkins et al. 2010). These articles generally described the structure as the characteristics of the existing health systems facilitating the delivery of these services. These characteristics relate to the availability, adequacy, affordability, and accessibility of existing services. Some studies explained structure as the availability of competent service providers, adequacy of facilities and equipment (Donabedian 2005; Duckers and Thormar 2015; Kilbourne et al. 2018; Watkins et al. 2010), and management of mental health workforce (Donabedian 2005; Duckers and Thormar 2015; Kilbourne et al. 2018). Similarly, affordability describes financial resources as well as the provision of sufficient funding (Duckers and Thormar 2015). Some studies described the structure as the accessibility of existing facilities, including physical and organizational settings (Donabedian 2005; Salzer et al. 1997). Further, structure is described as the governance of the existing mental health services, including the operations of the institutions, multi-agency planning group and coordination (Donabedian 2005; Duckers and Thormar 2015; Kilbourne et al. 2018), information technologies (Kilbourne et al. 2018), and the environment of mental health service delivery (Schroder et al. 2007).

A further study categorized structure of mental health services into four subcomponents namely community, institutional, provider, and consumer characteristics (Salzer et al. 1997). The community characteristics comprised the existing community resources, service demand, demographics, and economy whilst the institutional characteristics were the organization of services, organizational structure, funding sources, facility size, staff and patient ratios and sources of payment (Salzer et al. 1997). The provider characteristics are the provider demographics, such as health disciplines represented, years of experience, and training received, while the consumer characteristics are the consumers’ demographics, insurance coverage, clinical history, risk, and protective factors (Salzer et al. 1997). Generally, the institutional, provider and community-related factors describe the health system-related issues that contribute to the use of mental health services, while the consumer characteristics are the factors related to the individual (see Fig. 2).

Fig. 2
figure 2

Extended Donabedian model for measuring quality of mental health services

Full size image
Process

Several studies have conceptualized process as the actual treatment stage in mental health services. For example, the process highlights consumers’ participation in mental health service delivery (Duckers and Thormar 2015; McGrath and Tempier 2003; Schroder et al. 2007). Many studies have recommended that process should address interpersonal relationships and technical skills, especially the use of evidence-based practice (Donabedian 2005; Duckers and Thormar 2015; Kilbourne et al. 2018; Salzer et al. 1997; Watkins et al. 2010). The interpersonal aspect of process highlights the therapeutic relationship between clients and providers, communication and dissemination of information, as well as interprofessional decision-making as part of the treatment (Duckers and Thormar 2015; Salzer et al. 1997; Schroder et al. 2007). The therapeutic relationship component addresses the respect and dignity provided to consumers, establishing good therapeutic relationships between consumers and providing consumers with an adequate sense of security and relevant information, ensuring close consumer and service provider collaboration, as well as providing the consumer opportunities for self-determination (Schroder et al. 2007). The therapeutic relationship also includes measures to provide consumers with high levels of access, continuity, and trust (Schroder et al. 2007).

Moreover, the technical aspect of the process generally describes clinicians’ knowledge about appropriate intervention and best practices as well as the ability to accurately assess consumer problems (Donabedian 2005; Duckers and Thormar 2015; Kilbourne et al. 2018; Salzer et al. 1997). For instance, Donabedian (2005) suggested several measures to describe technical skills, including appropriateness, completeness, and redundancy of information; justification of diagnosis and therapy; and technical competence in performing diagnostic and therapeutic procedures.

Outcome

Most articles reviewed highlighted the measures used to assess the outcome of mental health services (Donabedian 2005; Duckers and Thormar 2015; Kilbourne et al. 2018; McGrath and Tempier 2003; Montgomery et al. 2009; Salzer et al. 1997; Thornicroft and Slade 2014; van Hasselt et al. 2015; Watkins et al. 2010). Several studies suggested that the outcome is generally measured according to the changes in the current and future mental health problems of consumers (Duckers and Thormar 2015; Salzer et al. 1997; Watkins et al. 2010). Thus, the evidence reviewed describes the outcome as the ultimate validator of mental health service effectiveness. Overall, the outcomes measure the effects of episodic mental health services on the wellbeing and health of individuals and populations (Donabedian 2005; Duckers and Thormar 2015). Typically, these outcomes assess whether the mental health services received by consumers actually improve symptoms (Duckers and Thormar 2015; Kilbourne et al. 2018; McGrath and Tempier 2003). The ability to explicitly identify the outcome domain will increase the theoretical coherence between mental health intervention and outcome (Thornicroft and Slade 2014).

The outcomes assessed can be categorized into clinical outcomes (Thornicroft and Slade 2014) and consumers’ perspective outcome (see Fig. 2) (Salzer et al. 1997; Schroder et al. 2007). The clinical outcome assessments are measured according to symptomatology, service use (admission rates and episodes of service), and recovery (mental health system reports) (McKay et al. 2012; Thornicroft and Slade 2014). Several studies have recommended that assessments of mental health service outcomes should not only be focused on clinical assessments, but also consider consumer-perspective outcomes, particularly those meaningful to consumers (Kilbourne et al. 2018; Reininghaus and Priebe 2012; Sixma et al. 1998) as well as specific populations, such as children (Kilbourne et al. 2018). Typically, consumer-perspective outcomes are evaluated by asking whether the services provided to consumers bring any differences in their lives. Understanding outcomes from this perspective can comprehensively determine the quality of mental health services beyond what happens within the clinical assessment of outcome (Kilbourne et al. 2018; McGrath and Tempier 2003).

Consequently, advocacy is increasing for including consumers’ perspectives of outcomes to balance the portfolio of measures. This outcome perspective can provide consumer experiences about the structure of the health systems and perceived self-reported outcomes (Kilbourne et al. 2018). The review findings revealed several indicators that can be used to assess the consumers-perspective outcome of mental health service. These indicators include changes in symptoms, functioning (restoration of social functioning) (Kilbourne et al. 2018; McGrath and Tempier 2003; Montgomery et al. 2009; Reininghaus and Priebe 2012), quality of life (McGrath and Tempier 2003; Montgomery et al. 2009; van Hasselt et al. 2015), satisfaction (McGrath and Tempier 2003; Montgomery et al. 2009; Salzer et al. 1997), personal recovery (Donabedian 2005; Schroder et al. 2007), physical health-related knowledge (Salzer et al. 1997), coping abilities (Montgomery et al. 2009), family caregiving burden (Montgomery et al. 2009), and economic empowerment (participation in economic activities) (Kilbourne et al. 2018; McGrath and Tempier 2003) (see Fig. 2).

Moreover, international consensus is emerging about recovery orientation in mental health services delivery (Donabedian 2005; Duckers and Thormar 2015; Kilbourne et al. 2018; Schroder et al. 2007; Thornicroft and Slade 2014). The recovery-oriented approach to assessing mental health service quality highlights evidence-based treatment that incorporates supportive guidance, conversational therapy, opportunities for occupation, and post-care follow-up (Schroder et al. 2007; Thornicroft and Slade 2014). In particular, this approach describes a deeply personal, unique process of changing one’s attitudes, goals, values, skills, and feelings, as well as living a hopeful, satisfying, and contributing life, even when mental illness causes some limitations (Thornicroft and Slade 2014). In addition, a recent review has recommended that a recovery orientation to assessing the quality of mental health services should be categorized according to five processes, that is, connectedness (social inclusion; community integration), hope and optimism, development of a positive identity, meaningfulness in life, and empowerment (Thornicroft and Slade 2014).

The Intersection of Structure, Process, and Outcome

Several studies have suggested the interconnections of the three components (structure, process, and outcome) used to assess mental health service quality (Kilbourne et al. 2018; McGrath and Tempier 2003; Salzer et al. 1997; Watkins et al. 2010). Generally, the three components are highly interlinked and hence can facilitate the understanding of consumers’ perspectives outcome of service quality (Kilbourne et al. 2018; Salzer et al. 1997). The structure and process measures are often argued to serve as proxies for the outcome measures (McGrath and Tempier 2003; Watkins et al. 2010). The extent to which a structure and process influence an outcome of interest is determined by the degree of the relationship (Salzer et al. 1997), which could be direct or indirect (Watkins et al. 2010). The ability to effectively combine the structure and process subcomponents that could help achieve quality mental health services would assist policymakers and administrators in making decisions about service efficacy (Salzer et al. 1997). For instance, Kilbourne et al. (2018) argued that an adequate structure could create the necessary infrastructure and equipment for reporting on processes and outcomes as well as conducting improvement activities (see Fig. 2). However, these measures do not provide adequate information on whether the quality of mental health services or outcome obtained, is acceptable to consumers. The indicator within each of the structure and process is expected to correlate with at least one of the outcomes of interest areas to achieve a quality indicator (Salzer et al. 1997).

Several studies have highlighted assumptions that need to be considered when establishing a relationship between each of the quality indicators (Kilbourne et al. 2018; Salzer et al. 1997). Although a good interpersonal relationship established between clinician, child, and family could build consumer’s satisfaction, this subcomponent may not be associated with outcome domains, such as functioning and symptoms (Salzer et al. 1997). A particular outcome of interest may not be sufficient to assess the quality of mental health services, and it may require examining the subcomponents of structures and processes that are known to lead to a desirable outcome (see Fig. 2).

Methodological Considerations in Assessing Quality of Mental Health Services

Methodological Considerations

The review findings identify several methodological issues that need consideration when assessing mental health service quality (Ahlfors et al. 2001; Arya and Callaly 2005; Attkisson and Greenfield 2004; Birleson et al. 2001; Donabedian 2005; Duckers and Thormar 2015; Grabowski et al. 2010; Kilbourne et al. 2018; McGrath and Tempier 2003; McKay et al. 2012; Montgomery et al. 2009; Salzer et al. 1997; Schene et al. 2000; Sixma et al. 1998; Thornicroft and Slade 2014). These methodological considerations are issues that can create challenges when assessing such quality, which is largely ascribed to mental health consumers’ sensitivity and vulnerability. In particular, such issues have several implications when assessing consumer perceptions about mental health services. Systematic consideration of the methodological issues is significant, particularly since it may reduce the risk of selection and information biases, reliability, and external validity of assessment results.

The review showed that there is value in considering the context and cultural setting of the services (Arya and Callaly 2005; Ching 2011; Grabowski et al. 2010; McKay et al. 2012; Salzer et al. 1997). More importantly, the effective recognition of the context and cultural norms of the society can inform evaluators about ways to contextualize the measurement instruments. Failure to address these issues can negatively affect the assessment, particularly in settings where the etiology of mental illness is largely conceptualized by the population within religious–cultural norms. Specifically, cultural differences are significant when assessing mental health consumers satisfaction (Ching 2011). Another consideration is that a valid quality of mental health service indicators in one culture may not be appropriate in others, and hence, a simple translation of instruments may not maintain the same meaning across different cultures (Ching 2011; Salzer et al. 1997).

The present review’s findings highlight the need to consider the outcome of interest (Duckers and Thormar 2015; McGrath and Tempier 2003; Thornicroft and Slade 2014) as well as perspective (of consumers or providers) (Sixma et al. 1998; Thornicroft and Slade 2014). The perspective of interest is critical to increasing understanding of the outcome indicator to use in assessing mental health service quality. Specifically, Thornicroft and Slade (2014) recommended three perspectives of interest when measuring the outcome of mental health services, which are those of consumers and/or family and friends, service providers, and the public. The family, caregivers, and friends are known to have more contact with consumers than service providers and hence may have powerful consequences.

The service provider perspective mostly describes the strength and weakness of existing health systems. In addition, public perspectives mostly address the outcomes of their investment in mental health services (taxation and financing). This is assessed through a reduction in antisocial behaviour such as aggressive begging, or in behaviour considered “odd”, such as shouting at voices (Thornicroft and Slade 2014). As discussed, the outcome of interest from consumers’ perspective can be categorized according to changes in symptoms, functioning (restoration of social functioning), quality of life, satisfaction, patient recovery, physical health-related knowledge, coping abilities, family caregiving burden, and participation in economic, social, education and political activities (see Fig. 2). Generally, the ability to provide a clear understanding of these outcomes of interest and perspectives is significant in effectively aligning the appropriate measurement instruments to use.

Another methodological issue that needs consideration is the evaluator or individual responsible for assessing mental health service quality (Ahlfors et al. 2001; Birleson et al. 2001). Such assessment by an evaluator directly involved in the service delivery can influence the assessment in several ways, positively or negatively. The review revealed that quality assessment of mental health services should be conducted by an external evaluator who represents the health care organization but is not directly involved in the care of the consumers (Ahlfors et al. 2001; Birleson et al. 2001). This finding is ascribed to the fact that service providers directly involved in service provision may negatively or positively bias the assessment (Ahlfors et al. 2001).

In addition, the review revealed the need to consider data collection methods and instruments to use in assessing the mental health service quality. This methodological consideration addresses issues such as selecting sources of information, determining process of consumer selection (Kilbourne et al. 2018), generating validated instruments and indicators (Ching 2011; Salzer et al. 1997), standardizing and structuring the interview (Schene et al. 2000; Sixma et al. 1998), and obtaining outcome data that are statistically applicable to inform decisions (McGrath and Tempier 2003). In particular, the source of information used to assess mental health service quality is critical, particularly in relation to the generalizability of the findings. This implies that evaluators should fully understand whether qualitative or quantitative data are relevant in assessing mental health service quality. In each case, critical consideration should be given to the process of collecting such data. The ability to address these methodological issues could help reduce any effects on the assessment results. Conversely, standardizing and structuring of the interview, as well as thorough training, could practically reduce effects of these methodological limitations, particularly any idiosyncratic characteristics of the interviewer (Schene et al. 2000; Sixma et al. 1998). In some instances, several adjustment approaches need to be employed to minimize the underlying risk factors associated with consumers. These measures are usually associated with the severity of illness, medical history/health status, and sociodemographic factors (Kilbourne et al. 2018).

Instruments Used to Measure Mental Health Service Quality

The instruments used to assess mental health service quality have relevance. Indeed, the ability to identify the appropriate instruments that are validated and culturally sensitive has important implications regarding the accuracy of quality assessment. However, these instruments may differ according to scope, content, and application. The review identified several instruments used to assess mental health service quality. A total of 32 instruments were identified for assessing consumers satisfaction (Ahlfors et al. 2001; Amatulli et al. 2005; Attkisson and Greenfield 2004; Boyer et al. 2009; Brannan et al. 1996; Ching 2011; Gigantesco et al. 2003; Hansson and Höglund 1995; Henderson et al. 2003; Ivarsson and Malm 2007; Miglietta et al. 2018; Montgomery et al. 2009; Pellegrin et al. 2001; Ruggeri et al. 2007; Schene et al. 2000), nine instruments for assessing quality of life (Amatulli et al. 2005; Montgomery et al. 2009; Schene et al. 2000; van Hasselt et al. 2015), five for assessing symptoms (Montgomery et al. 2009; Ruggeri et al. 2007; Thornicroft and Slade 2014), and five for assessing functioning (Montgomery et al. 2009; Thornicroft and Slade 2014). Similarly, one instrument each measured needs (Schene et al. 2000; Thornicroft and Slade 2014), caregiving (Schene et al. 2000), and costing of mental health services (Schene et al. 2000) (see Table 3).

Table 3 Measurement instruments
Full size table

The review showed that most of the instruments are designed specifically for mental health services, while others are used to measure general health care, including mental health. For instance, a recent systematic review showed that 25 out of 28 instruments for assessing satisfaction were developed specifically for mental health services, while three of them do not explicitly focus on mental health care (Miglietta et al. 2018). Four of the 28 instruments are widely used for measuring satisfaction. These instruments are the Client Satisfaction Questionnaire, Verona Service Satisfaction Scale (VSSS), Client Assessment of treatment, and Self-Rating Patient Satisfaction Questionnaire.

In addition, Montgomery et al. (2009) indicated that the Client Satisfaction Questionnaire is the oldest satisfaction instrument with known psychometric properties. Alternatively, the VSSS is the widely used validated questionnaire specifically designed to assess consumer satisfaction in a psychiatric setting (Ching 2011; Hansson and Höglund 1995). The VSSS is a validated multidimensional scale that has been used in a comparative, cross-national, cross-sectional study of consumers with mental illness. This instrument has been applied in several settings and intervention studies over the past decades and is proven to be a sensitive, useful tool that identifies the strengths and weaknesses of services and covers a wide range of services provision aspects (Ching 2011). The increasing number of satisfaction instruments reinforces the fact that satisfaction directly reflects consumer perceptions about mental health services.

Moreover, the Brief Psychiatry Rating Scale used to measure symptoms, is the most commonly used measure of symptom outcomes (Montgomery et al. 2009). Comparatively, while an increasing number of instruments are available for measuring mental health services outcomes, Montgomery et al. (2009) argued that there is a lack of information about instruments specifically designed to measure self-care. In particular, self-care tends to narrowly focus on consumers adherence to their prescribed regimen and appropriate use of services to manage their symptoms. Service providers use health education and support services to enhance adherence and compliance with such prescriptions (Montgomery et al. 2009). These consumers use several coping or safety strategies to manage challenging circumstances (Montgomery et al. 2009).

Content of the Instruments Used to Measure Mental Health Service Quality

The content of instruments used in assessing the quality of mental health services is relevant, particularly in helping evaluators to decide the sub-items to include in the assessment. The content of the instruments also helps researchers to understand the sub-items that could adequately explain consumer-perspective of the outcome. The study mapped the content of instruments used to measure mental health service quality according to structure, process, and outcome. However, the content differs across this spectrum of assessing mental health service quality. In fact, not all such outcome measures have content across the structure, process, and outcome, and the review reveals that only consumer satisfaction instruments have content across these three indicators. The process of assessing mental health service quality had the greatest content, followed by outcome and structure. For instance, a total of 23 measures included content for the process, 10 included for the outcome, and seven included content for structure (see Table 4). As explained earlier, the process constitutes the interpersonal relationships between consumers and providers as well as the technical skills of service providers (Duckers and Thormar 2015; Salzer et al. 1997). The actual treatment and medication, including the appropriateness of clinical guidelines (Donabedian 2005; Kilbourne et al. 2018; Salzer et al. 1997), occur at this stage, and thus, consumers are able to assess their satisfaction. This could possibly contribute to the increased content on measures of the process of mental health services. In addition, other indicators, such as quality of life, functioning, and caregiving, had no content for structure and process and were limited to the outcome. In fact, a total of nine measures had content relevant to the outcome, four had for functioning, and three for caregiving. The review findings suggest that no information is collected on structure and process when assessing consumer perspective outcome of mental health services using quality of life, functioning, and symptoms. This suggests that although structure and process can influence these outcome measures, no direct relationship exists when assessing mental health service quality.

Table 4 Content of measurement instruments
Full size table

Conclusion

This study synthesized evidence on the existing theories, methodological issues, and instruments used to assess consumer perspectives of mental health service quality. The review identified two main theories largely used to assess such quality: Donabedian and Programme. The Donabedian theory is the leading theory, widely used to facilitate understanding of mental health service quality. This model assesses such using three dimensions: structure, process, and outcome. This review concludes that several theoretical and methodological issues need critical consideration when assessing mental health service quality from consumers’ perspectives. These issues are largely centered on context and cultural norms of the services, outcome perspectives (of consumers or providers), outcomes of interest, the individual responsible for assessing such quality, sources of information, selection of consumers, generating indicators and validated instruments, standardizing and structuring of the interviews as well as obtaining outcome data that are statistically applicable to inform decisions. Additionally, the study identified several instruments used to assess mental health service quality, but mostly focused on satisfaction; other instruments measure the quality of life, functioning, and symptoms.

Strength and Limitations

The study has several strengths and limitations that require explanation. The review applied a systematic approach and methods to search and extract data using a predefined data extraction form. The data extraction form helps capture comprehensive information on the subject. Although several studies have attempted to systematically review the trend of measurement instruments, this review is unique, because it collectively synthesizes the theoretical and conceptual guidelines, as well as measurement instruments. The review did not limit the search to a specific period and hence possibly could identify most relevant articles.

The study has several limitations, largely pertinent to the search words, language limitations, and assessment of the methodological quality of included articles. Generally, the variations in key terms and concepts regarding theoretical and conceptual guidelines may have resulted in missing some relevant articles pertaining to the study. Similarly, limiting studies to only those published in English could have resulted in missing useful articles published in other languages. Again, the review did not assess the methodological quality of the included articles, using critical appraisal tools. However, the combination of clearly articulated search methods, consultation with research Librarian, and reviewing articles with multiple experts helped address the various limitations.

Implications for Policy and Clinical Practice

This integrative review showed that considerable evidence exists on theoretical and methodological issues as well as measurement instruments when assessing mental health service quality from consumer perspectives. More importantly, the evidence largely addresses methodological considerations as well as measurement instruments used to assess the quality of mental health services. In addition, the theories and concepts for assessing mental health service quality are consistent with the measurement instruments as well as their content. Based on the evidence, we make several recommendations to support policy and clinical practice:

  • The current awareness and advocacy for addressing the perspective of consumers in assessing mental health service quality should be prioritized in national and international quality assurance development.

  • Researchers and quality improvement personnel who are designing or evaluating the quality of mental health services should consider, at a minimum, the three domains of structure, process, and outcomes.

  • Clinicians and researchers who are designing or evaluating mental health service quality should consider several methodological issues. These methodological considerations can be integrated into the existing quality assurance indicators used in clinical practice.

  • Clinicians and researchers interested in measuring mental health service quality should not only address clinical outcome assessments but also consider consumers’ perspectives in such measurements.

  • In addition, the present review showed that the development of measurement instruments to assess mental health service quality from consumer perspectives has largely targeted satisfaction. Based on this evidence, we recommend that clinicians interested in measuring such quality should not only target consumer satisfaction but also consider other indicators, such as consumers’ quality of life, functioning (social disability), symptoms, caregiving burden, and coping strategies.