Abstract
Background
Intersectoral costs and benefits (ICBs), i.e. costs and benefits of healthcare interventions outside the healthcare sector, can be a crucial component in economic evaluations from the societal perspective. Pivotal to their estimation is the existence of sound resource-use measurement (RUM) instruments; however, RUM instruments for ICBs in the education or criminal justice sectors have not yet been systematically collated or their psychometric quality assessed. This review aims to fill this gap.
Methods
To identify relevant instruments, the Database of Instruments for Resource Use Measurement (DIRUM) was searched. Additionally, a systematic literature review was conducted in seven electronic databases to detect instruments containing ICB items used in economic evaluations. Finally, studies evaluating the psychometric quality of these instruments were searched.
Results
Twenty-six unique instruments were included. Most frequently, ICB items measured school absenteeism, tutoring, classroom assistance or contacts with legal representatives, police custody/prison detainment and court appearances, with the highest number of items listed in the Client Service Receipt Inventory/Client Sociodemographic and Service Receipt Inventory/Client Service Receipt Inventory–Children’s Version (CSRI/CSSRI/CSRI-C), Studying the Scope of Parental Expenditures (SCOPE) and Self-Harm Intervention, Family Therapy (SHIFT) instruments. ICBs in the education sector were especially relevant for age-related developmental disorders and chronic diseases, while criminal justice resource use seems more important in mental health, including alcohol-related disorders or substance abuse. Evidence on the validity or reliability of ICB items was published for two instruments only.
Conclusion
With a heterogeneous variety of ICBs found to be relevant for several disease areas but many ICB instruments applied in one study only (21/26 instruments), setting-up an international task force to, for example, develop an internationally adaptable instrument is recommended.
Similar content being viewed by others
Avoid common mistakes on your manuscript.
Health-related interventions often incur costs and benefits outside the healthcare sector (i.e. intersectoral costs and benefits [ICBs]) which, if relevant, should be considered in economic evaluations and cost-of-illness studies from the broader societal analytical perspective to arrive at unbiased conclusions. |
Reliable and valid instruments to measure such resource use in these sectors are pivotal. Based on a systematic review of existing instruments used in applied economic evaluations, this paper provides an overview of generic and disease-specific resource-use measurement instruments in the education and criminal justice sectors, including their psychometric properties. Relevant instruments will be included in the Database of Instruments for Resource Use Measurement (DIRUM; http://www.dirum.org). |
Many instruments identified in this review were specifically applied in one study only, potentially implying considerable duplication of work across studies. Hence, bundling individual efforts could be a more cost-effective strategy overall, and setting-up an international task force to support these activities, e.g. by fostering methodological transparency and developing an internationally adaptable, harmonized instrument including relevant ICB items, is thus suggested. |
1 Introduction
For economic evaluations to be a useful tool to inform policy makers regarding the allocation of scarce resources, accurate measurement and valuation of all relevant costs and benefits related to an intervention is key. Costs and benefits related to healthcare interventions that occur outside the healthcare sector, i.e. intersectoral costs and benefits (ICBs) [1, 2], have been shown to be a vital component in economic evaluations applying a societal perspective [3,4,5,6]. At the same time, taking a societal viewpoint in economic evaluations is recommended by guidelines for the use of economic evaluation in several countries [7], amounting to a proportion of approximately two-thirds of the analyzed countries in 2010 [8], and most recently also including the US [9]. The societal perspective implies that all benefits and costs that are directly or indirectly caused by the intervention (if significantly present) should be included in the economic evaluation, independently of who incurs them [10]. These costs and benefits can occur outside the healthcare system, including the education and criminal justice sectors, and household and leisure activities, in addition to productivity loss [6].
In recent research, generic methods for valuing ICB unit costs have been developed and applied in the context of the Dutch education and criminal justice system [11]. However, the issue of how to measure such intersectoral resource use in trial-based economic evaluations and cost-of-illness studies with standardized resource-use measurement (RUM) instruments has received little systematic scientific attention to date [12]. This might hinder the consideration of relevant ICB resource use in economic evaluations in the first place [9], and consequently lead to biased results from a societal perspective [13]. It also potentially creates heterogeneity in the measurement due to variabilities in considered cost components, which in turn impairs the comparability of cross-study cost estimates.
In addition, it is unclear to what extent the quality of existing ICB RUM instruments in terms of their psychometric properties has been evaluated. Sound instruments are pivotal to adequately capture the impact of healthcare interventions in other sectors. A review of health-related RUM instruments that led to the set-up of the Database of Instruments for Resource Use Measurement (DIRUM) in 2011 [14] showed that limited evidence is available regarding the validity, and especially reliability, of the listed instruments [15]. This presumably also applies to instruments measuring ICBs. One major exception includes the measurement of lost productivity, for which both a number of RUM instruments have been developed, validated, and the quality of their validation evidence critically appraised in recent publications [16, 17]. Similarly, regarding ICBs in the household and leisure sectors, instruments to measure informal care have been collected and/or assessed in previous research [18,19,20]. This is not yet the case for the education or criminal justice RUM instruments.
Against this background, this research focuses on health-related ICBs linked to resource use in the education sector (i.e. related to the impairment of educational achievement of individuals [2, 9]) and criminal justice sector (i.e. related to the costs of crimes [2, 9]), and has two aims. Our main aim was to provide a systematic overview of the characteristics of current RUM instruments used in health economic analyses that include ICB items capturing the impacts on the education and criminal justice sectors. Second, we sought to determine the existence of psychometric quality assessments of these instruments. This will not only provide practical guidance for researchers to identify sound instruments for ICB RUM in specific disease areas but also help detect disease areas for which such measurement in economic evaluations might be crucial. In addition, it will provide an evidence-base to potentially pioneer the development of standardized new items to measure relevant costs and benefits outside the healthcare sector.
2 Methods
2.1 Identification of Intersectoral Cost and Benefit (ICB) Instruments
2.1.1 Databases
To identify instruments with relevant ICB items, a stepwise approach was adopted. First, the open-access repository DIRUM (http://www.dirum.org) [14], listing 77 RUM instruments in July 2016, was hand-searched. DIRUM contains full texts of health-related RUM instruments, as well as information and/or references about respective instrument qualities. DIRUM has a search function allowing the selection of specific ‘items of resources being measured’ (e.g. criminal justice service, educational, other non-National Health Service), however, all of the 77 listed instruments were examined in detail.
Second, as DIRUM is not exhaustive [21], an extensive systematic literature search was conducted. Herein, the adopted research approach rests on the assumption that existing ICB instruments that measure such resource use are cited and/or listed in economic evaluations and cost-of-illness studies that were conducted from a societal perspective. Seven electronic databases were searched in July 2016, including MEDLINE (PubMed), EMBASE (Ovid), Social Science Citation Index (SSCI; Web of Science), PsycINFO, Econlit, Education Resources Information Centre (ERIC) and CINAHL (EBSCOhost). This covers two general medical literature databases (MEDLINE, EMBASE) [22], which, in combination with CINAHL and PsycINFO, are considered appropriate to identify economic evaluations [23]. SSCI, Econlit and ERIC were included to incorporate a social science, education and economic perspective, respectively. No date or language restrictions were applied (electronic supplementary Appendix Table A1). Both search strategy and database choices were discussed and agreed with the project team and an information scientist.
2.1.2 Inclusion and Exclusion Criteria
Eligibility criteria of the identified studies, framed around PICOS [24], included all non-institutionalized and institutionalized age groups of the population (P), regardless of the intervention (I), comparators (C) and outcomes (O), set up as a trial-based (non-simulation/non-model) full economic evaluation (i.e. comparing both costs and outcomes of the intervention group and at least one comparator) or as a cost-of-illness study, adopting a societal perspective (i.e. including costs in both the healthcare sector and other sectors) as the study design (S) and measuring ICBs in their analysis, published as a full paper or report. Consequently, study exclusion criteria included no original research, no full economic evaluation or cost-of-illness study, based on model or simulation, and not adopting a societal perspective. Studies building on nationwide population surveys were excluded, whereas articles focusing on the description of RUM instruments were also screened. All studies fulfilling the inclusion criteria irrespective of language were reviewed for referenced patient/carer-reported ICB RUM instruments in the education or criminal justice sectors (and information on relevant psychometric evidence). Instruments either had to measure education or criminal justice resource use, or the article had to refer to a specified (published) instrument that could potentially include such items. Relevant information about the instrument was extracted.
2.2 Retrieval of ICB Instruments
Full texts of instruments identified through DIRUM were readily available from the DIRUM website [14]. Full texts of the instruments identified through the database search of economic evaluations and cost-of-illness studies, if not included in the identified publication itself, were retrieved via two channels. A Google (Scholar) search was conducted first. If unsuccessful, the (corresponding) authors of the articles were then contacted via email, and a full version of the utilized instrument (and, in a second step, information about the psychometric properties) was requested. No language restrictions were applied in terms of the included instruments, and translators were used for extraction. Instruments had to be available as a full version (for free).
2.3 Identification and Retrieval of Studies Assessing Instrument Properties
Regarding psychometric properties that assess the quality of the identified instruments, this review focuses on their validity and reliability. Validity captures the degree to which an instrument measures what it intends to measure, while reliability refers to the ability of an instrument to do this in a consistent manner [25]. To identify studies assessing the instruments in this respect, the references of the studies listed in DIRUM under ‘instrument qualities’ were investigated. For the instruments included based on the systematic literature review, secondary database searches were carried out based on the names of the instruments and other relevant additional information found in the initially reviewed studies. The same databases were searched as for the identification of the instruments.
2.4 Screening of Studies and Instruments
The methodology and reporting of the systematic review were generally set up to be consistent with the Guideline for Conducting Systematic Literature Reviews in Economic Evaluation [24], the methods proposed by the Cochrane Collaboration [26], and the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement [27]. The PRISMA checklist is presented in electronic supplementary Appendix Table A2. No review protocol was registered. Titles and abstracts (level 1) were screened by SM. In addition, AŁ conducted a second independent screening of the titles and abstracts for 10% of all studies. Differences in inclusion (level 1) were found in approximately 20% of the studies, but did not affect the final inclusion of unique instruments, for which close to no disagreement was observed. This approach also seems justifiable given that all studies that did not explicitly contradict the inclusion criteria on level 1 were assessed further, independently of actually mentioning a (specific or non-specific) instrument in the title or abstract. Full-text screening of the articles (level 2), instrument extraction, and the additional search for psychometric evidence was independently conducted by two reviewers (SM, AŁ) and any disagreement resolved by discussion against the inclusion criteria. The instrument extraction sheet, which partly follows the taxonomy [15] designed for the set-up of DIRUM, was discussed with the project team and piloted on 10% of the included instruments.
3 Results
Of 3637 unique studies identified in the systematic review of the literature, 167 full texts were included for assessment of the instruments mentioned, cited or listed in these studies (Fig. 1). Following the screening of the full instruments, a total of 55 instruments were identified, resulting in 26 unique RUM instruments with ICB items in the education and/or criminal justice sectors after deduplication (Table 1); 8 instruments were retrieved based on the DIRUM search, 13 instruments were found based on given references in economic evaluations and cost-of-illness studies, and 5 instruments came up in both searches. Of the 26 instruments, five were included following email correspondence with the study authors as full texts were not publicly available. Twenty-one of the 26 instruments were found to be used only once, i.e. in one study, for one country.
PRISMA flowchart of electronic database search and DIRUM search. PRISMA Preferred Reporting Items for Systematic Reviews and Meta-Analyses, ICB intersectoral costs and benefits, DIRUM Database of Instruments for Resource Use Measurement. Asterisk refers to the instruments included from the DIRUM search
A descriptive overview of the characteristics of the included instruments is presented in Table 1. With the exception of the Dutch [28] RUM instrument, all instruments were published in English, mostly referring to the UK healthcare system [29,30,31,32,33,34,35,36,37,38,39,40,41], the US [43,44,45,46,47], Australia [48], Canada [49, 50], Cambodia [51] and Kenya [52]. Three instruments [42, 53, 54] were designed for multicountry studies, and were published in English and several other European languages. Mostly, instruments were used for research related to mental health problems or developmental disorders [28, 29, 31, 38, 39, 42, 43, 49], alcohol-related disorders or substance abuse [30, 36, 44, 50, 54], chronic conditions [32, 33, 37, 45] or other specific diseases [34, 35, 41, 46,47,48, 51,52,53]. In total, 14 instruments refer to children and/or adolescents [28, 31,32,33,34,35, 38, 39, 43, 46, 49, 51, 55, 56], three instruments refer to adolescents and adults [45, 47, 53], and nine instruments refer to the adult population only [29, 30, 36, 42, 44, 48, 50, 52, 54]. Although many instruments were designed for, or used in, multiple administration modes and forms of recording, 9 [29, 43,44,45,46, 48, 50, 51, 54] of the 26 instruments were administered in an interview set-up, including direct contact with a researcher.
3.1 ICB Instrument Content
All but two instruments not only include ICB-specific items but also a variety of healthcare utilization RUM questions. Education-related RUM items are included in 21 instruments, most frequently in instruments designed in the context of various specific disease areas [34, 35, 41, 46,47,48, 51,52,53], developmental disorders [31, 38, 39, 43, 49] and chronic conditions [32, 33, 45, 55]. Criminal justice RUM items are included in 13 instruments, most frequently related to developmental disorders [38, 39, 43, 49], alcohol-related disorders or substance abuse [30, 36, 44, 50, 54] and mental health in general [29, 42]. Seven instruments include items from both the education and criminal justice sectors. Extracted details on the instrument content regarding, for example, item wording are presented in the electronic supplementary Appendix Table A3.
As shown in Fig. 2, ICB items referring to the education sector mostly cover time missed at school due to the specific disease (15 items), followed by extra need for tutoring activities (six items) and classroom assistance (six items). Attendance at a special/boarding school is captured by five items, as is social/school functioning. ICB items referring to the criminal justice system most commonly refer to the use of lawyers or general legal assistance (11 items), police custody or prison detainment (9 items), appearances in criminal or civil courts (6 items), injuries (5 items), police contacts (5 items), or probation/correction services (5 items) [Fig. 3].
Education-related ICB items in the instruments. Instrument references by ICB items: (1) absenteeism from school [28, 32, 34, 35, 39, 41, 43,44,45, 47, 51, 53, 55]; (2) tutoring [31, 33, 41, 43]; (3) classroom assistance [31, 33, 38, 39, 49, 56]; (4) special school/boarding school [31, 33, 38, 39, 43, 49]; (5) school functioning [35, 45, 46]; (6) social functioning [46]; (7) adolescence/school counsellor [38, 39, 50]; (8) special school teacher [33, 43]; (9) special needs statement [31, 38]; (10) special class [43]; (11) school dropout [48, 52]; (12) other educational services [38, 39] [31, 35, 47,48,49, 51]. As more than one item per topic may be contained in an instrument, the number of references do not necessarily add up to the numbers indicated in the figure. ICB intersectoral costs and benefits
Criminal justice-related ICB items in the instruments. Instrument references by ICB item: (1) lawyer/legal assistance [33, 36, 38, 39, 49, 50]; (2) police custody/prison detainment [29, 30, 42, 44, 50, 54]; (3) criminal/civil court appearance [29, 30, 42, 44, 50]; (4) injury [36]; (5) police contact [29, 38, 39, 42, 50]; (6) probation/correction services [38, 39, 43, 50, 54]; (7) aggressiveness/violence [36, 54]; (8) traffic accident [36, 44]; (9) psychiatric assessment in custody [29, 42]; (10) property damage [38, 48, 54]; (11) youth offending team [38, 39]; (12) other services [29, 42] [36, 38, 44, 54]. As more than one item per topic may be contained in an instrument, the number of references do not necessarily add up to the numbers indicated in the figure. ICB intersectoral costs and benefits
3.2 Psychometric Properties of ICB Items
The psychometric properties were assessed for seven of the identified instruments, namely the Child and Adolescent Services Assessment (CASA) interview [43], Client Service Receipt Inventory (CSRI) [29], Client Sociodemographic and Service Receipt Inventory—European Version (CSSRI—EU) [42], cost diary by Bodden et al. [28], Work Productivity and Activity Impairment (WPAI) instrument [47], Health Service Utilization Inventory (HSUI) [50], and European Addiction Severity Index (EuropASI) [54]. In case of one instrument [51], validation was mentioned but no further details were given. In the economic evaluations citing the included instruments, most authors provided details on the development of the instrument, including, for example, the development team or information on which other instrument the relevant instrument is based on. Pilot testing of the instrument was reported for nine instruments.
Psychometric assessment of ICB-related items could be confirmed for two of these seven instruments; for the CASA instrument [43], both reliability and validity were determined. Test–retest reliability was found to be high for the most intensively used services, including juvenile justice; however, services used in the child’s natural setting, for example school services, were found to be reported with low reliability [57]. The assessment of concurrent validity (i.e. correlation of service use measured with two instruments) was restricted to mental health service use only, and, again, varied for individual services [58]. Construct validity and reproducibility of the general WPAI Questionnaire was established in earlier research [59]. For the WPAI Questionnaire plus Classroom Impairment Questions: Specific Health Problem (WPAI + CIQ:SHP, version 2.0) specifically, psychometric evidence is available for its allergy-specific (AS) first version (the WPAI + CIQ:AS), testing its discriminative and evaluative validity [60]. Except for time missed from the classroom, allergic rhinitis symptoms were found to be well-correlated for most impairment measures (discriminative validity). The same conclusion applies to the correlation analysis on the change in symptoms and change in time missed from the classroom (evaluative validity).
4 Discussion and Conclusion
Conducting economic evaluations from a broader societal perspective rather than from a narrow healthcare viewpoint is increasingly acknowledged as the gold standard [61], and also manifests in pharmacoeconomic guidelines across Europe [7] and in the US [9]. To be able to consider health-related impacts of interventions in the education or criminal justice sectors in practice, sound instruments measuring relevant resource use are a prerequisite. This review identified a total of 26 unique instruments with a variety of such ICB items. Most frequently, ICB items in the education sector measured resource use due to school absenteeism, tutoring activities, classroom assistance and school or social functioning. Resource use related to the criminal justice sector most commonly captured legal assistance, police custody or prison detainment, criminal or civil court appearance, injuries, police contacts or the use of probation/correction services. Psychometric evidence was found to be examined for seven instruments that, with two exceptions [43, 47], focused only on healthcare RUM items, hence generally leaving out ICB items from their assessment. For the two exceptions, validity (discriminative and evaluative) was assessed for one instrument (WPAI + CIQ:AS [47]), and reliability (test–retest) was assessed for another (CASA [43]).
From a methodological perspective, compared with patient-reported outcome measures, methods of RUM within economic evaluations have been less of a focal point [21]. Specifically, a culture of psychometric validation of RUM instruments for self-reported measurement is lacking in health economics [21, 62]. In a review of UK Health Technology Assessment (HTA) program funds trials, only approximately 30% of the studies were found to report on validation of their RUM data [63]. This review confirms such lack of psychometric evidence, also specifically for ICB items. Indeed, establishing validity, in particular for ICB in the education or criminal justice sectors, might prove even more challenging than for non-ICB items [64]. For example, it might prove difficult, in practice, to get access to alternative data sources (e.g. administrative information) for items such as the number and/or duration of police contacts, psychiatric assessments, received legal assistance, or the number of traffic tickets (electronic supplementary Appendix Table A3) to establish some degree of criterion validity [15]. This is a drawback, especially given that criminal justice resource use is likely a sensitive topic and potentially rather prone to, for example, measurement error [36]. For instance, earlier research suggests that a social desirability bias is more likely to be present in interview situations with a researcher than, for example, self-administered questionnaires [64, 89]. This issue could thus also be relevant for existing ICB items, given that an interview-type administration mode was used for 9 of the 26 identified instruments. Lack of psychometric assessment was also found with regard to piloting of the instruments, although pilot-testing is another critical step in ensuring reliability [63, 65]. For 9 of the 26 instruments included in this review, pilot-testing was reported, which is comparable to the proportion identified in the UK HTA review [63].
A recent systematic literature review by Leggett et al. [12] also analyses RUM instruments. RUM items outside the healthcare sector, including travel expenses, out-of-pocket costs and productivity losses, were concluded to be particularly rare in the 15 identified instruments. Other ICB items were not discussed in the review, which was generally restricted to publicly available, validated instruments designed for adult populations only. Overlap between the instruments identified by Leggett et al. [12] and this research is consequently limited to the CSRI instrument only. The lacking overlap is also likely due to this review’s indirect search approach (Sect. 2.1), contrasting the common method of directly identifying validated instruments through literature search in earlier reviews [12, 16, 66].
4.1 Strengths and Limitations
This review is the first to give a comprehensive overview of instruments including ICB items for measurement of resource use in the education or criminal justice system. It discloses both a heterogeneous variety of existing ICB RUM items in these sectors and reveals a lack of psychometric evaluation of these instruments. The latter finding strongly supports the choice of the adopted indirect search strategy. Relevant new instruments identified in the course of this review will be fed into DIRUM, which will help future researchers identify and retrieve instruments with ICB items in a more efficient manner. Indeed, the fact that the majority (21 of 26) of the included instruments were found to be specifically designed for and used in one study only points out substantial efficiency potential for researchers through sharing instruments. Finally, in most cases, the identified ICB items are part of broader instruments; in this sense, this review also provides an overview of existing international instruments for healthcare RUM, albeit not being the initial aim.
Note though that several limitations apply to the instrument identification strategy. Firstly, instruments that generally contain ICB items, but were used in an economic evaluation from a healthcare perspective only, were potentially overlooked. However, all instruments that were specified in the studies were checked for relevant ICB RUM items, even if inclusion of such elements was not to be expected from the respective economic evaluation or cost-of-illness analysis. At the same time, the high number of studies using non-specified, non-referenced, non-listed instruments detected in this review is striking. For example, of all 167 studies included for instrument assessment, approximately one in five articles (n = 31) reported using, for example, some ‘(adapted) standardized’ instruments or a general ‘economic’ instrument, and failed to provide more details. This lack of methodological transparency in instrument reporting, which was also seen in earlier research [64], hampers not only the potential comparability of these economic evaluations but also impairs the quality assessment of the measurement of the included cost components.
Secondly, this review was restricted to instruments mentioned in published health-related economic evaluations or cost-of-illness studies, and thus health services research in general. Looking at other disciplines such as, for example, education research, economic evaluations seem to be applied rather sparsely [67]. However, additional RUM instruments might be found, for example, in the literature on education program impacts [66]. Future research might thus want to consider looking into instruments developed in other disciplines. Trial registries could be an additional channel to identify more recent ICB measurement instruments.
Thirdly, comparable to the review by Leggett et al. [12], a quality assessment of the psychometric evidence was not carried out. This is a limitation since not only the validation findings themselves are crucial but also the quality of the methodology applied [16]. However, given the currently limited evidence published on instrument quality for ICB items (2 of 26 instruments), at this point such an assessment would be premature. Once more relevant studies have been conducted, the quality of the validation studies could be checked, e.g. by applying the COSMIN (COnsensus-based Standards for the selection of health Measurement INstruments) checklist [68], as was done by recent research [16, 66, 69].
4.2 Research and Policy Implications
When choosing an RUM instrument for use in a trial-based economic evaluation, it is crucial that this instrument covers the domains that are needed to capture the real-world economic consequences of an intervention in the specific disease area [42]. Based on the included instruments, ICB RUM items in the education sector seem particularly relevant not only for age-related developmental disorders affecting children and adolescents but also chronic diseases, including diabetes and chronic pain. In contrast, criminal justice service resource use seems more important in the fields of alcoholism and substance abuse, and developmental disorders, as well as mental health in general. Future economic analyses, particularly in these fields, should thus consider measuring ICBs from the education and criminal justice sectors, respectively.
Against the backdrop of the recently developed ICB classification scheme [2], the majority of ICB items in the education sector are captured by the existing instruments. Most items are found in the CSRI–Children’s Version (CSRI-C) [31], Studying the Scope of Parental Expenditures (SCOPE; based on CSRI) [49] and Self-Harm Intervention, Family Therapy (SHIFT) instruments [38, 39]. Regarding the criminal justice sector, most ICB items are included in the CSRI [29], CSSRI [42], SHIFT [38] and the instrument developed by Sommers et al. [44]. However, compared with the ICB classification scheme, existing instruments lack ICB items regarding, for example, child maltreatment, sexual assaults and crime consequences on victims, which should be taken into account when developing new items.
Besides validating existing ICB items, future research could focus on the development of a harmonized new instrument with a broad variety of relevant ICB elements. Indeed motivated by the fact that RUM instruments lack validation, RUM items from the DIRUM database are currently being reviewed by Thorn et al. [62] with the aim of developing a standardized RUM instrument for the UK. This instrument will focus on the health and social care sector and will exclude ICBs. Therefore, the first step in developing a new ICB instrument would include a literature search to identify the main cost-driving elements from relevant economic evaluations in a specific disease area [21]. This was selectively performed in prior research [11, 70]; however, given that the inclusion of ICBs does not seem to have a long tradition [71], these empirical studies might be missing such cost elements for this very reason [72]. Thus, collating ICBs that were mentioned in studies but not necessarily measured appears more reasonable [2]. The second step in the development of a new instrument [21] includes the identification of existing instruments, which was also one of the main aims of this review. Following this step, focus groups with healthcare professionals, experts in the education/criminal justice sectors and patients to discuss these instruments should be organized to develop a first version of a new instrument. Many instruments identified in this review were specifically developed for use in one study only, which potentially implies considerable duplication of work happening across studies. Therefore, bundling individual efforts could be a more cost-effective strategy overall. Setting-up an international task force to support these activities by, for example, reflecting on and exploring different sources of, for instance, electronic data as a means of validation, and also developing an internationally adaptable, validated instrument, could be the next step. Such an initiative should take into account the state of the art of RUM classifications by detailing the key components of an RUM [64, 71]. Following a structured taxonomy such as the one developed by Ridyard et al. [64], by providing a description of the data source, who completes the instrument, the administration mode, and the methods and medium of recording, clarity and methodological transparency could be established in the development of such a new instrument.
Data Availability Statement
The authors declare that the data supporting the findings of this study are available within the article and its supplementary electronic information files. Full instruments not directly retrieved from the literature but following email communication with the instrument developer may be directly requested from them based on the information provided (Table 1, ‘Reference and instrument identification’). Relevant instruments will be included in DIRUM (http://www.dirum.org).
References
Weatherly H, Drummond M, Claxton K, et al. Methods for assessing the cost-effectiveness of public health interventions: key challenges and recommendations. Health Policy. 2009;93(2–3):85–92.
Drost RM, Paulus AT, Ruwaard D, Evers SM. Inter-sectoral costs and benefits of mental health prevention: towards a new classification scheme. J Ment Health Policy Econ. 2013;16(4):179–86.
Krol M, Brouwer W, Rutten F. Productivity costs in economic evaluations: past, present, future. Pharmacoeconomics. 2013;31(7):537–49.
Krol M, Papenburg J, van Exel J. Does including informal care in economic evaluations matter? A systematic review of inclusion and impact of informal care in cost-effectiveness studies. Pharmacoeconomics. 2015;33(2):123–35.
Byford S, Raftery J. Perspectives in economic evaluation. BMJ. 1998;316(7143):1529–30.
Drost RM, van der Putten IM, Ruwaard D, Evers SM, Paulus AT. Conceptualizations of the societal perspective within economic evaluations: systematic review. Int J Technol Assess Health Care (accepted).
Claxton K, Walker S, Palmer S, Sculpher M. Appropriate perspectives for health care decisions. Working Papers no 054cherp, Centre for Health Economics, University of York; 2010.
Knies S, Severens JL, Ament AJ, Evers SM. The transferability of valuing lost productivity across jurisdictions. Differences between national pharmacoeconomic guidelines. Value Health. 2010;13(5):519–27.
Sanders GD, Neumann PJ, Basu A, et al. Recommendations for conduct, methodological practices, and reporting of cost-effectiveness analyses: second panel on cost-effectiveness in health and medicine. JAMA. 2016;316(10):1093–103.
Gold MR, Siegel JE, Russell LB, Weinstein MC. Cost-effectiveness in health and medicine. New York: Oxford University Press; 1996.
Drost RM, Paulus AT, Ruwaard D, Evers SM. Valuing inter-sectoral costs and benefits of interventions in the healthcare sector: methods for obtaining unit prices. Expert Rev Pharmacoecon Outcomes Res. 2017;17(1):77–84.
Leggett LE, Khadaroo RG, Holroyd-Leduc J, et al. Measuring resource utilization: a systematic review of validated self-reported questionnaires. Medicine. 2016;95(10):e2759.
Evers SM, Hiligsmann M, Adarkwah C. Risk of bias in trial-based economic evaluations: identification of sources and bias-reducing strategies. Psychol Health. 2015;30(1):52–71.
DIRUM, Database of Instruments for Resource Use Measurement. http://www.dirum.org/. Accessed 28 Jul 2016.
Ridyard CH, Dyfrig A, DIRUM Team. Development of a database of instruments for resource-use measurement: purpose, feasibility, and design. Value Health. 2012;15(5):650–5.
Noben CY, Evers SM, Nijhuis FJ, de Rijk AE. Quality appraisal of generic self-reported instruments measuring health-related productivity changes: a systematic review. BMC Public Health. 2014;14(1):1.
Tang K. Estimating productivity costs in health economic evaluations: a review of instruments and psychometric evidence. Pharmacoeconomics. 2015;33(1):31–48.
Tanco K, Park JC, Cerana A, et al. A systematic review of instruments assessing dimensions of distress among caregivers of adult and pediatric cancer patients. Palliat Supp Care. 2017;15(1):110–24.
Hoefman RJ, van Exel J, Brouwer W. How to include informal care in economic evaluations. Pharmacoeconomics. 2013;31(12):1105–19.
van Durme T, Macq J, Jeanmart C, Gobert M. Tools for measuring the impact of informal caregiving of the elderly: a literature review. Int J Nurs Stud. 2012;49(4):490–504.
Thorn JC, Coast J, Cohen D, et al. Resource-use measurement based on patient recall: issues and challenges for economic evaluation. Appl Health Econ Health Policy. 2013;11(3):155–61.
Glanville J, Paisley S. Identifying economic evaluations for health technology assessment. Int J Technol Assess Health Care. 2010;26(4):436–40.
Alton V, Eckerlund I, Norlund A. Health economic evaluations: how to find them. Int J Technol Assess Health Care. 2006;22(4):512–7.
Akers J, Aguiar-Ibáñez R, Baba-Akbari Sari A. CRD’s guidance for undertaking reviews in health care. York: Centre for Reviews and Dissemination (CRD); 2009.
Streiner DL, Norman GR, Cairney J. Health measurement scales: a practical guide to their development and use. Oxford: Oxford University Press; 2014.
Higgins JP, Green S. Cochrane Handbook for Systematic Reviews of Interventions. Version 5.1.0 2011. http://handbook.cochrane.org/. Accessed 21 June 2016.
Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Ann Intern Med. 2009;151(4):264–9.
Bodden DH, Dirksen CD, Bogels SM. Societal burden of clinically anxious youth referred for treatment: a cost-of-illness study. J Abnorm Child Psychol. 2008;36(4):487–97.
Beecham J, Knapp M. Costing psychiatric interventions. In: Thornicroft G, editor. Measuring mental health needs. London: Gaskell; 2001. p. 200–24.
Watson JM, Crosby H, Dale VM, et al. AESOPS: a randomised controlled trial of the clinical effectiveness and cost-effectiveness of opportunistic screening and stepped care interventions for older hazardous alcohol users in primary care. Health Technol Assess. 2013;17(25):1–158.
Client Service Receipt Inventory Children’s Version. http://www.dirum.org/instruments/details/45. Accessed 1 Aug 2017.
CSRI-CD, Client Service Receipt Inventory for Children with Diabetes. http://www.dirum.org/instruments/details/53#. Accessed 1 Aug 2017.
CSRI-Pain, Client Service Receipt Inventory for Adolescents with Chronic Pain—Parent report. http://www.pssru.ac.uk/blogs/csri/chronic-pain-csri/. Accessed 1 Aug 2017.
Powell CV, Kolamunnage-Dona R, Lowe J, et al. MAGNEsium Trial In Children (MAGNETIC): a randomised, placebo-controlled trial and economic evaluation of nebulised magnesium sulphate in acute severe asthma in children. Health Technol Assess. 2013;17(45):1–216.
Lock C, Wilson J, Steen N, et al. North of England and Scotland Study of Tonsillectomy and Adeno-tonsillectomy in Children (NESSTAC): a pragmatic randomised controlled trial with a parallel non-randomised preference study. Health Technol Assess. 2010;14(13):1–164, iii–iv.
Parrott S, Godfrey C, Heather N, Clark J, Ryan T. Cost and outcome analysis of two alcohol detoxification services. Alcohol Alcohol. 2006;41(1):84–91.
DIRUM: SCIPI RUM: instrument qualities 2016. http://www.dirum.org/instruments/details/79#. Accessed 1 Aug 2017.
SHIFT, Self-Harm Intervention, Family Therapy (SHIFT), Parent or Carers Questionnaire Booklet. http://www.dirum.org/assets/downloads/634426220067245338-SHIFT%20Parent%20or%20Carers%20Questionnaire%20Booklet.pdf. Accessed 1 Aug 2017.
SHIFT, Self-Harm Intervention, Family Therapy (SHIFT), Young Person Questionnaire Booklet. http://www.dirum.org/assets/downloads/634426213718379721-SHIFT%20Young%20Person%20Questionnaire%20Booklet.pdf. Accessed 1 Aug 2017.
Barrett B, Byford S. Collecting service use data for economic evaluation in DSPD populations: development of the Secure Facilities Service Use Schedule. Br J Psychiatry Suppl. 2007;49:s75–8.
SYCAMORE CSRI Questionnaire. http://www.dirum.org/assets/downloads/635028335262813537-SYCAMORE_CSRI_questionnaire.pdf. Accessed 1 Aug 2017.
Chisholm D, Knapp MR, Knudsen HC, et al. Client Socio-Demographic and Service Receipt Inventory-European Version: development of an instrument for international research. EPSILON Study 5. European Psychiatric Services: Inputs Linked to Outcome Domains and Needs. Br J Psychiatry Suppl. 2000;39:s28–33.
Burns BJ, Angold A, Magruder-Habib K, et al. The child and adolescent services assessment (CASA) Child Interview Version 5.0. http://devepi.duhs.duke.edu/casa.html. Accessed 1 Aug 2017.
Sommers BD, Fargo JD, Lyons MS, Shope JT, Sommers MS. Societal costs of risky driving: an economic analysis of high-risk patients visiting an urban emergency department. Traffic Inj Prev. 2011;12(2):149–58.
Wetterneck CT, Woods DW, Norberg MM, Begotka AM. The social and economic impact of trichotillomania: results from two nonreferred samples. Behav Intervent. 2006;21(2):97–109.
Zhou ZY, Koerper MA, Johnson KA, et al. Burden of illness: direct and indirect costs among persons with hemophilia A in the United States. J Med Econ. 2015;18(6):457–65.
WPAI + CIQ:AS. Work Productivity and Activity Impairment Questionnaire plus Classroom Impairment Questions: Allergy Specific, Version 2. http://www.reillyassociates.net/WPAI_AS.html. Accessed 1 Aug 2017.
Connelly LB, Webber J, Rowell D. Cost of Trauma Instrument (COTI), 2005. http://www.conrod.org.au/cms/resources-and-tools/cost-of-trauma-instrument-coti. Accessed 1 Aug 2017.
Genereaux D, van Karnebeek CD, Birch PH. Costs of caring for children with an intellectual developmental disorder. Disabil Health J. 2015;8(4):646–51.
Browne G, Gafni A, Roberts J, Goldsmith A, Jamieson E. Approach to the measurement of costs (expenditures) when evaluating health and social programmes. Hamilton: McMaster University; 1995.
Huy R, Wichmann O, Beatty M, et al. Cost of dengue and other febrile illnesses to households in rural Cambodia: a prospective community-based case-control study. BMC Public Health. 2009;9:155.
KNCV Tuberculosis Foundation. The Tool to Estimate Patients’ Costs, 2008. http://www.biomedcentral.com/content/supplementary/1471-2458-11-43-S1.PDF. Accessed 1 Aug 2017.
Aygoren-Pursun E, Bygum A, Beusterien K, et al. Socioeconomic burden of hereditary angioedema: results from the hereditary angioedema burden of illness study in Europe. Orphanet J Rare Dis. 2014;9:99.
Blacken PH, Pozzi G, Tempesta E, et al. European Addiction Severity Index (EuropASI) Cost A6: A guide to training and administering EuropASI interviews 1994. http://www.emcdda.europa.eu/html.cfm/index3647EN.html. Accessed 1 Aug 2017.
SCIPI RUM, SubCutaneous Insulin: Pumps or Injections (SCIPI) RUM, 2011. http://www.dirum.org/assets/downloads/634968699228637202-SCIPI_Questionnaire.pdf. Accessed 1 Aug 2017.
Diary SC. Syncamore Treatment Diary. http://www.dirum.org/assets/downloads/635750685168619541-Sycamore_Treatment_Diary.pdf. Accessed 1 Aug 2017.
Farmer EM, Angold A, Burns BJ, Costello EJ. Reliability of self-reported service use: test-retest consistency of children’s responses to the Child and Adolescent Services Assessment (CASA). J Child Fam Stud. 1994;3(3):307–25.
Ascher BH, Farmer EM, Burns BJ, Angold A. The child and adolescent services assessment (CASA) description and psychometrics. J Emot Behav Disord. 1996;4(1):12–20.
Reilly MC, Zbrozek AS, Dukes EM. The validity and reproducibility of a work productivity and activity impairment instrument. Pharmacoeconomics. 1993;4(5):353–65.
Reilly MC, Tanner A, Meltzer EO. Work, classroom and activity impairment instruments: validation studies in allergic rhinitis. Clin Drug Investig. 1996;11(5):278–88.
Ten Jönsson B. arguments for a societal perspective in the economic evaluation of medical innovations. Eur J Health Econ. 2009;10(4):357–9.
Thorn JC, Ridyard CH, Riley R, et al. Identification of items for a standardised resource-use measure: review of current instruments. Value Health. 2015;18(7):A688.
Ridyard CH, Hughes DA. Methods for the collection of resource use data within clinical trials: a systematic review of studies funded by the UK Health Technology Assessment program. Value Health. 2010;13(8):867–72.
Ridyard CH, Hughes DA, DIRUM Team. Taxonomy for methods of resource use measurement. Health Econ. 2015;24(3):372–8.
Johnston K, Buxton MJ, Jones DR, Fitzpatrick R. Assessing the costs of healthcare technologies in clinical trials. Health Technol Assess. 1999;3(6):1–76.
Schellingerhout JM, Verhagen AP, Heymans MW, et al. Measurement properties of disease-specific questionnaires in patients with neck pain: a systematic review. Qual Life Res. 2012;21(4):659–70.
Ladd HF, Edward EB. Handbook of research in education finance and policy. Abingdon: Routledge; 2012.
Terwee CB, Mokkink LB, Knol DL, et al. Rating the methodological quality in systematic reviews of studies on measurement properties: a scoring system for the COSMIN checklist. Qual Life Res. 2012;21(4):651–7.
van Ballegooijen W, Riper H, Cuijpers P, van Oppen P, Smit JH. Validation of online psychometric instruments for common mental health disorders: a systematic review. BMC Psychiatry. 2016;16:45.
Goodrich K, Kaambwa B, Al-Janabi H. The inclusion of informal care in applied economic evaluation: a review. Value Health. 2012;15(6):975–81.
Ridyard CH, Hughes DA. Review of resource-use measures in UK economic evaluations. In: Curtis L, Burns A, editors. Unit Costs of Health and Social Care 2015. Canterbury: Personal Social Services Research Unit, University of Kent; 2015. p. 22–30.
Mayer S, Kiss N, Laszewska A, Simon J. Health economic costing methods and reporting in Austria. Value Health. 2016;19(7):A363.
Patel A, Rendu A, Moran P, et al. A comparison of two methods of collecting economic data in primary care. Fam Pract. 2005;22(3):323–7.
Byford S, Leese M, Knapp M, et al. Comparison of alternative methods of collection of service use data for the economic evaluation of health care interventions. Health Econ. 2007;16(5):531–6.
Mirandola M, Bisoffi G, Bonizzato P, Amaddeo F. Collecting psychiatric resources utilisation data to calculate costs of care: a comparison between a service receipt interview and a case register. Soc Psychiatry Psychiatr Epidemiol. 1999;34(10):541–7.
Heinrich S, Deister A, Birker T, et al. Accuracy of self-reports of mental health care utilization and calculated costs compared to hospital records. Psychiatry Res. 2011;185(1–2):261–8.
Kilian R, Roick C, Bernert S, et al. Instruments for the economical evaluation of psychiatric service systems: methodological foundations of the european standardisation and the German adaptation [in German]. Psychiatr Prax. 2001;28(Suppl 2):74–8.
Browne GB, Arpin K, Corey P, Fitch M, Gafni A. Individual correlates of health service utilization and the cost of poor adjustment to chronic illness. Med Care. 1990;28(1):43–58.
Hendriks VM, Kaplan CD, van Limbeek J, Geerlings P. The Addiction Severity Index: reliability and validity in a Dutch addict population. J Subst Abuse Treat. 1989;6(2):133–41.
Dijkgraaf MG, van der Zanden BP, de Borgie CA, et al. Cost utility analysis of co-prescribed heroin compared with methadone maintenance treatment in heroin addicts in two randomised trials. BMJ. 2005;330(7503):1297–300.
Knapp M, Scott S, Davies J. The cost of antisocial behaviour in younger children. Clin Child Psychol Psychiatry. 1999;4(4):457–73.
Sleed M, Eccleston C, Beecham J, Knapp M, Jordan A. The economic impact of chronic pain in adolescence: methodological considerations and a preliminary cost-of-illness study. Pain. 2005;119(1–3):183–90.
Mauch V, Woods N, Kirubi B, et al. Assessing access barriers to tuberculosis care with the tool to Estimate Patients’ Costs: pilot results from two districts in Kenya. BMC Public Health. 2011;11(1):1.
UKATT Research Team. Cost effectiveness of treatment for alcohol problems: findings of the randomised UK alcohol treatment trial (UKATT). BMJ. 2005;331(7516):544.
STEPWICE Research Group. The effectiveness and cost-effectiveness of screening and stepped care interventions for alcohol use disorders in the primary care setting. Cardiff: Welsh Office of Research & Development; 2003.
Coulton S, Watson J, Bland M, et al. The effectiveness and cost-effectiveness of opportunistic screening and stepped care interventions for older hazardous alcohol users in primary care (AESOPS)—a randomised control trial protocol. BMC Health Serv Res. 2008;8:129.
Stade B, Ali A, Bennett D, et al. The burden of prenatal exposure to alcohol: revised measurement of cost. Can J Clin Pharmacol. 2009;16(1):e91–102.
Ramanan AV, Dick AD, Benton D, et al. A randomised controlled trial of the clinical effectiveness, SafetY and cost effectiveness of adalimumab in combination with MethOtRExate for the treatment of juvenile idiopathic arthritis associated uveitis (SYCAMORE). Trials. 2014;9:14.
Bowling A. Mode of questionnaire administration can have serious effects on data quality. J Public Health. 2005;27(3):281–91.
Acknowledgements
Open access funding provided by Medical University of Vienna. Many thanks to Gregor Franssen (Information Scientist at Maastricht University) for expert advice on setting up the search strategy, and Marianne Tilly for her support with study full-text retrieval. The authors also gratefully acknowledge the help of several members of the Department of Health Services Research at Maastricht University with language translations. Feedback from Joanna Thorn on an earlier version of the manuscript was greatly appreciated. We also gratefully acknowledge the valuable input from participants at the 2017 lowlands Health Economists’ Study Group (lolaHESG) conference in Rotterdam, The Netherlands. This manuscript was drafted during Susanne Mayer’s research stay at Maastricht University.
Author information
Authors and Affiliations
Contributions
The initial idea for this study was first brought up by SE and AP. SM developed the study concept, to which input was provided by AP, JS, RD, DR and SE. All co-authors were involved in the discussion of all steps of the review. SM conducted the literature search, with AŁ as the second screener. SM analyzed the data and wrote the manuscript, and all authors read and approved the final version of this article.
Corresponding author
Ethics declarations
Conflicts of interest
Susanne Mayer, Aggie T.G. Paulus, Agata Łaszewska, Judit Simon, Ruben M.W.A. Drost, Dirk Ruwaard, Silvia M.A.A. Evers declare that they have no conflicts of interest relevant to the content of this article.
Funding
No funding from external sources was received for this work.
Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Open Access This article is distributed under the terms of the Creative Commons Attribution-NonCommercial 4.0 International License (http://creativecommons.org/licenses/by-nc/4.0/), which permits any noncommercial use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
About this article
Cite this article
Mayer, S., Paulus, A.T.G., Łaszewska, A. et al. Health-Related Resource-Use Measurement Instruments for Intersectoral Costs and Benefits in the Education and Criminal Justice Sectors. PharmacoEconomics 35, 895–908 (2017). https://doi.org/10.1007/s40273-017-0522-4
Published:
Issue Date:
DOI: https://doi.org/10.1007/s40273-017-0522-4