Abstract
Background
Measuring objective resource-use quantities is important for generating valid cost estimates in economic evaluations. In the absence of acknowledged guidelines, measurement methods are often chosen based on practicality rather than methodological evidence. Furthermore, few resource-use measurement (RUM) instruments focus on the measurement of resource use in multiple societal sectors and their development process is rarely described. Thorn and colleagues proposed a stepwise approach to the development of RUM instruments, which has been used for developing cost questionnaires for specific trials. However, it remains unclear how this approach can be translated into practice and whether it is applicable to the development of generic self-reported RUM instruments and instruments measuring resource use in multiple sectors. This study provides a detailed description of the practical application of this stepwise approach to the development of a multi-sectoral RUM instrument developed within the ProgrammE in Costing, resource use measurement and outcome valuation for Use in multi-sectoral National and International health economic evaluAtions (PECUNIA) project.
Methods
For the development of the PECUNIA RUM, the methodological approach was based on best practice guidelines. The process included six steps, including the definition of the instrument attributes, identification of cost-driving elements in each sector, review of methodological literature and development of a harmonized cross-sectorial approach, development of questionnaire modules and their subsequent harmonization.
Results
The selected development approach was, overall, applicable to the development of the PECUNIA RUM. However, due to the complexity of the development of a multi-sectoral RUM instrument, additional steps such as establishing a uniform methodological basis, harmonization of questionnaire modules and involvement of a broader range of stakeholders (healthcare professionals, sector-specific experts, health economists) were needed.
Conclusion
This is the first study that transparently describes the development process of a generic multi-sectoral RUM instrument in health economics and provides insights into the methodological aspects and overall validity of its development process.
Similar content being viewed by others
Explore related subjects
Find the latest articles, discoveries, and news in related topics.Avoid common mistakes on your manuscript.
This study describes the development process of a multi-sectoral generic RUM instrument. The development process was structured into six steps, including the definition of the instrument attributes, identification of cost-driving elements, review of methodological literature, development of a harmonized cross-sectorial approach, and development and subsequent harmonization of questionnaire modules. |
The PECUNIA RUM is intended for the measurement of resource use in several societal sectors, including health and social care, education, (criminal) justice, employment and productivity, patient, family, and informal care. Together with compatible reference unit cost calculation tools (PECUNIA RUC Templates), the PECUNIA RUM can contribute to producing comparable cost data for informing optimal decision making. |
1 Background
Resource-use measurement (RUM) is a challenging and time-consuming but essential step in economic evaluations of health care interventions [1, 2]. Measuring valid and reliable quantities of resources utilized is of major importance for generating valid cost estimates [1, 3]. A wide variety of measurement methods exist, including the use of self-reported data by patients collected via questionnaires or cost diaries, the use of administrative data, and the use of expert panels [4, 5]. However, methodological guidance on developing an appropriate measurement method is limited. There remains a striking difference between the small amount of research on methods for the appropriate measurement of resource use compared with the large amount of evidence on outcome measurement within economic evaluations [1, 6, 7]. Due to the absence of a gold standard and acknowledged guidelines, the measurement method is often selected based on practicality rather than methodological evidence [1]. Although administrative data sources can provide accurate resource-use estimates, they often fail to capture potentially relevant societal costs (e.g. out-of-pocket expenses or productivity losses) [1]. Furthermore, linking multiple databases containing patient-level data can be challenging due to privacy regulations. Therefore, the use of self-reported data is currently the most feasible method of generating valid resource-use estimates for the measurement of broader resource use [1].
A number of self-reported RUM instruments are available, many of them accessible via the Database of Instruments for Resource-Use Measurement (DIRUM) [8]. DIRUM is a repository of RUM instruments based on patient recall for use in economic evaluations. As of March 2022, it contained 96 RUM instruments available in the form of diaries, logs or questionnaires. Of these instruments, the majority are designed for measuring disease-specific resource use, while only five instruments are meant for generic use [9,10,11,12,13]. In addition, most instruments focus on resource use in a limited number of sectors (e.g. health care and social care) [14,15,16,17]; few also include questions about broader resource use (e.g. informal care, education, and [criminal] justice) [9, 12, 18, 19]. Research shows that for some disease areas, resource use in sectors outside the health care sector can constitute a large proportion of the total costs and have a significant impact on the study results [20,21,22]. The growing interest in adopting a societal perspective in economic evaluations calls for appropriate measurement tools that enable measurement of multi-sectoral resource use [23]. There is little existing research on the feasibility, validity and reliability of RUM instruments [3, 11, 18, 19, 24], and even fewer studies report the process of instrument development [25,26,27]. This is particularly relevant because the decisions taken during the development process of an instrument, such as the framing and order of the questions, can affect the structure of the instrument and responses to its questions [28]. Furthermore, in several health-related disciplines (e.g. psychology, health promotion), it is common to report how new instruments were developed [29, 30]. However, this is not yet customary in health economics, in particular in relation to RUM instruments [27, 31, 32], even though transparent reporting of how RUM instruments are developed is crucial both for the developers and the users. For the former, the need for transparent reporting motivates the adoption of evidence-based methods when developing the instrument; for the latter, insights into how the instrument is developed allow for making judgements about whether it is appropriate for the given study. In addition, transparent reporting of the RUM instrument development process can support further research into the development of patient-reported measures as well as the use of mixed-methods and qualitative methods in health economics.
Thorn and colleagues [2] proposed a stepwise approach for the development and testing of RUM instruments in trial-based economic evaluations, based on discussions among a large group of health economists with expertise in RUM. The authors outlined an “ideal way of approaching instrument development”, which entails several stages, including planning, development, piloting and deployment of the instrument. Several studies reported using this approach for developing a cost questionnaire for specific trials [33,34,35] without, however, describing the process of applying this approach in detail. Therefore, it remains unclear how this approach can be translated into practice and to what extent it is applicable to the development of (1) generic self-reported RUM instruments, and (2) instruments measuring broader resource use.
This paper describes the practical application of the approach proposed by Thorn et al. [2] to the development process of the RUM instrument in the ProgrammE in Costing, resource-use measurement and outcome valuation for Use in multi-sectoral National and International health economic evaluAtions (PECUNIA) project. The PECUNIA RUM aims to address the above-mentioned research gaps by capturing generic resource use in several societal sectors, including (1) health care; (2) social care; (3) education; (4) (criminal) justice; (5) employment and productivity; and costs borne by (6) patients or (7) families, including informal care. This instrument will support the measurement of broader resource use and the adoption of a societal perspective in future economic evaluations. Furthermore, to the best of our knowledge, this is the first article that describes the development process of a generic multi-sectoral RUM instrument. This can be helpful for the researchers who would consider using the PECUNIA RUM in the future by providing insights into the feasibility, reliability and validity of the instrument.
2 Context: PECUNIA Project
The PECUNIA project is a network of 10 partners in six countries (Austria, Germany, Hungary, Spain, The Netherlands, and the UK) that aims to establish standardized costing and outcome assessment measures for conducting economic evaluations in Europe [36]. To reduce unnecessary variations in the resource-use and unit cost input data of economic evaluations [37], the PECUNIA project (2018–2021) aimed to develop a collection of internationally standardized, harmonized, generic and validated tools, including a self-reported, multi-sectoral RUM instrument. It was developed in conjunction with a compatible unit cost calculation tool, the PECUNIA Reference Unit Cost (RUC) Templates [38], intended for the generation of standardized and harmonized RUCs for economic evaluations [39], and for inclusion in a publicly accessible multi-sectoral, multi-country PECUNIA RUC Compendium [40]. Furthermore, to address the variations in the input cost data and to facilitate cross-country transferability of the results of economic evaluations, the PECUNIA RUM is intended to be applicable in multiple European countries. More information about the PECUNIA project structure can be found in electronic Supplementary File 1 and on the PECUNIA project website (https://www.pecunia-project.eu/).
3 Methods
The development process of the PECUNIA RUM was based on the approach proposed by Thorn et al. [2]. The approach covers all stages of instrument development and testing starting from the identification of relevant cost drivers to the piloting and deployment of the instrument. It also suggests optimal time points for considering important instrument attributes (e.g. recall period, length of questionnaire) as well as consultations with various stakeholders (healthcare professionals, patients, healthy volunteers) at multiple development and testing stages. The PECUNIA Group translated this theoretical method into a structured work plan in scientific discussions over 6 months. The development and testing process of the PECUNIA RUM was structured into nine steps with two milestones, outlined in Fig. 1. This current paper describes the application of this approach to the development of the instrument, i.e. the first six steps up until the instrument was ready for testing.
Methodological steps of PECUNIA resource-use measurement instrument development. The green boxes illustrate methodological steps undertaken to develop the instrument; the blue box indicates the milestone of the instrument development; and the grey boxes indicate further steps of instrument validation that are not covered in this article (own illustration)
-
Step 1.
Definition of Instrument Attributes
In step one, the main attributes of the PECUNIA RUM were conceptualized by the PECUNIA Consortium in line with the general objectives of the PECUNIA project over the course of several in-person and online meetings when conceptualizing the PECUNIA project proposal. General attributes of RUM instruments that ought to be considered were outlined by Thorn et al. and included perspective, setting of care, disease, population, and nature of intervention [2].
-
Step 2.
Identification, Definition and Classification of Main Cost-Driving Elements
In step two, cost-driving elements were identified and defined. First, an independent search of peer-reviewed and grey health economics literature was conducted to identify main cost-driving elements in health care, social care, education, (criminal) justice, patient, family, and informal care sectors. The cost drivers were extracted and compiled in a list. Second, the list of cost drivers was reviewed by sector-specific experts selected by purposive sampling via a survey (e.g. health care experts reviewed the list of main cost drivers in the health care sectors). Four literature reviews and subsequent expert surveys were independently conducted in parallel for (1) health and social care; (2) education; (3) criminal justice; and (4) patient, family, and informal care sectors by corresponding work package leads as described in electronic Supplementary File 1. More details on the methodology of each review and expert survey are available elsewhere [41,42,43].
The draft list of cost drivers needed to be defined and organized in conceptually harmonized clusters of units of analysis, which was done according to the PECUNIA Care Atom, a conceptual framework for semantic standardization [44]. Furthermore, the identified services were classified according to the Description and Evaluation of Services and DirectoriEs (DESDE) as modified for the PECUNIA project [45]. DESDE is a standard taxonomy for description, mapping and comparison of services that was developed over the past 20 years with the aim of improving the comparability of services across settings based on the content of the service rather than semantic equivalence [45]. Additional characteristics of services such as related interventions and professionals were classified according to the International Classification of Health Interventions (ICHI) and the multilingual classification of European Skills, Competences, Qualifications and Occupations (ESCO) as part of the overall PECUNIA Coding System [46].
For the employment and productivity sector, a different approach was implemented. A systematic literature review of measurement instruments for measuring productivity losses was conducted in order to identify whether any of the existing instruments were suitable for economic evaluations conducted from a societal perspective. The results of this review indicated that the Institute of Medical Technology Assessment (iMTA) Productivity Cost Questionnaire (iPCQ) was the optimal instrument for capturing all domains of productivity relevant for health economic evaluations. Therefore, this instrument was incorporated into the PECUNIA RUM as one of the modules, for the measurement of resource use in the employment and productivity sector. More details about the review are available elsewhere [47].
-
Step 3.
Scoping Review to Identify Methodological Recommendations
In step 3, methodological literature pertaining to RUM was reviewed to identify methodological recommendations for measuring resource use in economic evaluations. Relevant literature was identified via a systematic search of six electronic databases (EconLIT, EMBASE [Ovid], Education Resources Information Centre [ERIC], MEDLINE [PubMed], PsycINFO, and the Social Science Citation Index [SSCI; Web of Science]), hand search of literature available in DIRUM, and consultation with health economics experts with experience in RUM. The results of the scoping review provided an overview of methodological issues in RUM that need to be considered when selecting a measurement method. These included such issues as what type of resource use to measure, how to frame questions and how to deal with missing resource-use data. While the literature was not able to provide clear-cut recommendations, the overview of challenges of measuring resource use in economic evaluations guided the methodological decisions made in step 4. More details about the scoping review are available elsewhere [48].
-
Step 4.
Development of a Harmonized Methodological Approach
Given the multi-sectoral nature of the PECUNIA RUM and the fact that it was developed by multiple working groups that were responsible for particular societal sectors (as described in electronic Supplementary File 1), the initial instrument was comprised of standalone modules that needed to be harmonized to make sure that the instrument is cohesive. Therefore, in step 4, a harmonized approach towards the development of a RUM instrument from its separate modules was developed. To develop text for each module, existing RUM instruments were used for illustrative purposes for each of the cost drivers that had been identified and classified in step 2. If no suitable questions were available, new questions were to be written from scratch. Furthermore, consistent use of units of analysis depending on the setting of care was to be implemented throughout the instrument. Outpatient services corresponded to the measurement unit ‘per contact’, day care services corresponded to the measurement unit ‘per day’ and residential care services corresponded to the measurement unit ‘per night’. Decisions on the methodological aspects of the instrument were guided by the framework developed in the scoping review (step 3); all aspects of the framework were considered. The questions were to be developed to be clear and easily understandable by lay audience avoiding the use of double negatives, asking about more than one type of resource use per question and ensuring mutual exclusivity of the questions. Consistent recall period was to be implemented throughout the instrument. A Microsoft Excel template was developed to facilitate a structured approach to the development of the PECUNIA RUM modules for the module leads. The illustration of the template is available in electronic Supplementary File 2. The methodological approach was discussed with health economists at the PECUNIA satellite workshop at the conference of the International Health Economics Association (iHEA) in Basel (Switzerland) in 2019.
-
Step 5.
Development of Questionnaire Modules
In step 5, the harmonized approach (step 4) was employed to develop RUM instrument modules. The process described in step 4 was applied to the development of all PECUNIA RUM modules with the exception of the module on employment and productivity, which was based on an existing instrument, iPCQ. iPCQ was shown to be an optimal instrument for measuring productivity losses in economic evaluations conducted from a societal perspective [47] and it has been extensively used to measure productivity losses (i.e. absenteeism and presenteeism) of paid and unpaid work. The development process and the validation of the iPCQ have been described elsewhere [27, 49].
Also in step 5, the DESDE codes developed in step 2 were used to link the services included in the PECUNIA RUM with the same services included in other PECUNIA costing tools. This step was instrumental in ensuring that the tools would be complementary and allow for harmonized collection of resource-use and cost data for economic evaluations. The PECUNIA costing concept based on the PECUNIA Care Atom linked the PECUNIA RUM instrument and the other PECUNIA costing tools, including the PECUNIA RUC Templates [38] and the PECUNIA RUC Compendium [40], to ensure that sufficient level of detail of resource-use data is collected for costing purposes. Some questions were expanded to accommodate the collection of detailed resource-use data, which is necessary for accurate costing.
All working groups were asked to fill in the Microsoft Excel template (electronic Supplementary File 2) with the information relevant to the PECUNIA RUM module they were in charge of developing. The templates were collated by researchers from Maastricht University, University of Bristol, and Medical University of Vienna, who were leading the Horizontal Axis 3 ‘Measurement’ (as described in electronic Supplementary File 1).
-
Step 6.
Harmonization of Questionnaire Modules
Because the PECUNIA RUM modules were developed by different working groups or were based on an existing instrument (i.e. iPCQ), as described in electronic Supplementary File 1, the format, phrasing, order of the questions and answer options needed to be harmonized. This was carried out iteratively in multiple teleconferences and rounds of revision internally within the PECUNIA Consortium between October 2019 and April 2021. Every meeting was chaired and consensus was sought on each point. All changes were documented using track changes mode and circulated across all working groups involved in the instrument development process before being incorporated. The setup of each questionnaire module was standardized.
The first complete draft of the PECUNIA RUM was reviewed by an international group of 14 health economics experts in November 2019. Convenience sampling was used to recruit the experts with the aim of having a multi-country representation. During the meeting, the experts were presented with a copy of the draft RUM instrument and were asked to complete it reflecting on their experience. Their feedback was recorded and used for the development of the subsequent version of the instrument.
Subsequently, the text of the draft PECUNIA RUM underwent a formal wording review in February 2020, which was outsourced to linguistics specialists and focused on simplifying the language and structure of the questionnaire. This was followed by professional English-language editing in June 2020. Both of these steps ensured that the instrument was respondent-friendly and understandable for lay persons.
4 Results
4.1 Development Process of the PECUNIA RUM
Table 1 provides an overview of the development steps of the PECUNIA RUM and the main results associated with each step.
4.2 PECUNIA RUM
The PECUNIA RUM was designed to contain eight modules corresponding to various types of resource use (Table 2). Each module begins with a short description of its content, including the types of resource use measured and the applicable recall period. The suggested recall period was overall set at 3 months. However, since the evidence on the optimal recall period, in particular regarding employment and productivity [50, 51], remains inconclusive, this methodological choice will have to be further scrutinized in the testing phase. Consistent format, phrasing, wording, order of the questions and the answer options were implemented.
It is important to note that the number of questionnaire modules (eight) does not correspond to the number of sectors (seven) included in the PECUNIA RUM. This was done to facilitate the adaptability of the instrument to various contexts. For example, the module on medication use is separate from the module on resource use in the health and social care sectors as it might not be relevant to include it in every study. This allows omitting it entirely without compromising the coherence of the instrument.
Several techniques were employed to facilitate the feasibility and understandability of the questionnaire for the respondents. First, the questions were accompanied by instructions such as:
“Please tick all answers that apply and indicate the number of contacts you had with a given service”.
Second, examples were provided to aid recall such as:
“Specialist Medical Care (e.g. orthopaedist, psychiatrist, gynaecologist)”
Third, an additional explanation was provided to make sure that the respondents understood what type of resource use they needed to recall, for example:
“Have you used legal services (e.g. contact with a lawyer) in the past 3 months? This could be a face-to-face meeting, an online consultation or a phone call with someone working in legal services.”
Furthermore, skip logic was introduced to direct the respondents through the questionnaire (Fig. 2). For most types of resource use, respondents were first asked whether they have used a resource; if yes, follow-up questions were asked; if not, the respondent could jump to the following module. This allows the respondents to only focus on relevant questions, minimizing burden on respondents.
PECUNIA RUM (draft version): example questions from the section on education resource use (section E); the complete version of the instrument can be found via https://www.pecunia-project.eu/tools/rum-instrument
In total, the PECUNIA RUM underwent 23 revisions before the instrument was ready for preliminary validation with potential respondents. The complete version of the PECUNIA RUM is available on the website of the PECUNIA project (https://www.pecunia-project.eu/tools/rum-instrument).
4.3 Expert Feedback
At the PECUNIA satellite workshop in July 2019, the experts indicated that the development of a multi-sectoral RUM instrument that would be applicable to a variety of settings was a difficult process requiring a lot of consensus building. Nevertheless, it was agreed that such an instrument was needed to facilitate the comparability of the results of health economic evaluations.
When the draft instrument was presented to 14 health economists in November 2019, the experts agreed that the availability of a standardized measure was important. They also argued that the full version of the instrument would often not be applicable to a specific setting, as the relevance of cost categories is dependent on the nature of the intervention. The experts suggested that the instrument could be used as a basis that can be further adapted to specific research requirements. Developing a manual with suggestions regarding the selection of relevant cost categories and hence relevant modules of the PECUNIA RUM (e.g. consulting the patients and/or the literature) was recommended. The experts also raised concerns regarding the applicability of the PECUNIA RUM to the European context due to the differences in, for example, health care system structure. The planned validation activities and the link to the compatible method for the unit cost calculation were perceived by the experts as major strengths of the PECUNIA RUM.
4.4 Application of the Process
The method by Thorn et al. [2] was, overall, applicable to the development of the PECUNIA RUM. As proposed in the method, cost drivers and instrument attributes were defined at the start. Stakeholders (sector-specific experts and health economists) were involved at an early stage of the development process and potential respondents (patients and caregivers) will be involved at later stages when the instrument will be tested. Furthermore, the PECUNIA RUM was developed in conjunction with standardized costing tools, which allows for harmonized collection of resource-use and cost data. However, the development process of the PECUNIA RUM deviated from the method of Thorn et al. [2] in several ways. First, the method recommends that healthcare professionals are involved in the development process. However, due to the multi-sectoral nature of the PECUNIA RUM, a broader range of experts were involved in the development and testing, including experts from other sectors (e.g. education, [criminal] justice, productivity) and health economists. Second, while the method of Thorn et al. prescribes reliance on existing RUM instruments when developing a new one [2], the development of the PECUNIA RUM is guided by existing RUM instruments only to a limited extent. This can be attributed to the lack of comprehensive RUM instruments that incorporate cost categories beyond health care. Existing instruments were used as illustrative examples, but no parts of questionnaires or questions were used verbatim, with the exception of the module on employment and productivity, which was fully based on an existing RUM instrument (iPCQ). Furthermore, many existing RUM instruments were developed for use in specific contexts [52], while the PECUNIA RUM was meant to be a generic, internationally applicable instrument. Third, two additional steps in the planning phase of the Thorn et al. method [2] were needed to account for the complexity of developing a multi-sectoral RUM instrument, i.e. the scoping review (step 3) and establishing a uniform methodological basis (step 4), as well as one additional step in the development phase, i.e. harmonization of instrument modules (step 6).
5 Discussion
This article aimed to transparently describe the practical application of the method described by Thorn et al. [2] for the development of the PECUNIA RUM. To our knowledge, this is the first study that outlines the development of a generic multi-sectoral RUM instrument. The methods described by Thorn et al. [2] were, overall, applicable to the development of this generic multi-sectoral RUM instrument. However, because the instrument comprises eight modules developed by four distinct research teams, as well as due to the inherent differences among the sectors included in the PECUNIA RUM, additional steps such as establishing a sound methodological basis (step 4) and harmonization of the instrument modules (step 6) were necessary. On the other hand, a scoping review (step 3) is not likely to be a necessary step for developing new RUM instruments, as the current scoping review’s findings have broader applicability beyond this study. Furthermore, in addition to healthcare professionals, other sector-specific and health economics experts needed to be involved in the development process. This is due to the inclusion of the cost categories beyond health care and the overall complexity of the instrument, for which the involvement of healthcare professionals would not be sufficient to get insights into the feasibility of the instrument.
Compared with other studies that reported using the same method for the development of a RUM instrument [33,34,35], the current study provides more detail and insights into the development process. This could be helpful by providing guidance to researchers undertaking the development of RUM instruments, and also by providing insight into the context in which methodological choices behind the current instrument were made. Several previous studies reported the development process of either a disease-specific RUM instrument [25] or an instrument intended for measuring resource use related to productivity losses [27, 53], which limits the generalizability of the process to the development of generic multi-sectoral RUM instruments. In contrast, the current study provides a structured stepwise approach to instrument development based on available best practices [2] and presents the development process of an instrument that relies significantly less on existing RUM instruments. Furthermore, this study demonstrates that the methodological approach to the development of an RUM instrument is determined by the type of RUM instrument being developed. For example, in the case of the PECUNIA RUM, the involvement of a broader range of experts in the instrument development process was needed due to the multi-sectoral nature of the instrument.
Compared with other generic RUM instruments available in DIRUM [8], the PECUNIA RUM allows for measuring resource use in more sectors (e.g. education, [criminal] justice) and for collecting standardized evidence for informing optimal decision making. Connections to the DESDE-based PECUNIA coding system [54], which facilitates the comparability of services, and to the compatible PECUNIA costing tools [38,39,40] allow for producing comparable resource use and cost estimates across studies. The PECUNIA RUM instrument was developed by a multinational consortium and will be tested in several European countries in addition to formal translatability assessment and concept elaboration to ensure its applicability to various geographical, linguistic, cultural and system settings. In addition, the flexible structure of the instrument makes it possible to adapt the instrument to a specific setting by selecting only relevant modules. This allows researchers to make a trade-off between, on the one hand, having an incomplete instrument that is easier to complete and, on the other hand, a comprehensive instrument that is too lengthy and difficult for the respondents to fill in.
The PECUNIA RUM can be compared with other comprehensive RUM questionnaires covering broader resource use, such as the Client Service Receipt Inventory (CSRI) [55], the Treatment Inventory of Costs in Patients with psychiatric disorders (TiC-P) [11], or the Health Economics Questionnaire (HEQ) [9, 56]. The aim of developing the PECUNIA RUM is not to replace other existing instruments but to offer a broader alternative, which has certain benefits such as a valid, structured and documented development process, inclusion of more resource-use items and cost categories within and beyond the healthcare sector, and a modular structure, which makes the instrument more easily adaptable to various contexts.
5.1 Reflections on the Development Process
Several aspects of the development process need to be mentioned. First, using a different methodological approach to the development of the PECUNIA RUM could have resulted in a different instrument, both in terms of the content as well as the structure. The PECUNIA Group selected the method developed by Thorn and colleagues [2] as, to the best of our knowledge, it is the only available systematic method for the development of a RUM instrument in health economics. This approach was generally applicable given the aim of the PECUNIA project. However, due to the complexity of the instrument, additional steps were needed. Second, as argued by the health economics experts, the applicability of the instrument to various settings (e.g. patient groups, disease areas) is difficult to ensure during the development phase. This would require additional testing of the instrument in different patient groups with respect to disease and national context. Although the PECUNIA RUM will initially be tested among former mental health service users and carers, the cost categories included in the instrument are also relevant for other conditions (e.g. chronic diseases) [52]. Third, instruments used for data collection in research are developed by researchers with high educational level and specialized knowledge, while the instruments are usually completed by patients and caregivers with varying levels of education and (health) literacy. To ensure understandability of the text, a formal wording review and revision of the text by a professional English editor were conducted. Fourth, RUM instruments are commonly used to collect resource-use data in randomized controlled trials alongside many other measures. This means that participants are often asked to fill in multiple questionnaires. Therefore, it is important to determine the appropriate length of the questionnaire by finding a balance between sufficient detail to obtain necessary data and patient burden. Testing the instrument with the potential end users will provide more insights into whether the current length is optimal.
5.2 Implications for Further Steps
To optimize the draft PECUNIA RUM, piloting will be conducted in the form of think-aloud interviews. Mental health service users and informal caregivers of patients with mental illness will be consulted in relation to the completeness, difficulty and feasibility of filling in the instrument. Furthermore, the instrument will undergo linguistic translatability assessment to prepare it for formal translation to multiple languages. The results of the preliminary validation and piloting will be used to develop the final, integrated, multi-sectoral PECUNIA RUM [57]. The PECUNIA RUM is planned to undergo formal psychometric testing and several further versions of the instrument are planned to be developed (e.g. child and proxy versions). More extensive, i.e. midi and maxi, versions of the instrument may also be developed that would allow for an even more in-depth data collection for specific health care systems and costing purposes. As suggested by the health economics experts, a user guide will be developed with the aim of providing guidance on how to use the PECUNIA RUM and how to apply it to specific settings. Furthermore, costs in other categories (e.g. environmental costs) might be of relevance for inclusion in economic evaluation. Modules on resource use in other sectors can be developed following the process described in the current paper and added to the PECUNIA RUM in the future. This is a particular strength of the structure of the PECUNIA RUM given that it can be easily adapted to include new modules without compromising its structural coherence.
6 Conclusion
This study describes the practical application of the method outlined by Thorn et al. [2] to develop the PECUNIA RUM, a multi-sectoral generic RUM instrument [57]. The process was divided into six steps starting from the identification of general attributes of the instrument (e.g. perspective, disease), to the harmonization of the instrument modules in relation to the format, phrasing and order of the questions to form a coherent instrument. While the method by Thorn et al. [2] was, overall, suitable for the development of the PECUNIA RUM, additional steps were needed to account for the complexity of developing an instrument to measure broader resource use. Further steps are needed to fully test the validity of the instrument in different settings and population groups.
References
Noben CY, et al. The exchangeability of self-reports and administrative health care resource use measurements: assessement of the methodological reporting quality. J Clin Epidemiol. 2016;74:93-106.e2.
Thorn JC, et al. Resource-use measurement based on patient recall: issues and challenges for economic evaluation. Appl Health Econ Health Policy. 2013;11(3):155–61.
Leggett LE, et al. Measuring resource utilization: a systematic review of validated self-reported questionnaires. Medicine (Baltimore). 2016;95(10): e2759.
Ridyard CH, Hughes DA. Methods for the collection of resource use data within clinical trials: a systematic review of studies funded by the UK Health Technology Assessment program. Value in Health. 2010;13(8):867–72.
Drummond MF, et al. Methods for the economic evaluation of health care programmes. Oxford: Oxford University Press; 2015.
Ridyard CH, Hughes DA, and Dirum Team, Development of a database of instruments for resource-use measurement: purpose, feasibility, and design. Value in Health. 2012;15(5):650-655.
Byford S, et al. Comparison of alternative methods of collection of service use data for the economic evaluation of health care interventions. Health Econ. 2007;16(5):531–6.
DIRUM. Database of Instruments for Resource Use Measurement [cited 2020 Jan 15]. http://www.dirum.org/. Accessed 15 Jan 2020.
Simon J, Mayer S. HEQ—health economics questionnaire v5.0 (08–09-2016). Vienna: Department of Health Economics, Center for Public Health, Medical University of Vienna; 2016.
Broadbent DM, et al. Individualised screening for diabetic retinopathy: the ISDR study—rationale, design and methodology for a randomised controlled trial comparing annual and individualised risk-based variable-interval screening. BMJ Open. 2019;9(6): e025788.
Bouwmans C, et al. Feasibility, reliability and validity of a questionnaire on healthcare consumption and productivity loss in patients with a psychiatric disorder (TiC-P). BMC Health Serv Res. 2013;13(1):217.
Wallace P, et al. Virtual outreach: a randomised controlled trial and economic evaluation of joint teleconferenced medical consultations. Health Technol Assess. 2004;8(50):1–106.
Simon J, Mayer S. HEQ (Health Economics Questionnaire) COVID-19 (24-02-2021). 2021.
Molassiotis A, et al. The effectiveness and cost-effectiveness of acupressure for the control and management of chemotherapy-related acute and delayed nausea: assessment of Nausea in Chemotherapy Research (ANCHoR), a randomised controlled trial. Health Technol Assess. 2013;17(26):1–114.
Hay A, et al. Paracetamol and ibuprofen for the treatment of fever in children: the PITCH randomised controlled trial. Health Technol Assess. 2009;13(27):1–163.
Peveler R, et al. A randomised controlled trial to compare the cost-effectiveness of tricyclic antidepressants, selective serotonin reuptake inhibitors and lofepramine. Health Technol Assess. 2005;9(16):1–134.
Watson J, et al. AESOPS: a randomised controlled trial of the clinical effectiveness and cost-effectiveness of opportunistic screening and stepped care interventions for older hazardous alcohol users in primary care. Health Technol Assess. 2013;17(25):1–158.
Knapp M, Scott S, Davies J. The cost of antisocial behaviour in younger children. Clin Child Psychol Psychiatry. 1999;4(4):457–73.
Ascher BH, et al. The child and adolescent services assessment (CASA) description and psychometrics. J Emot Behav Disord. 1996;4(1):12–20.
Wu EQ, et al. Cost effectiveness of pharmacotherapies for attention-deficit hyperactivity disorder. CNS Drugs. 2012;26(7):581–600.
Drost RM, et al. Conceptualizations of the societal perspective within economic evaluations: a systematic review. Int J Technol Assess Health Care. 2017;33(2):251–60.
Simon J. Health economic analysis of service provision. In: Geddes JR, Andreasen NC, Goodwin GM, editors. New Oxford textbook of psychiatry. Oxford: Oxford University Press; 2020.
Drost RM, Paulus AT, Evers SM. Five pillars for societal perspective. Int J Technol Assess Health Care. 2020;36(2):72–4.
Mauch V, et al. Assessing access barriers to tuberculosis care with the tool to Estimate Patients’ Costs: pilot results from two districts in Kenya. BMC Public Health. 2011;11(1):1–9.
Klingshirn H, et al. RECAPDOC-a questionnaire for the documentation of rehabilitation care utilization in individuals with disorders of consciousness in long-term care in Germany: development and pretesting. BMC Health Serv Res. 2018;18(1):329.
Chisholm D, et al. Client socio-demographic and service receipt inventory–European version: development of an instrument for international research: EPSILON Study 5. Br J Psychiatry. 2000;177(S39):s28–33.
Bouwmans C, et al. The iMTA productivity cost questionnaire: a standardized instrument for measuring and valuing health-related productivity losses. Value Health. 2015;18(6):753–8.
Bowling A. Mode of questionnaire administration can have serious effects on data quality. J Public Health. 2005;27(3):281–91.
Andresen R, Caputi P, Oades L. Stages of recovery instrument: development of a measure of recovery from serious mental illness. Aust N Z J Psychiatry. 2006;40(11–12):972–80.
Chen MY, et al. Adolescent health promotion scale: development and psychometric testing. Public Health Nurs. 2003;20(2):104–10.
Connell J, et al. The importance of content and face validity in instrument development: lessons learnt from service users when developing the Recovering Quality of Life measure (ReQoL). Qual Life Res. 2018;27(7):1893–902.
Keetharuth AD, et al. Recovering Quality of Life (ReQoL): a new generic self-reported outcome measure for use with people experiencing mental health difficulties. Br J Psychiatry. 2018;212(1):42–9.
Golsteijn RH, et al. Cost-effectiveness and cost-utility of a Web-based or print-delivered tailored intervention to promote physical activity among adults aged over fifty: an economic evaluation of the Active Plus intervention. Int J Behav Nutr Phys Act. 2014;11(1):122.
Mijnarends D, et al. Burden-of-illness of Dutch community-dwelling older adults with sarcopenia: Health related outcomes and costs. Eur Geriatric Med. 2016;7(3):276–84.
Van Eeden M, et al. An economic evaluation of an augmented cognitive behavioural intervention vs. computerized cognitive training for post-stroke depressive symptoms. BMC Neurol. 2015;15(1):266.
PECUNIA. Vision and Mission [cited 2020 Jan 17]. https://pecunia-project.eu/. Accessed 17 Jan 2020.
Fischer C, et al. Harmonization issues in unit costing of service use for multi-country, multi-sectoral health economic evaluations: a scoping review. Heal Econ Rev. 2022;12(1):1–13.
PECUNIA Group. PECUNIA Reference Unit Cost Templates (PECUNIA RUC Templates) (Version 1.0/2021). 2021. 10.5281/zenodo.5037573.
Mayer S, et al. In search for comparability: the PECUNIA reference unit costs for health and social care services in Europe. Int J Environ Res Public Health. 2022;19(6):3500.
PECUNIA Group. PECUNIA Reference Unit Costs Compendium (PECUNIA RUC Compendium) (Version 1.0/2021). 2021. https://doi.org/10.5281/zenodo.5040068.
Pokhilenko I, et al. Exploring the identification, validation, and categorization of costs and benefits of education in mental health: The PECUNIA project. Int J Technol Assess Health Care. 2020;36(4):325–31.
Janssen LM, et al. Exploring the identification, validation, and categorization of the cost and benefits of criminal justice in mental health: the PECUNIA project. Int J Technol Assess Health Care. 2020;36(4):418–25.
Fischer C, Mayer S, Perić N, Simon J, on behalf of the PECUNIA Group. Establishing a comprehensive list of mental health-related services and resource use items in Austria: a national-level, cross-sectoral country report for the PECUNIA project. PLoS ONE. 2022;17(1):e0262091.
Simon J, et al. Inter-sectoral costs and benefits of mental care in Europe: European research project PECUNIA. In: Fourteenth Workshop on Costs and Assessment in Psychiatry—The Value of Mental Health Services. 2019. Venice: Journal of Mental Health Policy and Economics.
Salvador-Carulla L, et al. Development and usefulness of an instrument for the standard description and comparison of services for disabilities (DESDE). Acta Psychiatr Scand. 2006;114:19–28.
Castelpietra G, et al. Disambiguation of psychotherapy: a search for meaning. Br J Psychiatry. 2020;219(4):532–7.
Hubens K, et al. measurement instruments of productivity loss of paid and unpaid work: a systematic review and assessment of suitability for health economic evaluations from a societal perspective. Value in Health. 2021;24(11):1686–99.
Janssen LM, et al. Aspects and challenges of resource use measurement in health economics: towards a comprehensive measurement framework. Pharmacoeconomics. 2021;39(9):983–93.
Beemster TT, et al. Test–retest reliability, agreement and responsiveness of productivity loss (iPCQ-VR) and healthcare utilization (TiCP-VR) questionnaires for sick workers with chronic musculoskeletal pain. J Occup Rehabil. 2019;29(1):91–103.
Severens JL, et al. Precision and accuracy in measuring absence from work as a basis for calculating productivity costs in The Netherlands. Soc Sci Med. 2000;51(2):243–9.
Goetzel RZ, Ozminkowski RJ, Long SR. Development and reliability analysis of the Work Productivity Short Inventory (WPSI) instrument measuring employee health and productivity. J Occup Environ Med. 2003;45(7):743–62.
Mayer S, et al. Health-related resource-use measurement instruments for intersectoral costs and benefits in the education and criminal justice sectors. Pharmacoeconomics. 2017;35(9):895–908.
Koopmanschap MA. PRODISQ: a modular questionnaire on productivity and disease for economic evaluation studies. Expert Rev Pharmacoecon Outcomes Res. 2005;5(1):23–8.
Gutierrez-Colosía M, et al. Magnitude of terminological bias in international health services research: a disambiguation analysis in Mental Health. Epidemiol Psychiatric Sci. 2022;31: e59.
Beecham J, Knapp M. Costing psychiatric interventions. In: Thornicroft G, Brewin CR, Wing J, editors. Measuring mental health needs. Gaskell/Royal College of Psychiatrists; 2001. p. 200–24.
Kingslake J, et al. The effects of using the PReDicT Test to guide the antidepressant treatment of depressed patients: study protocol for a randomised controlled trial. Trials. 2017;18(1):1–10.
PECUNIA Group. PECUNIA Resource Use Measurement (PECUNIA RUM) Instrument (Version 1.0/2021). 2021. https://doi.org/10.5281/zenodo.5036941.
Acknowledgements
The authors would like to thank the experts who participated in the PECUNIA satellite workshop in July 2019, during which the methodological approach to the development of the PECUNIA RUM was presented and discussed. The authors would also like to thank the experts who participated in the expert meeting at the annual ISPOR conference in 2019, during which the draft version of the PECUNIA RUM was presented, and Nataša Perić (Medical University of Vienna) who facilitated the organization of this meeting. This manuscript was discussed at the 2020 EuHEA PhD Student-Supervisor and Early Career Researcher and at the lolaHESG conference in September 2020. The authors would like to thank the discussants and all others who were present and contributed to the discussion. This manuscript was written on behalf of the PECUNIA Group. Members of the PECUNIA Group are: Medical University of Vienna: Principal investigator (PI): Judit Simon; team members (in alphabetical order): Michael Berger, Claudia Fischer, Agata Łaszewska, Susanne Mayer, Nataša Perić. University Medical Center Hamburg-Eppendorf: PI: Hans-Helmut König; team members (in alphabetical order): Christian Brettschneider, Marie Christine Duval, Paul Hinck, Johanna Katharina Hohls, Alexander Konnopka, Louisa-Kristin Muntendorf. Corvinus University of Budapest: PI: Valentin Brodszky; team members: László Gulácsi. Maastricht University: PI: Silvia M.A.A. Evers; team members (in alphabetical order): Ruben M.W.A. Drost, Luca M.M. Janssen, Aggie T.G. Paulus, Irina Pokhilenko. Erasmus University Rotterdam: PI: Leona Hakkaart-van Roijen; team members (in alphabetical order): Kimberley Hubens, Ayesha Sajjad. Servicio de Evaluación del Canario de la Salud: PI: Pedro Serrano-Aguilar; team members (in alphabetical order): Lidia García-Pérez, Renata Linertová, Lilisbeth Perestelo-Pérez, Cristina Valcárcel-Nazco. PSICOST Scientific Association: PI: Luis Salvador-Carulla; team members (in alphabetical order): Nerea Almeda, Pilar Campoy-Muñoz, Carlos R. García-Alonso, Mencía R. Gutiérrez-Colosía, Cristina Romero-López-Alberca. London School of Economics and Political Science: PI: A-La Park University of Bristol: PI: William Hollingworth; team members (in alphabetical order): Sian Noble, Joanna Thorn.
Author information
Authors and Affiliations
Consortia
Corresponding author
Ethics declarations
Funding
This study was conducted alongside the ProgrammE in Costing, resource use measurement and outcome valuation for Use in multi-sectoral National and International health economic evaluAtions (PECUNIA), which has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement no. 779292.
Conflicts of interest
Irina Pokhilenko, Luca M.M. Janssen, Aggie T.G. Paulus, Ruben M.W.A. Drost, William Hollingworth, Joanna C. Thorn, Sian Noble, Judit Simon, Claudia Fischer, Susanne Mayer, Luis Salvador-Carulla, Alexander Konnopka, Leona Hakkaart van Roijen, Valentin Brodszky, A-La Park, and Silvia M.A.A. Evers have no conflicts of interest to report.
Authors’ contributions
Funding for this work was secured by Prof. J. Simon, Prof. S.M.A.A. Evers, Prof. W. Hollingworth, Dr. L. Hakkaart van Roijen, Prof. V. Brodszky, Prof. L. Salvador-Carulla, Prof. A. Konnopka, A. Park, and Prof. P. Serrano-Aguilar. All authors contributed to the conception, design, and the conduct of the study. The first draft of the manuscript was written by I. Pokhilenko and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.
Ethics approval and consent to participate
Not applicable.
Informed consent
Not applicable.
Availability of data and materials
Not applicable.
Code availability
Not applicable.
Additional information
The members of the PECUNIA Group are mentioned in Acknowledgments section.
Supplementary Information
Below is the link to the electronic supplementary material.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License, which permits any non-commercial use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc/4.0/.
About this article
Cite this article
Pokhilenko, I., Janssen, L.M.M., Paulus, A.T.G. et al. Development of an Instrument for the Assessment of Health-Related Multi-sectoral Resource Use in Europe: The PECUNIA RUM. Appl Health Econ Health Policy 21, 155–166 (2023). https://doi.org/10.1007/s40258-022-00780-7
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s40258-022-00780-7