Background

Patient safetyFootnote 1 is a worldwide problem. Among the countries in the Organisation of Economic Cooperation and Development (OECD), one in 10 patients are harmed while receiving hospital care [1,2,3,4], and nearly 50% of such cases are considered preventable [5]. Worldwide, four out of 10 patients are harmed while receiving health care in a hospital setting, and 80% of such cases could have been prevented [5].

Similar evidence has been found in Canadian hospitals, revealing that “one in fourteen patients suffer from some form of harm, with a third of such cases being preventable” [6]. In addition, the latest evidence in Canada reveals that deaths related to such incidents occur every 13 min [7]. A report called “Measuring Patient Harm in Canadian Hospitals” reveals that, in 2014–2015, harm was experienced by patients during one out of every 18 hospital stays, or 138,000 hospitalizations. Of those, 30,000 (or one in five) involved more than one form of harm [8]. Moreover, medical errors in both the acute and home care settings can cost $6800 per patient, resulting in $2.75 billion in additional costs each year in Canada [8]. Estimated costs related to incidentsFootnote 2 and accidentsFootnote 3 in hospitals represent the costliest form of care, accounting for over $58 billion per year across the country [10]. Similarly, recent evidence demonstrates that 15% of total hospital expenditures and activities in OECD countries is a direct result of adverse events. Estimates show that the total cost of harm in these countries alone amounts to trillions of US dollars every year [4]. One out of every seven Canadian dollars spent on health care is spent treating the effects of patient harm in hospital care [11]. Therefore, it should come as no surprise that investing in the cost of prevention is much lower than the cost of care required due to harm [4].

Today, it is widely recognized that patient engagement (PE)Footnote 4 can help improve outcomes and reduce the burden on health services and on patient safety [12,13,14,15,16,17,18,19,20,21]. Indeed, partnering with patients for the sake of their own health and care is known to be a key component for developing the highest quality of healthcare [12, 19, 20, 22,23,24,25,26]. This is why implementing PE strategies offers undeniable value to health care systems by helping reduce by up to 15% the burden of patient harm in hospital care, offering potential savings of billions of dollars each year [4]. So it is imperative for healthcare organisations to be able to assess their PE strategy and implementation status in their organisations to gain a better understanding of their PE performance, specifically regarding risk management (RM), for patient safety (PS) [27].

Patient engagement (PE) has grown in importance as a priority for ensuring quality of care and patient safety (PS) in many Canadian and provincial organizations [28,29,30,31]. The knowledge on the best strategies for building a safer health care system grounded in collaboration between patients and healthcare institutions for PS has not yet been properly synthesized [13, 23, 32]. Such strategies address all parts of the system holistically rather than as silos. There is emerging evidence and leading practices on how to implement PE by involving patients in PS [23, 31, 33], but the impacts of such best practices has not been properly researched and demonstrated [12, 19, 20, 23, 34, 35]. In addition, system-wide identification of PS practices is necessary for better-quality and safer healthcare systems [12, 19, 20, 23, 34, 35]. However, existing PE in PS tools mostly identify initiatives at the clinical, organizational or strategic level, without capturing the whole system (see Table 1 for more information: Adapting questions in the tool with the latest evidence from 2017 to 2020).

In this context, the province of Quebec, Canada created a PE in PS questionnaire in 2017. The questionnaire was designed to assess strategies implemented at the level of the health system to integrate PE in PS practices in healthcare organizations [31, 34]. This tool was validated and used from 2017 to 2019 in all healthcare organizations (n = 24) in Quebec [35]. Knowing that the questionnaire built in Quebec was created in French, CPSI was asked to adapt the tool in the English language to help healthcare managers assess a system-wide integration of PE in PS practices across the rest of the Canadian country and around the world. The tool incorporated concepts of “Safety I” (situations that can go wrong) and “Safety II” (what goes right and the system’s ability to succeed despite conflicts, uncertainties and risks) [36]. It tracks changes over time based on organizational best practices of PE in PS.

The purpose of this article is to present the development of a pan-Canadian survey tool to be used by subject matter experts in PE and PS (PS managers, risk managers or a task group, and patient partners) in order to self-assess the nature of PE in PS structures, strategies and factors at the system level in Canadian healthcare organizations (CHOs) and follow up on improvements in these PE in PS strategies over time. A five-step process was used (see Fig. 1 for more information):

  • Step 1: Literature review and revision of the initial tool

  • Step 2: Translation of the French questionnaire into an English version

  • Step 3: Creation of the Canadian expert advisory group

  • Step 4: Assessment, adaptation and editing

  • Step 5: Pilot testing and pre-validation of the tool

Fig. 1
figure 1

Pan-Canadian survey tool development processes

Full size image

Methodology

In this section, we present the five different steps taken to develop the Pan-Canadian PE in PS survey tool. Please refer to Fig. 1 for the summary of the different phase to build the survey tool.

Step 1: literature review and revision of the initial tool

Methods

We performed a literature review to complement the work performed developing the French questionnaire and to capture publications published from 2017 to 2019 regarding best practices in PS and PE and/or evaluation tools to survey PE on PS. Health and social sciences databases (PubMed, Medline, Cochrane, CINAHL, EMBASE) were consulted using the keywords “tool” OR “assessment” OR “measurement” OR “questionnaire” AND “patient engagement” OR “patient participation” OR “patient involvement” OR “patient safety.” We also integrated comments and suggested modifications collected during the final validation phase of the French tool, conducted in the province of Quebec, Canada [34] and from a study conducted in France [37] (see Tables 1 and 2 for more information).

Table 1 Step 1: Adapting questions in the tool using the latest evidence from 2017 until 2020
Full size table
Table 2 Step 1: Results from data collected from the study conducted in Quebec and France. Data collected from the study conducted in Quebec and France [38]:
Full size table

Findings

The literature review highlighted four tools for evaluating PE and/or PS at different system levels (see Table 1). Among the four tools identified, none tackled both PE and PS at the organizational or system level. The first tool focused on PE in PS at the clinical level, the second tool focused on PE only at the organizational level. The third and fourth tools focused on PE in research methods and its evaluation.

Suggested modifications from previous studies conducted in Quebec and France were also analyzed and integrated into the assessment of the creation of the pan-Canadian PE in PS survey tool (see Table 2 for detailed information). In summary, participants in the Quebec and French studies reported that they generally had a good understanding of the tool and its questions as well as the instructions and the choice of the participating team in answering the questions:

“The tool helps us really frame our strategies and could be used as a self-assessment tool for healthcare institutions [ … ] It will be used for sure.”

“You're asking very pertinent questions, and that helps me.”

“The choice of having a team answer the questionnaire is very important [ … ] It has to be a group that answers the questions. No individual has a comprehensive view of what is going on in the institution.”

At the end of the evaluation of the Quebec and French studies, nine questions were deleted to avoid repetition and the rewording of certain items, and some questions were scaled to improve understanding. Questions 67–73 were repetitions so they were deleted (N = seven repeated questions deleted); two questions pertaining to Family Medicine Groups were also deleted, as they were confusing since not all healthcare organizations did not have Family Medicine Groups and since these questions were specific to the context of the province of Quebec. Other limitations of the previous questionnaire were about the percentages in the questionnaire that were difficult to complete since healthcare organizations did not necessarily know the percentage of engagement of their users in different instances and committees. We thus, modified these sections not to include the percentages, but to just let the participants answer about the degree of involvement of their users (ex: I don’t know, never, sometimes, always, in progress).

Another limitation were about questions on policies and procedures which needed not to just include if the organization had a certain policy or procedure, but also how these policies were integrated and experienced in the organization.

“The absence of policies and procedures does not reflect dedication. Pay attention to the presence of procedures and policies and how they are experienced.”

Modifications were brought to these questions (N = 25 questions on policies and procedures) and clarification were added to the main question. Here is one example:

  1. Q.

    32. The organization has a formal policy on disclosure

    32a. The disclosure policy is periodically evaluated

    32b. The disclosure policy is evaluated in partnership with patient advisors?

Step 2: translation of the French tool into an English version

Methods

The French tool was translated into English and validated following the methodology (back translation) proposed by the Agency for Healthcare Research and Quality [38] and the methodology recommended by the United States Bureau of the Census [39] (see Fig. 2 for more information on the four steps used in the back translation).

Fig. 2
figure 2

the four steps used in the back translation

Full size image

Findings

The back translation was done in four steps:

  1. 1.

    Translation of the French tool into an English version by the research team.

  2. 2.

    Translation of the English version back into the original language by an external resource

  3. 3.

    Comparing that new translation with the original French tool

  4. 4.

    And, reconciling any meaningful differences between the two versions and bringing about the corrections.

Thus, the tool was translated from the French version used in the Quebec study [34] into an English version, and the English version was translated back into French to confirm the terms used. During the translation and the back-translation, we made sure not to change or modify the meaning or depth of the items in order to not affect the validity of the content.

Step 3: creation of a Canadian expert advisory group

Methods

To adapt the French version, a Canadian expert advisory group was created to adapt the tool to suit a pan-Canadian context, considering the specific features of Canadian health systems. The criteria used to select members were: their knowledge of and expertise in the Canadian health system, patient engagement and patient safety; having already collaborated with CPSI, and their province of origin (for purposes of representation). An initial list of experts was established by CPSI and the research team, emails were sent to the identified individuals, and a final selection was made by the researchers and CPSI using the selection criteria mentioned above.

Findings

In total, eight members from five Canadian provinces were recruited and each of them filled out the survey. Thus having eight surveys completed. The eight members are comprised of: 1 academic, 1 patient representative, 4 quality improvement and patient safety specialists, and 2 experts from Accreditation Canada. The research team in charge of developing the study objectives and methodology had five members (three PhD students, one researcher expert in PE, and one advisor from CPSI as a PS and PE projects expert) (see Table 3). In total the working group had 13 members, met 16 times (the research group met with the expert advisory group 4 times and 12 times alone), and on average the meetings lasted 2–3 h.

Table 3 Composition of the research team & the expert advisory group for the PE in PS pan-Canadian survey tool
Full size table

Step 4: assessment, adaptation and editing

Step 4.1: assessment and adaptation of the tool content

Methods

The Canadian expert advisory group reviewed the initial translated English tool for assessment and adaptation purposes. The expert advisory group were also instructed in their revision of the tool to focus on each question and on a more general aspect of it by keeping in mind the flowing interrogations for content and construct validity,Footnote 5 as well as for and reliabilityFootnote 6:

  1. 1.

    Wording of the questionnaire?

  2. 2.

    Themes covered?

  3. 3.

    What is missing or redundant?

  4. 4.

    Relevance of the questions?

  5. 5.

    Target population (who can answer the questionnaire)?

  6. 6.

    Can the questionnaire be used to assess the reality of patient safety in Canada (so that CPSI and the provinces can use the findings to make better decisions and update the PE in PS guide)?

The goal of the initial tool review and assessment by the expert advisory group was to work as a group using a consensus-building approach to evaluate every single dimension of the initial questionnaire, in order to arrive at a consensus on the content and construct validity as well as on the reliability of the entire tool (i.e. so that all the dimensions used in the Canadian survey would be as relevant, reliable, and exhaustive as possible to fully explore PE in PS in Canada). Note that McDowell and Newell (1987) suggested this method as a way of confirming the content validity of a questionnaire when studying an unknown or new phenomenon in a large organizational or geographic context, such as in our case. McDowell and Newell (1987) suggest that questionnaire dimensions are better developed if they are defined and set up on the basis of the consensus, representativeness, relevance, and exhaustiveness of the constituent items of the concept or the topic to be studied [9, 39].

Based on their knowledge of and expertise in Canadian PE in PS, the expert advisory group assessed all the dimensions of the entire initial tool, item by item. When certain items were unclear, incomplete, or inconsistent, the expert advisory group proposed how they could be corrected or further developed, and then, the expert advisory group would agree or disagree as a group and the consensus was calculated democratically by vote. The highest votes will then be selected to reflect the final decision to be brought as a modification in the survey (to note that we did not have any major disagreements. The expert advisory group came to a consensus through a collaborative strategy). Through the assessment and adaptation process, the expert advisory group was able to create, modify, and adjust certain items to reinforce the initial tool’s (the translated English version) validity and consider the latest trends in PE in PS as practised by CHOs. For example, the initial translated English version tool did not have questions covering the leaders’ own perceptions of change, the methods, and the techniques used by organizations to measure and assess the PE in PS outcomes and change. The initial translated English version tool review sometimes led to minor changes to items or simply adaptations, and at other times it led to the creation of new items based on new trends in PE in PS. For instance, a new item was created on the CHOs’ managers’ perspectives on the impacts of PE in PS and which factors most influence PE in PS, whether by enhancing or limiting PE in PS. Other new items were created in the section on demographic characteristics and information on the respondents’ organizations, such as level of experience with the PE approach, competencies in the field, and level of PE understanding.

Findings

The assessment and review of the initial translated English version tool was the key stage in the entire tool development process. The content and construct validity resulted in the creation of a complete adapted version of the tool consisting of 75 items in 5 sections on the following 10 dimensions: (1) demographic characteristics, (2) experience level, (3) incentives, (4) strategies, (5) level of intervention, (6) structure and resources, (7) activities, (8) factors, (9) impacts, and (10) improvements (see Table 4). As per the reliability of the survey, all the experts in the expert advisory group when completing the survey for the second time, were able to give the same answer they gave to the questions with minor differences in their answers to new questions or modified questions.

Table 4 Presentation of the dimensions developed in the questionnaire
Full size table

Step 4.2: assessment and adaptation of the wording of the tool for pan-Canadian use

Methods

In this step, all the members of the expert advisory group carefully examined the wording of every aspect of the tool and suggested improvements when they found inconsistent expressions and vague vocabulary. In addition, two members of Accreditation Canada (AC) performed a careful reading and a deep examination of the tool’s wording, based on their knowledge of and expertise in Canadian healthcare evaluation standards. Here again, a consensus-building approach was adopted to integrate the feedback and the suggestions provided by all the expert advisory group members on the tool’s wording and language.

Findings

At the end of this step, our tool was formally defined as pan-Canadian. This step allowed us to adjust its wording to ensure that it would be understood by all CHOs. The expert advisory group’s comments on the adjustments made to the tool’s wording are found in Appendix A. Please refer to Tables 4 and 5 for more information on the changes made to obtain the final version of the pan-Canadian PE in PS survey.

Table 5 Step 4: Before and after tool’s adaptation: Layout of the tool
Full size table

Step 4.3: editing and assessment of technical aspects

Methods

This stage was focused on placing the questionnaire on the online platform, testing the technical aspects, and revising the questionnaire before the pilot test. The research team received specific training from the information technology staff at CPSI on how to manage and edit on the online platform. Then the online tool was internally tested by the members of the research team.

Findings

At the end of this stage, the questionnaire was set up online, approved by the working group, and considered ready for use in the pilot test. Please refer to Table 5 to see how the questionnaire was developed and adapted from the initial version (French version) to the final version (pan-Canadian version). In addition, please refer to the following link for the pan-Canadian survey, available on the CPSI online platform: https://survey.patientsafetyinstitute.ca/n/zz16p.aspx

Step 5: pilot testing and pre-validation of the tool

Method

In this final phase, we tested and validated the tool among real CHOs. To this end we asked members of the National Health Engagement Network (NHEN), a community of practice, to participate in our pilot test. Once some members had agreed to participate and had given their consent to help test the survey, an email was sent to them explaining the instructions for completing the questionnaire along with the link to the online survey.

Findings

The questionnaire was completed in September 2020 by six organizations: 2 in British Columbia, 2 in Ontario, and 2 in Newfoundland. Among these 6 organizations, 3 have a mandate to provide acute care, 2 provide long term care, and 1 is specialized in mental health care. For each organization, the tool was completed simultaneously by a team of three members (one manager in charge of quality and risk management, one person in charge of PE, and one patient advisor). The average time to complete the questionnaire was 52 min, with a maximum of 67 min and a minimum of 27 min.

Once all the responses had been received, the research team exported and analyzed the results on CPSI’s online platform dedicated to the collection and processing of survey data. Based on the pilot test results and participants’ comments, the research team made the necessary adjustments and amendments directly in the tool and emailed it along with the pilot test results to the members of the expert advisory group for final review and approval. Then, all expert advisory group members reviewed the pilot test results, revised the entire tool, approved the changes and adjustments, and gave their final approval. As per the validity and reliability of the survey, the participants were able to complete the survey as per the expert advisory group. We were able to see a consistency in the type of answers giving to a specific question.

Following the pilot test, the tool was considered ready to be used officially in the final validation step: “at the pan-Canadian level.” Table 6 presents the tool items that were modified, adjusted, or adapted based on the pilot test results.

Table 6 Step 5: Aspects of the tool that have been modified, adjusted, or adapted based on the pilot test results
Full size table

Table 5 shows the final sections (layout) of the pan-Canadian PE in PS tool.

Discussion

The Pan-Canadian PE in PS survey is an innovative tool to help self-assess the nature of PE in PS structures in CHOs and to monitor changes over time. To our knowledge, no similar tool exists in Canada or elsewhere in the world to identify strategies and initiatives related to PE in PS at a system level. To test its validity, the pan-Canadian PE in PS survey tool was reviewed by an expert advisory group and tested in a pilot test (see Step 5 of the methodology development process).

This article presents a description of the five-step process used to adapt, develop, translate and validate an existing tool on patient engagement (PE) in patient safety (PS), which was co-constructed with patients in the CHOs. The pan-Canadian PE in PS survey tool, intended as a self-assessment tool to be used by subject matter experts in PE and PS who integrate patients, was developed in five steps: (1) a literature review and revision of the French tool; (2) translation of the French questionnaire to English; (3) creation of a working group; (4) assessment, adaptation and editing; and (5) pilot testing and pre-validation of the tool. A preliminary step was also conducted to validate the tool’s content validity (Step 0).

The final version of the PE in PS pan-Canadian survey tool comprises of 75 questions divided into four sections with ten dimensions: Section 0 contains questions for demographic identification of the participants (Q1 to Q5); Section 1 (Q6 to Q17) has general questions to establish participants’ level of experience and organizational incentives for PE in PS; Section 2 (Q18 to Q33) contains questions related to PE processes, such as strategies, activities, structures, resources and factors; Section 3 (Q34 to Q67) has questions on PE in PS processes, such as activities, strategies, structures, resources and factors in place; and Section 4 (Q68 to Q75) contains questions on the context and impact of PE in PS initiatives in the CHOs. More specifically, these questions are focused on outcome identification, improvement mechanisms and strategies, evaluation mechanisms, and indicators.

The added value of the pan-Canadian PE in PS survey tool

The research team identified six forms of added value provided by the pan-Canadian PE in PS survey tool. First, this is the only self-assessing tool that identifies PE in PS at a system level in healthcare organizations. Based on our research and to our knowledge, no such tool exists at a system level, either in Canada or elsewhere in the world. Second, the tool informs policy and strategic decisions at a national level. At that level, leaders are able to understand the spread and depth of PE in PS across Canada in order to demonstrate what works, which in turn will strengthen the commitment to safe care (by offering evidence-based programs, thereby contributing to one of the five goals of an exceptional healthcare system, as set out in the Canadian Quality and Patient Safety (CQPS) framework [40]). At the provincial level, the tool allows leaders to understand PE in PS in their jurisdictions and how it compares with that of other provinces in order to focus and coordinate their efforts. At the organizational/operational level, the tool allows leaders to understand what works and to implement the practices that can most effectively improve safety. Third, the tool informs practice, identifying the factors, mechanisms, and strategies that effectively improve patient safety through engagement (reduce and prevent harm, reduce the economic burden of patient safety incidents). Fourth, the tool promotes partnering with patients to improve patient safety at the organizational level, and but also to improve care safety at all system levels in Canada. Fifth, the tool and process can be transferred for use in different contexts around the world. Sixth, the tool helps in the CHO accreditation preparation process (e.g. as a survey instrument, a component of the QmentumFootnote 7 program of Accreditation Canada).

Strengths of the tool development process

First of all, the tool was adapted from an existing tool tested in a very interesting study conducted in the province of Quebec [34]. This helped create a solid foundation for our tool development process. Second, the creation of the expert advisory group was a tremendous strength in the process used to develop the survey tool. The remarkable expertise and knowledge in PE and PS among the members of the expert advisory group, and the fact that they came from different provinces and worked at different management levels and strategic positions in the CHOs, helped us by enhancing the content validity but also by reinforcing the overall methodology (see Table 3 for more information). And lastly, the involvement of Accreditation Canada (AC) was a great asset, especially in terms of structuring our methodology, defining and selecting the right wording, and connecting with CHOs for the pilot project.

Limitations

Five limitations of this study were identified during the development of the pan-Canadian PE in PS survey. The first is the fact that an expert translator was not used to translate the initial tool from French to English. However, the research team and the expert advisory group were able to use a back translation method to interpret and adapt the wording to the survey’s pan-Canadian context and review the terms frequently employed by the CHOs in their work environments. The second limitation, which is related to the first, concerns understandings of certain terms. Even though we worked closely with many Canadian experts active in different Canadian heath systems, we believe that this added value resulted in different meanings being ascribed to certain wordings and definitions. For this reason, we incorporated common definitions of certain words that we found differed from one province or CHO to another. We also believe that such discrepancies were few in number. The third limitation concerns the fact that only one patient partner was involved as a member of the expert advisory group. However, the patient partner belongs to a patient organization where the individual has access to an extensive patient partner network. The fourth limitation involves the ability of healthcare professionals to facilitate patient interaction and participation by encouraging patients to ask questions related to patient safety and take an active part in their care and in the healthcare system. Further research will need to identify strategies engage patients in the process of engaging patients to participate in completing surveys in a team setting [41].

The fifth limitation concerns the requirement of having a group of three people (patient safety participant, patient engagement participant, and a resource patient or patient partner) complete the survey together. This could be a major issue for the survey as it becomes more widely used across Canada. While this may add value, it also reduces the chances that the survey will be completed, resulting in a potentially lower response rate. We nevertheless kept this requirement of a group of three people completing the PE in PS survey tool, because of the importance of receiving feedback from the various departments of CHOs.

Conclusion

The task of developing a new tool by adapting or translating an existing tool into another language and broader context might seem overwhelming. Perhaps the greatest challenge was to produce a tool that is linguistically comprehensible, psychometrically sound, and efficient and effective for use in organizational research settings. This article provides a description of the process used to develop, translate and validate an existing tool for investigating how to engage patients (PE) in patient safety (PS) within Canadian healthcare organizations (CHOs).

The tool is currently being used by healthcare organizations in Quebec and France [37]. A version in Portuguese is currently being prepared to assist healthcare managers in monitoring changes in PE in PS at the system level.

Above all, it must be said that such a tool can meet the WHO’s need to identify and compare PE in PS initiatives around the world. The PE in PS tool that we have created therefore has the potential to help health care organizations identify gaps in their PE in PS as a way to reduce incidents and accidents related to errors by integrating PE into their safety practices. It will also give health care organizations access to a database of recognized international PE in PS practices and strategies, so that they can improve their practices and monitor improvements over time.

The next step in the pan-Canadian PE in PS tool will be use in an official pan-Canadian final validation study in CHOs.