Evaluation of VHP Program

Abstract

A thorough and methodical evaluation of any program is imperative in determining its effectiveness. Prior to this chapter, a multitude of evidence-based practices and the importance of their application in promoting and ensuring vessel health and preservation have been described. The question is, how do we promote and ensure adherence to these sound practices and guidelines that when implemented will improve patient clinical outcomes and patient experience? How do we best implement these practice changes? And, what is the “right” way to evaluate components of a vessel health and preservation program? The objective of this chapter is to describe implementation strategies and to explore strategies for evaluation of a VHP program.

1 Introduction

The last quadrant of the VHP model includes processes to evaluate the success of your implementation, identify gaps and weaknesses, consider evaluation of new products to meet a need, and focus on patient and clinician satisfaction measurements. Improvement of care is impossible without an established process of evaluation. Right evaluation for VHP program application includes outcome measurement of complications, observation of policy performance, and plan to provide education for staff departments and units with negative outcomes or practice deficiencies. A multimodal quality program applies education of guidelines and recommendations ensuring that practices are consistent, and that staff is well informed. Adding new products should involve an established process for evaluation and trialing to ensure performance at the company stated levels. Integrated within a VHP program is the evaluation of products, supplies, and technology, both existing and considering new product trial testing. Each product should undergo periodic assessment to determine performance according to the facility needs and expected application. Each of these areas of evaluation requires measurement criteria, periodic checks for application, and compliance, all to ensure a successful and high-quality program.

According to the VHP model, the right evaluation must include audits and outcome monitoring which lead to identification of educational needs and use of products that can contribute to improved outcomes (Moureau and Carr 2018). Outcomes include complication rates, observation of compliance to organizational policies and procedures, and evaluation of clinician competencies. Examples of specific outcome measurements include catheter-related bloodstream infection rates, venous thrombosis rates in patients with PICCs, peripheral IV-related complications, and catheter occlusion rates. Process outcomes that might be evaluated in a VHP program include appropriate use of PICCs versus midline catheters, site care, and dressings performed in accordance with policy, intactness of VAD dressings, prompt removal of VADs when no longer necessary, and compliance with adhering to the components of the central line insertion checklist.

Outcomes are not just measured but are used to validate performance or to identify needed areas for quality improvement interventions. Audit and feedback are a widely used and recommended strategy in implementation and evaluation of practice changes. It is defined as a summary of clinical performance that may include recommendations for action; information is gathered over a specified period of time and used to increase group awareness of theirs and/or others’ practice (RNAO 2012; Ivers et al. 2012). Audit and feedback are usually used in conjunction with other interventions, namely, educational strategies. As stated by Ray-Barruel (2017), ongoing audits are used to ensure compliance with care and will provide the evidence required to ensure that an organization is on track to excellence in VAD care. Clearly, without attention to program evaluation, deficiencies and patient outcomes would not be identified, and opportunities for performance improvement including necessary clinician education would not be identified.

2 Implementation and Quality Improvement Strategies

Both implementing and sustaining changes in practice are quite challenging. The Cochrane Effective Practice and Organization of Care (EPOC) group classifies quality improvement strategies used to change practice into four main categories (Flodgren et al. 2013). These strategies may be used either alone or in combination with improve adherence:

• Organizational: May include changes in professional roles or skill mix changes, for example, champion leaders or implementation of a specialty team.

• Regulatory: For example, government required healthcare initiatives.

  • Example: Requirements for infection or other outcome reporting.

• Financial: Constraints of reimbursement based on clinical outcomes or patient satisfaction. This is a strong motivator for implementing practice changes!

  • Example: In the United States, central line-associated bloodstream infection (CLABSI) rates in hospitals are publicly reported on the Medicare.gov website (“hospital compare”). Hospitals face penalties for high CLABSI rates, and any vascular access device-related BSI is considered a preventable event with treatment not reimbursable under medicare. The patient experience which includes satisfaction measures also factors into reimbursement.

• Professional interventions: The focus of this chapter may include educational strategies, the use of local opinion leaders, and the use of audit and feedback and reminders.

The Society for Healthcare Epidemiology/Infectious Disease Society of America (SHEA/IDSA) (Marschall et al. 2014) identifies an implementation strategy for CLABSI prevention that can also be applied to implementation of other evidence-based practices. It includes the following four steps:

• Engage: Includes multidisciplinary teams that include frontline and senior leadership in the process and outcome improvement plan, identification of local champions (e.g., vascular access/infusion clinicians), and focus on a culture of safety.

• Educate: The use of a variety of educational methods and strategies.

• Execute: Use quality improvement methodologies (e.g., plan-do-study-act) to structure prevention efforts, to standardize care processes (e.g. guidelines, bundles), and to create “redundancy” such as with visual reminders, checklists, posters, screen saver messages, etc.

• Evaluate: Include audits of both process (e.g. compliance with insertion bundle) and outcome (CLABSI rate) measures and link to initial and ongoing competency assessments and the critical step of providing feedback to all healthcare staff. Examples include compliance with insertion checklists with graphs showing cumulative compliance with process measures.

In the Infusion Therapy Standards of Practice, the Infusion Nurses Society addresses the importance of quality improvement programs including monitoring of adverse outcomes including infection rates, reporting such data regularly to both clinicians and leadership, and minimizing and eliminating barriers to change and improvement (Gorski et al. 2016). The Standards for Infusion Therapy from the Royal College of Nursing (2016) include similar recommendations and specifically address the need for auditing as an ongoing process in order to monitor, maintain, and improve clinical practice in infusion therapy and the need for timely dissemination of audit results to develop a culture of learning and quality improvement. The epic3 guidelines (Loveday et al. 2014) recommend the use of quality improvement interventions to support and manage VADs including the use of audit and feedback of compliance with practice guidelines.

The Registered Nurses’ Association of Ontario (RNAO 2012) provides a “knowledge to action” framework. Prior to implementation of interventions, an important step is to recognize and understand barriers or roadblocks as well as “facilitators” to practice change. Facilitators are positive characteristics supporting practice change such as positive staff attitudes and beliefs, leadership support, organizational “champions,” and interorganizational collaboration and networks (RNAO 2012). Examples of barriers to implementing practice changes include:

• Potentially negative attitudes by clinical staff, often related to organizational problems or issues such as inadequate staffing, high patient acuity levels, and organizational changes.

• Lack of interest or lack of knowledge which may also be impacted by many competing clinical priorities (e.g., reduce incidence of falls, pressure ulcers, nonvascular-associated infections).

• Guidelines or practice changes are not integrated or “hard-wired” into organization practices or policies/procedures (RNAO 2012).

It is important to acknowledge that the presence of policies does not result in adherence to best practices. In a survey of 975 US hospitals enrolled in the National Health and Safety Network, data were provided on the presence of policies in 1534 ICU units. There was a widespread presence of policies relative to CLABSI prevention (87–97%); however adherence to such policies ranged from 37 to 71% (Stone et al. 2014).

Another potential barrier to implementing evidence-based practices is how and in what way do key persons assess and champion the evidence. If the evidence is believed to be weak, implementation of a practice is not likely. The perceived strength of evidence relative to evidence-based infection prevention practices was studied in a survey of infection prevention personnel (Saint et al. 2013). Relative to CLABSI prevention, 90% or more survey respondents perceived the following evidence as strong: skin antisepsis with chlorhexidine, maximal sterile barrier precautions, and avoiding the femoral site. The researchers recommend that when translating evidence into practice, consideration should be given as to how key individuals assess the strength of evidence.

Conduction of an organizational survey may be a helpful strategy for identifying facilitators and barriers. Once understood, it is important to build upon identified facilitators and examine ways to tackle the barriers. Examples include addressing the educational needs and the level/type of education needed, providing adequate time for implementation, and exploring motivational strategies.

3 Exploration and Effective Use of Audit and Feedback Strategies

As stated earlier, audit and feedback are a widely used and recommended strategy in implementation of practice changes. A systematic review of literature was performed to explore features of educational interventions that led to competence in aseptic central line insertion and care in acute care settings (Cherry et al. 2010). Based upon a review of 47 studies that met the researchers’ inclusion criteria, educational interventions were most effective and more likely to result in outcome improvement when used in conjunction with audit and feedback as well as availability of the clinical supplies consistent with the education. Education alone is not enough to improve outcomes and clinical practice!

In a Cochrane study including 140 randomized trials involving audit and feedback, the effect on professional behaviors and patient outcomes ranged from little/no effect to a substantial effect (Ivers et al. 2012). While most of the reviewed studies measured impact on physician practice, some involved nurses and pharmacists. Concluding that audit and feedback can lead to small but important improvements in practice, the researchers found audit and feedback strategies are most effective when:

• Baseline performance is low to begin with.

• It is provided more than once.

• It is provided by a supervisor or colleague.

• It is provided verbally as well as in writing.

• There are clear goals/targets and an action plan.

In a follow-up analysis to the Cochrane review, Ivers et al. (2014) state that audit and feedback are one of the most studied quality improvement interventions. While again asserting that audit and feedback lead to small but potentially important practice changes, they state that “the appropriate question is not: ‘can audit and feedback improve professional practice?’ but: ‘how the effect of audit and feedback interventions can be optimized?’” (Ivers et al. 2014). Citing evidence that repeated feedback is more effective, the researchers opine that studies continue to evaluate interventions after only one cycle of feedback. There remains the need to identify the key, active ingredients of audit and feedback that will lead to a greater impact of interventions, and produce more generalizable outcomes.

Using a modified grounded theory approach involving 72 clinicians, including nurses, in ICU settings, the use of audit and feedback was perceived as fragmented and variable, not transparent, and provided information that was neither timely nor “actionable” (Sinuff et al. 2015). To improve the process, the researchers suggest the following: providing rationale for practice changes and for the audit process, incorporating feedback into daily activities such as rounds, developing audit and feedback criteria that are specific and transparent, and providing information that can be translated into specific actions. Furthermore, the feedback must be timely. For example, finding out that there were two-line infections in the previous month gives little information if the patient is not identified and what were potential breaches in expected practice. Participants in the study feedback should provide specific actions “which can be acted upon more immediately” (Sinuff et al. 2015: 396).

Another small qualitative study sought to describe the perception of nurses on the effectiveness of audit and feedback (Christina et al. 2016). Nursing practice relative to hand hygiene, the use of smart pumps, and intravenous site assessments was evaluated by a nurse manager or clinical nurse specialist who entered the patient’s room with a checklist and documented the nurse’s work weekly or biweekly. Feedback was provided to the nurse shortly after the audit. Three themes identified by the nurses included:

• 1. Relevance: The need to understand the purpose of audit and feedback and the link between the audit criteria and patient outcomes.

• 2. Timing and feedback: Some nurses felt that the audit process occurred too early during the shift, and they were not able to complete the expected work in such limited time.

• 3. Individual factors: A perception of criticism resulted in greater tendency to respond in a more negative manner.

Summarizing implications for nurse managers, the researchers suggest making the audit and feedback process relevant and transparent in its purpose, involving nurses in planning and developing the process and providing feedback with clearly evident goals and action plans to make feedback less personal (Christina et al. 2016).

Strategies for improving the audit and feedback process are summarized in Table 20.1.

Table 20.1 Strategies for improving the audit and feedback process

Fig. 20.1

Clinician evaluation tool for VHP (used with permission of Teleflex)

4 Examples: Successful Integration of Audit and Feedback into Practice

A quality improvement project implemented in a two-site hospital system involved the use of nurse-specific report cards providing feedback on central line management (Morrison et al. 2017). Using both visual and documentation audits, the authors sought to understand if known risk factors (e.g., maintaining dry intact dressing, maintaining IV set integrity) associated with central line-associated bloodstream infection (CLABSI) could be decreased. Using feedback intervention theory, information that is timely, detail focused, based on goals, and aimed at changing behavior and improving performance was provided. Nurses were provided a four-part computer-based educational module, and a trial of an alcohol disinfection cap was initiated before the audits. During the auditing period, additional interventions included house-wide implementation of the alcohol disinfection cap, updating of policies and procedures, and redesigned central line dressing/implanted port needle insertion trays. The weekly visual audit consisted of observation of dressings (e.g., dry and intact, chlorhexidine-impregnated dressings, dressings dated) and administration sets (e.g., labeling, use of disinfection caps, no looping) and also a documentation review (e.g., site assessment and care, catheter patency, dressing change date corresponds to date on dressing).

Over the 16-week period, 487 nurses received a “report card,” which included 620 central lines; 113 failed the visual audit with 54% having at least one CLABSI contributing factor. Over time, there was a decrease in CLABSI contributing factors which the authors believed resulted from the personalized report cards. The nurses’ responses to the report cards were briefly discussed and included justification of failures (not my fault), excuses (float nurse), and dismay. Outcome data included a decrease in CLABSIs; statistical significance of this outcome was not reported. Based upon the evaluation of this quality improvement study data, additional organizational changes were made including a face-to-face line care module for newly hired nurses, mandatory annual CLABSI prevention education, immediate intervention to correct failures at the time of the audit, and redesign of the electronic medical record IV documentation (Morrison et al. 2017).

In another example, a reduction in CLABSIs was reported over a 5-year period. Daily rounds of all central lines in an ICU setting revealed a substantial number of occluded lines as well as inadequate clinical practices including unlabeled dressings, improperly placed chlorhexidine foam dressings, and breaches in aseptic technique (Matocha 2013). Outcome data was used to support practice changes, product changes, and education and competency training.

In an emergency department, placement and care of peripheral IV catheters (PIV) was addressed using audit and feedback (Fakih et al. 2012). A quasi-experimental pre- and post-study methodology was used. Pre-study data included 10–20 observations of PIV placement per week, evaluation of documentation, and about 10 intravenous medication administrations observed per week. Steps observed include hand hygiene, skin antisepsis including air drying, maintaining aseptic technique during placement, and application of a sterile dressing. Implementation included education with a pre- and posttest as well as the same observations done pre-study. Two hundred twenty completed PIV placements were observed during the study period. Very poor compliance and lack of knowledge was evident relative to proper insertion technique pre-implementation (4.8%) and improved over time (30.9% during implementation, 31.7% post-implementation).

While the researchers found that audit and feedback were associated with a statistically significant improvement in documentation and compliance with PIV procedural steps, it is notable that overall compliance with the PIV insertion steps only improved to a little over 30%. When breaking down compliance into the five steps, hand hygiene prior to patient contact was about 20% pre-implementation and remained under 40% during implementation and post-implementation. Compliance was much greater, approaching or greater than 80% with the other PIV steps.

In a study of compliance with a transfusion bundle, monthly team audit and feedback were compared to monthly team audit and feedback plus individual feedback (Borgert et al. 2016). Compliance was significantly improved when the individual nurse was provided timely feedback in addition to the monthly team audit and feedback.

5 Summary

Implementing and sustaining evidence-based practice changes is challenging but is imperative in providing the best care and outcomes for the patients we serve. Increasingly, organizations are facing regulatory and reimbursement requirements and constraints that compel changes in clinical practice. Key interventions steps in implementation include involvement of a team that includes leadership, local champions, and clinical experts, an understanding of both positive characteristics of an organization that support practice change and barriers to change, and employment of sound quality improvement strategies. It is important to realize that education and well-written policies and procedures, while important and certainly needed as part of any implementation plan, are not enough.

Once implemented, there must be an ongoing evaluation of its effectiveness. The right evaluation of a vessel health and preservation program must include audit and feedback which is used in conjunction with other interventions including educational strategies. Audits will include clinician feedback, patient responses, process (e.g., the presence of inadequate dressings), and outcome (e.g., CLABSI, other complications) measures (Figs. 20.1 and 20.2). Audit and feedback are provided in a timely manner, provided more than once, provided by a supervisor or colleague both verbally and in writing, and should include clear goals/targets and an action plan. Furthermore, the audit and feedback process should be transparent to and understood by clinicians. Future research must focus on the key components of audit and feedback that will lead to greater impact of interventions.

Fig. 20.2

Patient evaluation tool for VHP (used with permission of Teleflex)

Case Study

Most of the patients on the hospital’s large surgical unit have peripheral IV catheters placed for routine fluid replacement and IV medication administration. During a team meeting, the nurses have expressed that they frequently remove and replace the catheters due to phlebitis. A review of medical records led by the unit’s quality improvement team documented a high phlebitis rate of 30%. Additionally, based upon a 2-day point prevalence study of observed peripheral IV sites, the team noted that most of the catheters were placed in the hand/wrist area. A review of the clinical policy relative to peripheral IV catheter placement, care, and management was found to be outdated. A hospital-wide team was pulled together to revise the policy and update to current recommendations especially relative to site selection and catheter gauge size.

• 1. What next steps could be taken in promoting the updated peripheral IV catheter policy?

• 2. How might adherence to the new policy changes be measured?

• 3. How should the measurement process be presented to the staff nurses?

• 4. How might the changes be addressed in a daily practice?

Summary of Key Points

• When planning to implement changes in practice, put together a team that includes leadership, local champions, and clinical experts.

• Explore and understand organizational characteristics that will support practice changes as well as any barriers to change.

• Staff education and policies/procedures are necessary in implementation of practice changes, but they are not enough.

• The right evaluation of a vessel health and preservation program must include audit and feedback that includes both process and outcome measures.

• Audit and feedback must be provided in a timely manner, provided more than once, provided by a supervisor or colleague both verbally and in writing.

• Clear goals/targets and an action plan should be part of the audit and feedback process.

• The audit and feedback process should be transparent to and understood by clinicians.