FormalPara Key Summary Points

Performance measures can be used to evaluate treat-to-target strategies in rheumatoid arthritis.

Audit and group feedback is acceptable and valuable exercise for rheumatologists to review individuals practice results.

Results from audit and feedback can be used to drive quality improvements in rheumatology clinics and future evaluation of the impact of this process on patient outcomes is planned.

Digital Features

This article is published with digital features to facilitate understanding of the article. To view digital features for this article go to https://doi.org/10.6084/m9.figshare.12961769.

Introduction

Treating-to-target (T2T) is solidified as the preferred treatment paradigm in rheumatoid arthritis (RA) care [1]. T2T compels frequent reassessment of disease activity and adjustment of medications to achieve clinical remission, or at the very least, a low-disease activity state (LDA) [1]. Principles of T2T are endorsed by major rheumatology guidelines worldwide [1,2,3,4,5] but practice variation may occur in busy clinical practices where the implementation of frequent clinical visits and use of composite measures of disease activity can be challenging [6].

Audit and feedback of clinical practice is one strategy that can be used to improve compliance with desired clinical behavior [7] such as adherence to T2T strategies, and it is endorsed as a mechanism for practice reflection and quality improvement by the Canadian Rheumatology Association (CRA) [8] and by the Royal College of Physicians and Surgeons of Canada [9]. Strategies for effective audit and feedback [10] have been leveraged by the Physician Learning Program (PLP) for physicians in the province of Alberta, Canada [11]. The PLP is available through a physician-driven Alberta Medical Association (AMA) benefit program with the vision that “by 2025, all Alberta physicians will care for patients in a supportive culture, driven by evidence-informed, reflective practice improvement.”

In the present study we collaborated with the PLP to operationalize and report on 5 performance measures from a recently developed national quality framework [12] to create individualized physician reports and presented the work in a group feedback session. The national quality framework was developed through extensive consultation with patients, healthcare providers, and healthcare leaders, and reflects the strategic priorities for the delivery of high-quality RA care [13]. From the collection of 21 performance measures (PMs), five which address T2T principles were selected for operationalization, testing and implementation using a pragmatic quality improvement and research platform called Rheum4U [12]. The measures address the following T2T concepts: frequent patient reassessment, monitoring patients using composite disease activity measures and targeting management to achieve remission or low disease activity (LDA).

The aim of this study was therefore to describe the operationalization and reporting of five PMs from the national RA Quality Improvement Framework [13] and to describe rheumatologists’ experiences with receiving individualized reports of their performance on the measures and the group feedback process.

Methods

Study Design and Setting

This study was undertaken in 2 phases: (i) operationalization of 5 PMs and development of aggregate group and individualized practice reports for participating rheumatologists through the PLP program and (ii) evaluation of rheumatologists’ experiences of the audit and group facilitated feedback process using surveys and interviews. This project was undertaken at 2 rheumatology clinics affiliated with the University of Calgary Division of Rheumatology. Ethical approval was granted by the University of Calgary Conjoint Health Research Ethics Board (REB16-0556) and all participants (patients and providers) provided written informed consent for the study including publication of results. The study was performed in accordance with the Helsinki Declaration of 1964, and its later amendments.

Clinical Data Source

Rheum4U was the platform used in this study, and is a pragmatic research and quality improvement tool developed in 2016 and implemented in 2 University-based clinics [12] with a view towards future integration into an electronic medical record (EMR). Clinic patients at both sites with inflammatory arthritis are eligible for participation in Rheum4U and provided informed consent. Basic demographic data is collected at entry into the cohort and participants are emailed a reminder to complete patient reported outcomes in advance of each visit with their usual rheumatologist. The rheumatologist reviews the patient-entered data at the time of consultation and records tender and swollen joint counts, inflammatory marker results and a physician global assessment; the Rheum4U platform calculates the Clinical Disease Activity Index (CDAI) and/or the Disease Activity Score-28 (DAS28) with either the C-reactive protein (CRP) or erythrocyte sedimentation rate (ESR) or both depending on available data. A research assistant is available to facilitate data entry.

Inclusion Criteria

Patient records included in the evaluation simply required a rheumatologist-reported diagnosis of RA and at least 1 clinic visit. Rheumatologists recruiting patients to Rheum4U provided consent to receive their rheumatologist practice reports and provide feedback through survey and interview.

Performance Measure Operationalization

Analysis of patient data was carried out for each measurement year for which a full year of data was available (2017 and 2018) and results were reported at the aggregate level and by individual rheumatologist. T2T concepts addressed by the PMs included: yearly follow-up (PM1), disease activity measurement (PM2), timely follow-up for those patients not in remission (PM3), attainment of LDA or remission within 6 months (PM4), and percent of patients in remission (PM5). Measurement specifications [13] and measures where a sensitivity analysis was conducted are shown in Table 1. For individual rheumatologist practice reports, patient records were excluded from the analysis of PMs requiring two visits (PM3 and PM4) if the individual patient had visits with multiple rheumatologists (as attribution of care to a single rheumatologist’s practice was not possible). Results were stratified according to duration of RA disease: < 1 year since diagnosis (early RA), and > = 1 year since diagnosis (established RA).

Table 1 Overview of treat-to-target (T2T) performance measures evaluated during the audit and associated sensitivity analysis
Full size table

Descriptive analysis was performed to describe patient demographic information at cohort inception. For categorical variables, frequencies and proportions were reported. For continuous variables, mean (standard deviation, SD), median (25th and 75th percentile, Q1, Q2) and range values were reported as appropriate. Data analysis was carried out using R 3.4.0 (Vienna, Austria; https://www.R-project.org/) [14].

Rheumatologist Practice Reports for T2T Audit and Feedback

The Calgary Audit and Feedback Framework [10] describes the evidence-informed approach for audit and feedback employed by the PLP who co-facilitated this project. Central to the framework are the following prerequisite activities of: relationship building, ensuring a meaningful topic choice that is actionable with clear data sources, and a report of data that is easy to understand and co-created with end-users, as well as to facilitate feedback from the group consensus [10].

A 21-page individualized and confidential rheumatologist practice report with background on the project, methods, and graphics depicting performance for each of the five PMs was developed in collaboration with the PLP team and distributed to participating rheumatologists electronically and in a paper report. Each report displayed an individual rheumatologist’s practice results compared to the aggregate. Prior to receiving their practice reports, rheumatologists completed a practice self-assessment survey developed by the PLP and adapted for the present project including the following: demographic and practice characteristics, questions about the individual’s motivation for participating in the audit and feedback session, and their estimated self-assessment of their performance on the PMs. The T2T results were discussed at an hour-long Rheumatology Grand Rounds where participants had the ability to review aggregate data as a group while reviewing their own practice reports, and discuss findings, potential explanations, and solutions.

Rheumatologist Experience with the T2T Audit and Feedback Process

Rheumatologists who received individualized practice reports were invited to complete a brief, 8-item survey about the acceptability and usefulness of the practice report. Participants rated five items on a 5-point Likert scale from 1 strongly disagree to 5 strongly agree and also reported the time they took to review the report, reflect upon it and whether they claimed any maintenance of certification credits. They were also invited to participate in an interview to further discuss their experience with the process and their opinions about the practice report. Semi-structured interviews were conducted in person or by phone by 1–2 research associates with experience in qualitative methods (VB and NS). To generate additional feedback on the process and practice reports, rheumatologists were asked to share their opinions about the evaluation framework, its implementation, and considerations for future sustainability of the initiative. Interview durations ranged from 22 to 47 min and were audio-recorded and transcribed verbatim by a transcriptionist.

Transcripts were de-identified and NVivo© 12 software (QSR International) was used for data management. Two coders (CB and VB) used a six-step thematic analysis [15, 16]: (1) coders reviewed transcripts and familiarized themselves with the data; (2) initial codes were generated. A series of regular meetings between the coders helped to (3) search for themes and (4) ensure mutual understanding of the themes. Coders then (5) independently re-examined each transcript and met to achieve consensus on final themes. Finally, the coders (6) produced a report on the themes and associated quotes which was shared with the research team.

Through this process, we used multiple strategies to ensure the elements of trustworthiness [17] (dependability, credibility, confirmability, and transferability) were met. Dependability refers to the extent that the study could be replicated with similar findings by other researchers. Two independent coders reviewed the decision-making record to control for biases and to make sure the process was logical. The coders met for peer debriefing and feedback throughout the process. An audit trail was used to document decisions and to ensure dependability and confirmability. Credibility was established through sharing final results with interview participants and our multidisciplinary research team. The report included rich descriptions of findings accompanied by quotations so that readers could judge the generalizability of the results addressing the element of transferability [15].

Results

Rheumatologist Characteristics and Aggregate Results

Eleven of 12 eligible rheumatologists consented to participate. Ten completed the pre-assessment survey and had a median (Q1, Q2) time in practice of 7.5 (3, 30) years. Rheumatologists reported a median of 14.5 (12, 20) patient consultations per day and a median of 14.3 (6, 24) half-day clinics per month. Rheumatologists’ pre-assessment of their practice on each PM is depicted alongside aggregate measured performance in Rheum4U (Tables 2, 3, 4). Figure 1 depicts rheumatologists’ agreement with five potential reasons for participation in the audit and feedback. Reflection on their own self-practice and discussing improvement strategies with peers had the highest rates (50%) of agreement.

Table 2 Results for actual and physician-estimated RA performance measures 1 and 2 stratified by year of measurement and disease duration
Full size table
Table 3 Results for RA performance measures 3 and 4 reported using clinic visits and patients and stratified by year of measurement and disease duration and physician estimates of performance
Full size table
Table 4 Actual and physician-estimated performance measure 5 (% patients in remission at any time) and sensitivity analysis of patients in remission at all clinic visits, reported using clinic visits and patients as units of analysis, and stratified by year of measurement and disease duration and physician estimate of measure performance
Full size table
Fig. 1
figure 1

Pre-report self-assessment regarding rheumatologist motivation for participation in the audit and feedback project (n = 10)

Full size image

Baseline demographics for the RA Rheum4U cohort (n = 448) are shown in Supplemental Table 1. The majority of patients were female (74.8%) with a median baseline age of 57.5 (Q1, Q2: 46.7, 65.4) years and a baseline disease duration of 4.99 (Q1, Q2:1.0, 11.3) years. There were 76 (65.5%) patients included in the cohort with < 1 year of disease duration.

The results of adherence to PM1 and PM2 are shown in Table 2 and stratified by year of reporting and by disease duration. There were overall high rates of yearly follow-up (89.4% in 2017 and 86.5% in 2018) with the lowest rates of follow-up seen in early RA patients in the 2018 measurement year (80%). All patient records met PM2 (% with assessment of disease activity at \(\ge\) 50% of visits) in all measurement years.

The results of adherence to PM3 and PM4 are shown in Table 3 using either the clinic visits or the patient records as the unit of measurement. Across all years of measurement and disease duration suboptimal performance was found on PM3 (% of patients seen within 3 months if not in remission) with < 30% of clinic visits meeting the measure in each year with slightly improved performance in early RA. A sensitivity analysis using LDA resulted in an improved performance of approximately 8% with greater improvements seen again in early RA (Supplemental Table 2). The median time in months between a visit where remission was not achieved and the next clinic visit was 4 months (3.5 months if LDA used at the index visit) (Supplemental Table 3).

In PM 4, over half of patients with active RA attained a LDA state within 6 months (61.5% in 2017 and 69.4% in 2018), with higher rates observed in patients with early RA (64.3% in 2017 and 68.7 in 2018). In the sensitivity analysis (Supplemental Table 4), the rates of LDA or remission increased when using 9- or 12-month windows for measure reporting (e.g., up to 76.9% of patients meeting the measure in a 12-month window in 2017).

Over 70% of patients were in remission at least once at any time during the measurement year by any composite disease activity score Table 4.

Group Feedback Session

During the group feedback session of these results, the following three main topics were discussed: (1) Rheum4U platform was seen as a beneficial driver of change in practice and it increased documentation of disease activity. While division members recognized some challenges with its use including possible selection bias in included participants, it was widely recognized that the data could be useful to identify areas in need of practice improvement. Increased uptake of the platform was encouraged. (2) The need for an early RA protocol was identified as a potential solution to help better standardize practice and improve patient outcomes. (3) Need to expand disease monitoring and protocols more broadly within the clinic beyond Rheum4U and early RA was encouraged to prevent inequities in care and outcomes for other patient populations.

Rheumatologist Experience with Audit and Feedback

Five rheumatologists completed feedback surveys and six volunteered to participate in an interview. Only two rheumatologists did both, therefore feedback in some form was obtained from 9/11 (81%) who received audit reports. The feedback from the surveys was favorable with all responses \(\ge\) 4 on all questions (supplemental Fig. 1). Rheumatologists estimated they spent between 4 and 60 min reading the report and between 4 and 60 min reflecting upon it. Few planned to claim maintenance of certification credits for reflecting upon their practice (n = 2 of five survey respondents).

Seven themes emerged from the analysis of the interview transcripts: measurement selection, PLP involvement in quality reporting, report formatting, interpretation of results, utility of results, Rheum4U as a data source, and reaction to individualized results. The themes and selected quotations are shown in Table 5. The first major theme was that of measurement selection. All of the rheumatologists agreed that the PMs selected for the practice report were appropriate and reflective of rheumatology guidelines. However, some rheumatologists had concerns about the composite measures of disease activity. They highlighted the inherent limitations of these measures, such as the omission of patients’ joint counts of lower limb joints using a 28-joint count in their estimation, omission of deformities and potential to be supplanted in the future by better measures. Rheumatologists were interested to see other PMs in future reports including: raw numbers of tender and swollen joints, cost of treatment, patterns of treatments (including steroid use), and patient employment status. Rheumatologists were also interested in seeing metrics on their practice with respect to management of other diseases (e.g., gout and psoriatic arthritis).

Table 5 Themes and selected quotations from rheumatologist interviews about the audit and feedback process
Full size table

In the second theme, with respect to the involvement of the PLP in the process of quality reporting, the rheumatologists thought in general it was “useful” as this group is experienced with presenting data and creating reports and there was no financial cost to the Division. Some urged caution with having a third party analyze and interpret data without rheumatologist involvement in the future while others were ambivalent about the need for the PLP to be involved in future reporting.

With respect to the report formatting (Theme 3), it was deemed easy to read and understand. The background clearly communicated the importance and rationale for the report. Most rheumatologists thought an appropriate frequency of reporting would be yearly with some expressing a preference for electronic reporting or a dashboard for continuous reporting instead of paper reports. One rheumatologist highlighted that the report would also be important for clinic managers to see.

A major theme and source of discussion was on interpretation of results (Theme 4). Many rheumatologists mentioned patient factors which could influence results and their interpretation including duration of disease, patient choice (for example decision not to take biologic therapy), patient complexity, and even rheumatologist duration of practice. Strategies suggested by rheumatologists to help with interpretation of results included stratification by disease duration or patient age. Simply reporting rheumatologist practice characteristics compared to their colleagues was another suggestion. Formal risk adjustment strategies were discussed as other possible ways to address differences in patient/practice characteristics, but rheumatologists seemed to favor stratification or reporting practice characteristics as these were believed to be easier to interpret.

With respect to the utility of the results (Theme 5), rheumatologists discussed in what format they would need to see the results for them to be actionable for quality improvement. They regarded the potential to view data longitudinally over time as important to notice trends. While all rheumatologists acknowledged that seeing the data was helpful as they had never had a practice report before, some wanted even more granular data so that they could better support improvement in patient outcomes. They also suggested instead of comparing to group mean performance to consider comparing to the top quartile of rheumatologists to better contextualize data and motivate improvements. Some rheumatologists expressed the need for more sensitivity analysis for measures involving time to remission or follow-up.

Rheum4U as a data source was the 6th theme discussed with challenges and benefits identified with the data source influencing measure results. One challenge expressed by all rheumatologists, except one, was that the Rheum4U data source was likely not representative of their entire practices with some estimating only “10 to 20%” of their practice was included (Rheumatologist 2), this led to some questioning the generalizability of the results. They expressed that patients who did not speak English, the elderly, those with insecure shelter and lower socioeconomic status limiting computer access, or those who had lower computer literacy were less likely to participate, and performance may be lower in these patients.

In contrast, rheumatologists valued having disease activity scores calculated automatically in Rheum4U and reported that for patients not included in the platform that they were much less likely to calculate composite measures of disease activity. Help from the research coordinator in ensuring completeness of data entry was also appreciated. In the future, the rheumatologists hoped that the research and quality improvement platform could be integrated into the EMR or that the metrics could be calculated using their entire clinic population.

The final theme that emerged was the reaction to individualized results. Most rheumatologists thought that their results were not surprising and were in line with how they perceived they practiced and expressed that this was “reassuring”. A few rheumatologists were surprised by their performance results and one wondered if this had anything to do with differences in the way they reported disease activity. Almost all were surprised at the low rates of follow-up within 3 months for patients not in remission and expressed that there were likely system factors which made this PM challenging to meet.

Rheumatologists described that the report and comparison to their peers was valuable. Some rheumatologists also welcomed the opportunity to obtain continuing medical education credits for their time spent reviewing and reflecting on their practice. When contemplating changing individual practice, those who perceived that they were doing well comparable to their peers expressed little incentive to make changes. Some thought if the data were more representative of their whole practice (say from an EMR) that they might have greater incentive to make practice changes. Barriers to booking changes were not believed to be entirely within the rheumatologist’s control with some citing different managers for clinic administrators and nurses which made change more challenging.

Discussion

In this study, we present the results of a quality improvement initiative using audit and group feedback to optimize T2T strategies in RA and report on rheumatologists’ experience with the process. For participating rheumatologists, this represents the first time they had viewed their data compared to their peers with respect to their RA practices as no current systems are in place in our center to track and trend rheumatologist performance or patient outcomes. The results demonstrate high rates of disease activity measurement that were facilitated by the Rheum4U platform used to collect the data; however, suboptimal follow-up times were found for patients not in remission, especially those with longer disease duration. Rheumatologists described many potential factors, including patient- and system-level factors, which could influence scheduling and may have contributed to this finding. Most rheumatologists participating valued the report and yet believed it would be unlikely to drive practice change as they thought they were in line with their peers in terms of patient outcomes. Those who thought their outcomes were not in line with their peers described possible contextual factors such as length of practice (with those in practice longer potentially having a larger number of patients with disease that was difficult to control or greater challenges adhering to follow-up standards given busy practices) or patient complexity that could explain differences seen and/or described possible changes in the way they would monitor disease activity in the future.

Rheumatologist behaviors in this audit and group feedback followed a particular predictable cyclical pattern as described by Cooke et al. [18], with the overarching themes of “interpreting the data”, “understanding evidence about best practices” leading to “change cues and change talk”. While our qualitative data were examined thematically and different named themes arose, many overlapped with the themes, subthemes, and sub-divisions from Cooke et al. [18]. For example, under Cooke’s overarching theme of “interpreting the data” are the subthemes of “reactions”, “understanding/questioning”, “justifying/contextualizing” and “reflecting”. In our qualitative work, we also elicited a theme of “reactions to individualized results”. The theme of “interpretation of results” also overlapped significantly with Cooke’s “justifying/contextualizing” theme as rheumatologists reflected on how data is affected by other “system”, “personal”, “patient factors” (Cook el al.’s subdivisions for this sub-theme) [18]. Our physicians did not find fault with the measures selected in relation to “evidence about best practices”. During our group feedback session “change cues and change talk” resulted in three main potential areas for change and quality improvement: (1) expansion of use of the Rheum4U platform (or data collection in general), (2) the need for an early RA protocol, and (3) expansion of similar initiatives in other patient populations.

Based on the feedback from rheumatologists received, the primary challenge encountered with this audit and feedback process was the data source. Our academic center has a hospital EMR that serves as a repository for patient information used in inpatient and outpatient settings but it does not collect, track or trend patient outcomes required for monitoring treat-to-target processes. This led to the development of a research and quality improvement platform (Rheum4U), although not all patients are enrolled in the optional platform, which impacted how rheumatologists viewed the results of our audit and feedback process. The present study has confirmed that the measures selected would be important to track on a broader scale amongst all clinic patients as they were important and relevant to rheumatologists and encouraged stricter adherence to T2T principles. This work also highlights the importance of tracking patient outcomes using the Rheum4U platform.

In the United Kingdom a national clinical audit of early inflammatory arthritis care was conducted which revealed high rates of DAS28 recorded in the charts (91%) at baseline with increasing rates of unrecorded DAS28 over follow-up [19]. In those with available data, > 60% achieved a reduction of > 1.2 from baseline DAS28 scores; however, 11% still had high disease activity after 3 months [19]. A major finding from the audit was a need to improve systems for capturing, coding and integrating patient data from outpatient clinics [19]. In the USA, the Rheumatology Informatics System for Effectiveness (RISE) is a national electronic health-record based registry that passively collects data from rheumatology practices and can be used to monitor quality of care [20]. RISE represents an example of a largescale effort to better track and trend quality of arthritis care. Electronic quality measures have been specified for use in RISE, which focus on processes of care [21]; however, due to the complexities of developing high-stakes RA outcome measures, which could affect rheumatologist compensation [22], work is still ongoing in the USA to develop and endorse RA outcome measures. Given the lower stakes of quality improvement in our center, no formal risk-assessment strategy has been employed for the outcome measures we evaluated. In Canada, primarily time-consuming chart reviews have been used in studies on audit and feedback [23], although methods of leveraging electronic records to evaluate quality of care are emerging [24].

While our study provides the first assessment of the RA PMs to facilitate T2T endorsed in the National RA Quality Framework [13], a number of limitations should be noted as well as potential consequences of this process. As highlighted by our rheumatologist participants, the high adherence to PMs observed may have been affected by patient selection, and the nature of the data source (e.g., Rheum4U data fields and automatic calculation of disease activity scores). Performance may be lower when measured on entire clinical practices, using EMR or paper chart reviews. The included measures did not account for time to treatment start. As treatment delays could impact time to remission/LDA, in future we aim to develop strategies such as linkage to pharmacy dispensation data to better determine timing of treatment changes/starts. We also did not capture insurance status of participating patients. In Alberta there is readily available medication insurance at low cost through a provincial program or through social assistance programs and we anticipate it is less likely that delays or lack of insurance coverage contributed to poor outcomes. Nevertheless, the predictors of poor outcomes according to insurance status and sociodemographic factors could be explored in the future. The results are also from a single academic center, which could have impacted performance results and rheumatologists’ views on quality reporting. It is likely that rheumatologists participating in Rheum4U, and particularly those agreeing to interviews had more favorable perspectives on quality monitoring than the general population of rheumatologists. A potential consequence of the voluntary nature of participation for patients and physicians in this quality improvement endeavor is the potential to further increase inequities in T2T practices between enrolled and non-enrolled patient participants who may differ on characteristics such as age, computer, and health literacy, and language spoken.

Conclusions

The audit and feedback of T2T measures has proven a valuable exercise for our center. In the future, we have plans to provide continuous reporting on these measures through a quality improvement initiative in our Department of Medicine and to expand measurement to other key areas. A new EMR will be available in our region shortly, and further work will aim to expand measurement to all patients seen in clinic to better track and trend areas in need of quality improvement among the entire patient population. Optimizing T2T strategies takes time and effort and health system infrastructure including IT platforms to assist with tracking/trending, viewing and imputing the data required for these measures will be central to a sustainable quality improvement strategy. Relying on busy individual providers to make shifts in practice without additional resources and/or incentive to do so is likely to be less successful. Leveraging allied health providers in rheumatology in the collection and tracking of quality is another strategy that should be explored during further implementation to improve sustainability of tracking and improving T2T strategies. A key area of future study will be to evaluate the impact of efforts to provide quality reports to rheumatologists on improving practices and ultimately patient outcomes.